Archive for the ‘Cloudfork’ category

« Smalltalk in The Cloud home page
« Older Entries

CloudforkSSO on Pharo with stunnel

February 23, 2011

The latest version of CloudforkSSO (version 1.1.0) on Pharo uses the Zinc-HTTP library. This library contains a good http client. Actually it contains multiple http clients, depending on your requirements you can pick one of them.

One feature Zinc-HTTP doesn’t support (yet?) is secure http. Most of the OAuth and OpenID providers require https connections so this is a problem. There is a good workaround for this: you can use stunnel to handle the https protocol.

I used the following stunnel.conf on OS X:

; protocol version (all, SSLv2, SSLv3, TLSv1)
sslVersion = SSLv3

; security enhancements for UNIX systems
; for chroot a copy of some devices and files is needed within the jail
chroot = /opt/local/var/lib/stunnel/
setuid = nobody
setgid = nogroup
; PID is created inside the chroot jail
pid = /stunnel.pid

; performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
;compression = zlib

CAfile = /opt/local/etc/stunnel/cacert.pem

; SSL client mode
client = yes

[google]
accept = 127.0.0.1:20011
connect = http://www.google.com:443

[twitter]
accept = 127.0.0.1:20012
connect = api.twitter.com:443

[facebook]
accept = 127.0.0.1:20013
connect = graph.facebook.com:443

The cacert.pem file is required to validate the server certificates. I downloaded a version from http://curl.haxx.se/ca/cacert.pem.

Now the Smalltalk side of things:

Load the latest CloudforkSSO version:

Gofer new
	squeaksource: 'Cloudfork'; 
	package: 'ConfigurationOfCloudforkSSO';
	load.
(Smalltalk at: #ConfigurationOfCloudforkSSO) project latestVersion load: 'Tests'

Instruct CloudforkSSO to use the tunnels for the secure hosts:

	self default httpClient: ((CFHttpClientTunnelAccess new)
		client: CFHttpClientZincAccess new ;
		tunnel: 'www.google.com' through: 'localhost:20011' ;
		tunnel: 'api.twitter.com' through: 'localhost:20012' ;
		tunnel: 'graph.facebook.com' through: 'localhost:20013' ;
		tunnel: 'open.login.yahooapis.com' through: 'localhost:20014' ;
		tunnel: 'openid.hyves-api.nl' through: 'localhost:20015' ;
		yourself )

Also see the example CFServicePharoUtils class>>setupHttpsTunnels. You can test the connection:

CFPlatformServiceUtils default httpClient httpGet: 'https://www.google.com/'

The demo application at http://sso.doit.st is also using stunnel.

Note that there are alternatives for stunnel. One of them is described here.

Categories: Cloudfork
Tags: , , , ,

Comments: 1 Comment

CloudforkSSO – OpenID and OAuth support for Smalltalk

February 15, 2011

With the CloudforkSSO library you can let the users of your Seaside web application login using their Google or Yahoo accounts. This works using the OpenID2 protocol. CloudforkSSO also contains OAuth support. With this protocol you can ask users for permission to access their data on other websites.

Some providers that you can integrate with:

Google – Google supports OpenID2 and OAuth1. You can use them separately or together. With OpenID2 users can identify themselves using their Google accounts. It is also possible to get some information about the user. For example the user’s name and email address.

Google’s OAuth1 support is really powerful. With it you can access and modify (if the user agrees!) a users calendar, contact, documents and a lot more.

Twitter – Twitter supports OAuth1. You can ask a user for read-only or full permission.

Facebook – Facebook supports OAuth2. You can access the social graph of a user.

A demo of this library runs at http://sso.doit.st.

OpenID2 and OAuth1 are pretty complex protocols. CloudforkSSO implements part of the protocol, enough to support the major providers like Google, Yahoo and Twitter. OAuth2 is a simple protocol. If OAuth2 is all you need CloudforkSSO is probably overkill. A HTTP Client with ssl support is enough.

I developed CloudforkSSO in Pharo Smalltalk. Most providers require secure communication via https. As far as I know, the only HTTP Client for Squeak and Pharo that supports https is WebClient with SqueakSSL. This works fine on Windows but on Ubuntu Linux the SqueakSSL plugin doesn’t work with some providers, for example Twitter and Yahoo. Hopefully this will be fixed soon.

There is also a VA Smalltalk port on VAStGoodies.com which does work on Windows and Linux.

The package Cloudfork-SSO-Seaside contains a demo component that shows how you can use the functionality. Note that for OAuth you need an API key and secret for the provider you want to use. For OpenID it is important to set the correct realm. This is the host and port where your app is running. You can set these configuration properties using the Seaside configuration app:

Categories: Cloudfork
Tags: , ,

Comments: 6 Comments

EC2 AMI with Smalltalk and Seaside: Update 3

October 10, 2010

Today I published a new version of the Amazon EC2 image with Smalltalk and Seaside. The third version contains the following software:

  • Ubuntu Server 32bit 10.04 LTS (release 2010-09-23)
  • Apache HTTP Server 2.2
  • SqueakVM 4.0.3.2202 Unix VM
  • PharoCore 1.1.1
  • Seaside 3.0

The AMI identification: ami-c2fe0aab

The update to the Ubuntu server makes it easier to run the AMI as a Micro (t1.micro) instance type. This is currently the smallest and cheapest EC2 image type. The image type is very well suited for running Seaside web applications. When your application becomes more popular you can scale by using a bigger instance type or by running multiple instances.

Categories: Cloudfork
Tags: , , ,

Comments: Be the first to comment

A new public EC2 AMI with Smalltalk and Seaside

August 29, 2010

This weekend I created a public Amazon Machine Image (AMI) setup as a Smalltalk web server with a Linux SqueakVM, a Pharo 1.1 Smalltalk image and the Apache2 web server. Anyone can use this AMI as a robust and scalable runtime platform for their Seaside applications.

In this article I will try to explain how you can you can use and customize this image. First their are some prerequisites:

Using EC2 is not free, you have to pay for each hour the server runs and you have to pay for storage and bandwidth. Seaside runs fine on the smallest 32bit AMI that costs $0,085 cents per hour. Bandwidth is $0,15 per GB (the first GB is free). You can get a discount on the costs per hour if you reserve you AMI’s beforehand.

The AMI is based on a 32bit Ubuntu 10.04 server AMI. The Ubuntu server is a very minimal install, I have added Apache2 and the SqueakVM. I have also setup a Pharo image with the Seaside runtime packages and Apache2 is configured in such a way that all requests are forwarded to Seaside.

How can you start using this AMI?

Start an EC2 instance using the public AMI ami-0800ea61 (ubuntu-10.04-i386-smalltalk-server-v01). The AMI is available in the US East region. This is the cheapest region :-) When you start the image you have to select a security group. This defines the firewall rules for the AMI. You can select the default group or some other group that allows TCP connections for the ports 22 and 80.

When the image is started it should have a generated a public DNS name. When you enter this name in the address bar of your browser, you should see something similar to http://ec2demo.doit.st If the web page is shown it means that everything is running OK.

You can connect to the running AMI instance using ssh. You should login with the userid ubuntu. No password is required because a private key is used for authentication. The Pharo image that runs on the AMI includes RFB, a VNC server implementation. With a VNC client we can remotely control the Smalltalk image. The best way to do this is to tunnel the VNC connection (port 5900) through ssh:

ssh -i id_rsa-gsg-keypair -L 5900:localhost:5900 ubuntu@ec2demo.doit.st

Now you can connect to the image using a VNC client use display number 0 and the password is seaside. Note that you have to keep port 5900 closed for direct access otherwise this would be very unsecure.

Through the VNC client you have access to the Smalltalk image. You should see a workspace. I used the code in this workspace to build the image from a standard PharoCore 1.1 image. Note that I only loaded the runtime parts of Seaside and some small examples. This image is used for hosting an application and not for development so it is probably safer and more efficient to leave out the development packages.

If the image suits your needs you can just load your packages through Monticello. There is no config web application to configure your Seaside components. You can do this via the workspace or better, make a class side initialize method that does this automatically when the package is loaded. I used the the Apache2 configuration described in the Seaside book (the non-clustered setup for now). You can read the Deployment chapter for more details. You should have you app running within a few minutes.

If the Smalltalk image does not suit your needs you can replace it with another Squeak or Pharo image. The image is stored in the directory /srv/seaside If your new image has a different name you need to update the run script in the same directory.

Of course you can also install more Ubuntu packages. You could install a local database system like MySQL or PostgreSQL. But if you are really into cloud computing you probably don’t want to maintain a database yourself. Instead you can use RDS, an AWS managed MySQL service or SimpleDB. SimpleDB is a key-value store with a simple API and great scalability. You can use the Cloudfork ActiveItem Smalltalk library to persist your Smalltalk objects to SimpleDB.

To setup a nice domain name for your app (like ec2demo.doit.st) you should reserve a static IP using the AWS Console. You can then go to your domain name registrar and update the name servers to point to this IP number.

Ok, this is about it. If you have questions or ideas on how to improve the Smalltalk AMI please let me now. I think the current AMI can be used as a production environment for Seaside apps.

Categories: Cloudfork
Tags: , , ,

Comments: 3 Comments

Amazon AWS Region Endpoints in Europe

June 1, 2010

To use the Cloudfork classes for services located in Europe (Ireland), you need to change the serviceUrl property such as:

|sdb|
sdb := CFSimpleDB new.
sdb serviceUrl: 'http://sdb.eu-west-1.amazonaws.com'
Service URL
SimpleDB sdb.eu-west-1.amazonaws.com
SQS eu-west-1.queue.amazonaws.com
EC2 eu-west-1.ec2.amazonaws.com
SNS sns.eu-west-1.amazonaws.com
S3 Set Bucket location constraint to EU
RDS rds.eu-west-1.amazonaws.com

A comprehensive list can be found over at Elastician

Categories: Cloudfork
Tags: , , ,

Comments: Be the first to comment

Testing Cloudfork AWS SimpleDB based classes

September 20, 2009

The Cloudfork framework includes an alternate implementation of CFSimpleBase that stores all items in memory. The CFSimpleDBEmulator was initially created to support the unit testing of the ActiveItem framework. With the exception of some query constructs, it implements the complete api and therefore is suitable for unit testing your own applications as well.

| emulator domain item |
emulator := CFSimpleDBEmulator new.
domain := (emulator createDomain: 'myapp.development') result. 
item := CFSimpleDBItem newNamed: 'user.dennis'.
item valueAt: 'birthday' put: '20060916'  ;valueAt: 'hobby' put: 'soccer'.
domain putItem: item.

After running the tests, you can inspect the emulator to see what items have been stored in which domains and what attributes they have. If you store the emulator in some class var then you can keep the data around for development too.

ActiveItem
Because the ActiveItem framework is build on top of SimpleDB, the same emulator class can be used to unit test those applications. ActiveItem uses a globally shared CFSimpleDB instance so you only need to replace that with an emulated instance. 

CFActiveItem activateWithSimpleDB: CFSimpleDBEmulator new.

Using the CFActiveItemSerializer you can even dump the items to a local Filesystem for development convenience.

Categories: Cloudfork
Tags: , , , , ,

Comments: Be the first to comment

Secure access to AWS from VisualWorks

July 9, 2009

Cloudfork implements the REST API of the Amazon Web Services using both secure (https) and non-secure (http) communication. In order to use the https protocol to access S3,SimpleDB,SQS or EC2, you need to prepare the Smalltalk image by registering a trusted certificate. Without that certificate, your application will produce an error saying “CA Not in Trust Registry!” (CA = Certificate Authority). The steps below describe how to register the correct certificate in a VisualWorks (or WebVelocity) image.

Install HTTPS
Unless already loaded in your image, you need to install the HTTPS parcel (use Parcel Manager).

Export Certificate
Amazon WebServices uses the following root certificate “VeriSign Class 3 Secure Server CA”. You can verify this by inspecting the chain object in the debugger that can be opened if you have a failed secure test.

One way to get this certificate file is to export it from the list of certificates known to your Internet Browser. For FireFox users, open Preferences>Advanced>Encryption>View Certificates. Under VeriSign, Inc., select the certifcate, export it using the format “X.509 Certificate with chain (PEM)” and name it “VeriSignClass3SecureServerCA.pem”.

Import Certificate
The following script will import the Base-64 encoded certificate file.

| certificate registry |
registry := Security.X509.X509Registry default.
certificate := Security.X509.Certificate fromFile:'VeriSignClass3SecureServerCA.pem'.
registry addCertificate: certificate.

Please be aware of what is stated in the VisualWorks SecurityGuide.pdf (page 72): “Adding a CA certificate to your registry is deceivingly simple and does not convey the degree of trust actually involved in that action. Be sure to understand what it is you are trusting a CA to do and ensure that it matches the security requirements of your application.”

Run the Tests
Results of the secure Cloudfork Integration tests should all be in the green now.

Categories: Cloudfork
Tags: , , , , ,

Comments: 1 Comment

HTTP Clients for Squeak

May 25, 2009

Cloudfork-AWS makes the Amazon Web Services (AWS) S3, SQS and SimpleDB easily accessible from Smalltalk. All the communication between the Smalltalk image and AWS is done via HTTP. So a HTTP Client is an important requirement for Cloudfork-AWS.

Cloudfork-AWS needs more than just handling simple HTTP GET and POST requests, the following features are also needed:

  • Setting custom request headers – S3 uses custom headers for authentication and for attaching meta-data to S3 objects. We need to be able to set these headers. This feature is also required for range requests, with these requests you can download a part of a S3 object instead of downloading the entire object.
  • Access to the response headers – So we can read the meta-data of S3 objects.
  • Support for PUT, HEAD and DELETE requests – Also required for S3. PUT is required for storing objects and creating buckets. DELETE is required for removing objects and HEAD for getting the object meta-data without downloading the object itself.
  • HTTPS support – The AWS services can be accessed via plain HTTP or via secure HTTPS. The choice is up to the client. But the release notes of the latest releases of SimpleDB mention that HTTP support will be deprecated and that future versions will require HTTPS.
  • HTTP/1.1 support – Not a must have feature but version 1.1 requests can be more efficient than version 1.0 requests because of the keep-alive feature of version 1.1. With this feature socket connections can be reused between requests.
  • Streaming uploads and downloads – Also not a must have feature for most use cases. Only when large s3 objects need to be handled.
  • Proxy support – Not a requirement of one of the AWS services but a feature that is often required by client configurations.

Note that most of these features are only required for S3 and not for SQS and SimpleDB. SQS and SimpleDB only use GET and POST requests and the authentication is done in the URL and not through HTTP header fields. The HTTP responses of SQS and SimpleDB always contain XML and the maximum size is about 8KB for SQS and 1MB for a SimpleDB resultset so streaming support is not required.

As far as I know there are three HTTP clients available for Squeak:

  • The HTTPSocket class – This class is part of the Network-Protocols package and is part of the standard images of the latest Squeak and Pharo versions.
  • SWHTTPClient – This is an extensive HTTP client library. It was originally developed for Dolphin Smalltalk and was ported to Squeak. The latest release is not fully compatible with the latest Squeak release. There are a number of class extension collisions.
  • CurlPlugin – This is a Squeak plugin that uses the libcurl C library, libcurl is a well-known and powerful open source “URL transfer library” with support for HTTP, FTP and many other protocols.

HTTPSocket

This is a very simple implementation of a HTTP client in a single class. HTTP GET and POST requests are supported, access to the headers is also possible and simple proxy configurations are also supported. HTTP version 1.1 is not supported, HTTPS is also not possible.

The current version of Cloudfork-AWS does not work with HTTPSocket as a HTTP client. With the provided functionality it should be possible to support the SQS and SimpleDB API’s. But when I use HTTPSocket I get an AWS error telling me that the signature calculated is wrong. I think this is because HTTPSocket always adds the port number to the host header field. Cloudfork doesn’t do this when it calculates the signature so you get a mismatch. It is on my todo list to fix this.

SWHTTPClient

SWHTTPClient is a full featured HTTP client library. It supports HTTP/1.1, access to the header fields and the PUT, HEAD and DELETE methods. Streaming uploads and downloads are also possible. The one thing that is not supported or that I couldn’t get working is HTTPS. Perhaps it’s possible to get this working by plugging in the Cryptography package but I have no idea how.

Another issue is that SWHTTPClient is not fully compatible with the latest Squeak and Pharo releases. The package contains some class extensions that override exiting methods with different behavior. For example the String>>subStrings: method.

Cloudfork-AWS can use SWHTTPClient, all AWS features work except HTTPS. I have fixed all the incompatibilities I bumped into. The patched version of SWHTTPClient is available from the Cloudfork project page on SqueakSource.

CurlPlugin

The installation of this library is a bit more work. You need the place the correct binaries for your platform in the Squeak installation directory and load the CurlPlugin package from SqueakSource. If you load the package you may get a warning that the class CurlPlugin cannot be loaded. This is no problem, you can still use the plugin through the Curl class. The CurlPlugin class is only needed if you want to create a new version of the plugin or support a new platform.

The libcurl library that the CurlPlugin uses supports all the HTTP features we need and many more. It is one of the bests HTTP client libraries around. And it’s open source. It has an optional integration with openssl which provides the functions required for HTTPS.

The current version of the CurlPlugin doesn’t expose all the features of libcurl. Currently HEAD and DELETE requests are not supported. It is also not yet possible to set the header fields for a requests. The other methods work very well and HTTPS also works fine.

Cloudfork-AWS

For the SimpleDB and SQS services the CurlPlugin is the best HTTP client. All the required features are there and the performance is very good. SimpleDB and SQS also work with the SWHTTPClient, only without HTTPS support. If the Curl class is present in your image Cloudfork-AWS will use this class for all SimpleDB and SQS service calls, otherwise the SWHTTPClient is used.

The current CurlPlugin doesn’t support all the features required by the S3 service. For this reason the Cloudfork S3 functionality requires the SWHTTPClient.

Future work

I think the CurlPlugin has the potential to become a very good HTTP client library for Squeak and Pharo. It will also be relatively easy to maintain this library because all of the complex work of supporting the different protocols is implemented in libcurl. This C library has a very large community and is well maintained. I will try to extend the plugin and add the missing features.

I will also try to make Cloudfork-AWS compatible with HTTPSocket. This will not be the best performing solution but it can be an easy starting point.

Categories: Cloudfork
Tags: , , , , , ,

Comments: 5 Comments

Composition relations in Cloudfork-ActiveItem

April 20, 2009

In UML, the composition relation between objects is a special association that is used to model a “private-container” relationship. The typical class-room example is the Car object having 4 Wheel objects. Although you can replace wheels on a car, one particular Wheel object is never shared with other Car objects.

In Amazon SimpleDB there is no concept of relations ; it is a simple storage of items having attributes (key-value pairs). The Cloudfork-ActiveItem framework can map these relations to foreignkey-like attributes but that should be used with care. Because SimpleDB is not a relational database, operations such as Joins are simply not possible. However, mapping the composition relation fits much better in the SimpleDB storage model. The notion of a SimpleDB item being a container of information is just what it is meant to be.

To illustrate how ActiveItem supports this design construct, I will give an example that models multiple-choice questions for an exam training application. A Question is a composition of 4 Choices ; one of them is the correct answer to that question.

Question class>>describe: aQuestion

  aQuestion
    hasString: #code ;
    hasText: #text ;
    ownsMany: #choices

Choice class>>describe: aChoice

  aChoice
    hasText: #text ;
    hasBoolean: #isAnswer

When saving a Question, ActiveItem will create one SimpleDB item with both the attributes of the Question and the attributes of each Choice:

code -> '010-0001'
text -> 'What language is best for writing Web applications?'
choices.1.text -> 'Java'
choices.1.isAnswer -> 'false'
choices.2.text -> 'Ruby'
choices.2.isAnswer -> 'false'
choices.3.text -> 'PHP'
choices.3.isAnswer -> 'false'
choices.4.text -> 'Smalltalk'
choices.4.isAnswer -> 'true'

By supporting composition, class Choice can be a normal ActiveItem subclass with its own attribute description. However, as you can see in the example, Choice objects do not need an id (actually the collection index is the id).

Because of limitations to the number of attributes per item (currently 256), this composition solution is not suitable for arbitrary large collections. If you expect this for your model then I suggest you use the normal hasMany: method that maps associations using “foreign-key” attributes.

Categories: Cloudfork
Tags: , , , ,

Comments: Be the first to comment

Problems with Daylight saving time in VA Smalltalk

March 29, 2009

All the requests that Cloudfork-AWS sends to the Amazon web services contain the current date and time in Coordinated Universal Time (UTC). If this timestamp differs more than a few seconds from the current time you get an error. For example the S3 error: RequestTimeTooSkewed – The difference between the request time and the current time is too large. The reason for this time check is security, it prevents “record en playback” attacks.

So systems that make use of AWS must have the correct time and also the timezone must be correct. Otherwise the conversion to UTC will give the wrong result. A few days ago this all worked perfectly in VA Smalltalk, but tonight all AWS calls fail :-( Last night we in The Netherlands switched to Daylight saving time (DST). VA Smalltalk doesn’t seem to handle this very well. A call to “DateAndTime now” still returns an offset from UTC as one hour instead of two. It seems that this is a known problem.

Until this problem is fixed we have to use a less than elegant solution to get things working again. We have added a “DSTMode” flag, when this flag is true we subtract an extra hour when converting to UTC. You can enable this mode by executing:


CFPlatformServiceVASTUtils enableDSTMode: true

Support for the DSTMode was built into Cloudfork version jvds.79.

Categories: Cloudfork
Tags: , , , ,

Comments: 3 Comments

« Older Entries
Follow

Get every new post delivered to your Inbox.