In this article I will try to explain how you can you can use and customize this image. First their are some prerequisites:

• You need an Amazon AWS account and you should sign up for the Elastic Compute Cloud (EC2) service.

Using EC2 is not free, you have to pay for each hour the server runs and you have to pay for storage and bandwidth. Seaside runs fine on the smallest 32bit AMI that costs $0,085 cents per hour. Bandwidth is $0,15 per GB (the first GB is free). You can get a discount on the costs per hour if you reserve you AMI’s beforehand.

The AMI is based on a 32bit Ubuntu 10.04 server AMI. The Ubuntu server is a very minimal install, I have added Apache2 and the SqueakVM. I have also setup a Pharo image with the Seaside runtime packages and Apache2 is configured in such a way that all requests are forwarded to Seaside.

How can you start using this AMI?

Start an EC2 instance using the public AMI ami-0800ea61 (ubuntu-10.04-i386-smalltalk-server-v01). The AMI is available in the US East region. This is the cheapest region When you start the image you have to select a security group. This defines the firewall rules for the AMI. You can select the default group or some other group that allows TCP connections for the ports 22 and 80.

When the image is started it should have a generated a public DNS name. When you enter this name in the address bar of your browser, you should see something similar to http://ec2demo.doit.st If the web page is shown it means that everything is running OK.

You can connect to the running AMI instance using ssh. You should login with the userid ubuntu. No password is required because a private key is used for authentication. The Pharo image that runs on the AMI includes RFB, a VNC server implementation. With a VNC client we can remotely control the Smalltalk image. The best way to do this is to tunnel the VNC connection (port 5900) through ssh:

ssh -i id_rsa-gsg-keypair -L 5900:localhost:5900 ubuntu@ec2demo.doit.st

Now you can connect to the image using a VNC client use display number 0 and the password is seaside. Note that you have to keep port 5900 closed for direct access otherwise this would be very unsecure.

Through the VNC client you have access to the Smalltalk image. You should see a workspace. I used the code in this workspace to build the image from a standard PharoCore 1.1 image. Note that I only loaded the runtime parts of Seaside and some small examples. This image is used for hosting an application and not for development so it is probably safer and more efficient to leave out the development packages.

If the image suits your needs you can just load your packages through Monticello. There is no config web application to configure your Seaside components. You can do this via the workspace or better, make a class side initialize method that does this automatically when the package is loaded. I used the the Apache2 configuration described in the Seaside book (the non-clustered setup for now). You can read the Deployment chapter for more details. You should have you app running within a few minutes.

If the Smalltalk image does not suit your needs you can replace it with another Squeak or Pharo image. The image is stored in the directory /srv/seaside If your new image has a different name you need to update the run script in the same directory.

Of course you can also install more Ubuntu packages. You could install a local database system like MySQL or PostgreSQL. But if you are really into cloud computing you probably don’t want to maintain a database yourself. Instead you can use RDS, an AWS managed MySQL service or SimpleDB. SimpleDB is a key-value store with a simple API and great scalability. You can use the Cloudfork ActiveItem Smalltalk library to persist your Smalltalk objects to SimpleDB.

To setup a nice domain name for your app (like ec2demo.doit.st) you should reserve a static IP using the AWS Console. You can then go to your domain name registrar and update the name servers to point to this IP number.

Ok, this is about it. If you have questions or ideas on how to improve the Smalltalk AMI please let me now. I think the current AMI can be used as a production environment for Seaside apps.

|sdb|
sdb := CFSimpleDB new.
sdb serviceUrl: 'http://sdb.eu-west-1.amazonaws.com'

A comprehensive list can be found over at Elastician

| emulator domain item |
emulator := CFSimpleDBEmulator new.
domain := (emulator createDomain: 'myapp.development') result.
item := CFSimpleDBItem newNamed: 'user.dennis'.
item valueAt: 'birthday' put: '20060916'  ;valueAt: 'hobby' put: 'soccer'.
domain putItem: item.

After running the tests, you can inspect the emulator to see what items have been stored in which domains and what attributes they have. If you store the emulator in some class var then you can keep the data around for development too.

ActiveItem
Because the ActiveItem framework is build on top of SimpleDB, the same emulator class can be used to unit test those applications. ActiveItem uses a globally shared CFSimpleDB instance so you only need to replace that with an emulated instance.

CFActiveItem activateWithSimpleDB: CFSimpleDBEmulator new.

Using the CFActiveItemSerializer you can even dump the items to a local Filesystem for development convenience.

The following features are available in the initial version:

• Symmetric encryption and decryption using RC4, DES, TripleDES and AES-256
• The MD5, SHA and SHA-256 hash functions
• The HMAC-MD5, HMAC-SHA and HMAC-SHA-256 message authentication functions.

The API is pretty simple to use. To encrypt a String or ByteArray using a password you can use the class methods defined in WinCryptCipher:

| encryptedBytes |
encryptedBytes := WinCryptAES encrypt: 'my-dirty-little-secret' using: 'puppy'

The results is always a ByteArray. This ByteArray can be decrypted and converted to a String:

| encryptedBytes decryptedBytes |
encryptedBytes := WinCryptAES encrypt: 'my-dirty-little-secret' using: 'puppy'.
decryptedBytes := WinCryptAES decrypt: encryptedBytes using: 'puppy'.
^decryptedBytes asString

The ciphers use a key with a specific size for encryption and decryption. When you provide a password string the key is derived from this password using the MD5 hash algorithm. It is also possible to skip this step and supply your own key.

The WinCrypt library is open source and is published on VAStGoodies.com. The included SUnit tests show how the library can be used. The underlying Windows functions are documented on MSDN.

Install HTTPS
Unless already loaded in your image, you need to install the HTTPS parcel (use Parcel Manager).

Export Certificate
Amazon WebServices uses the following root certificate “VeriSign Class 3 Secure Server CA”. You can verify this by inspecting the chain object in the debugger that can be opened if you have a failed secure test.

One way to get this certificate file is to export it from the list of certificates known to your Internet Browser. For FireFox users, open Preferences>Advanced>Encryption>View Certificates. Under VeriSign, Inc., select the certifcate, export it using the format “X.509 Certificate with chain (PEM)” and name it “VeriSignClass3SecureServerCA.pem”.

Import Certificate
The following script will import the Base-64 encoded certificate file.

| certificate registry |
registry := Security.X509.X509Registry default.
certificate := Security.X509.Certificate fromFile:'VeriSignClass3SecureServerCA.pem'.
registry addCertificate: certificate.

Please be aware of what is stated in the VisualWorks SecurityGuide.pdf (page 72): “Adding a CA certificate to your registry is deceivingly simple and does not convey the degree of trust actually involved in that action. Be sure to understand what it is you are trusting a CA to do and ensure that it matches the security requirements of your application.”

Run the Tests
Results of the secure Cloudfork Integration tests should all be in the green now.

This is a first release of the Smalltalk library; in this release not all the features are available from Smalltalk yet. But all the common functionality can already be used:

• HTTP GET, POST and HEAD requests
• Support for setting custom request headers and access to the response headers
• Support for secure http (https) including support for client certificates
• Streaming downloads and uploads
• Proxy support
• Server and proxy authentication support

We developed this library because the standard HTTP client of VA Smalltalk is not fully stable on Windows when using secure connections on multi-core systems. Nationaal Spaarfonds (a part of Delta Lloyd Group) has a VA Smalltalk application in production that needs to call secure webservices. This works fine most of the time but sometimes causes the image to freeze. See the Instantiations forum for a discussion of this problem.

WinHttpClient can be plugged into the VA Smalltalk webservice stack as a replacement for the standard HTTP client. Our stress tests show that the stability problems disappear after this switch.

The WinHttpClient library is open source and is available on VAStGoodies.com. If you are interested please try it out. If you have suggestions on how we can improve the API we would love to hear them. Contributions to add missing features are also very welcome.

WinHttpClient API

The API is structured the same way as the Windows winhttp library. You can find the documentation on MSDN: http://msdn.microsoft.com/en-us/library/aa384273(VS.85).aspx

Important concepts are Session, Connection and Request. Before you can do anything with the library you need a Session instance:

| session response |
session := WinHttpSession new.
response := session submitGetRequest:
	'http://en.wikipedia.org/wiki/Smalltalk'

The WinHttpSession class has a number of submitXXX methods. These methods create and send HTTP requests and handle the responses. They return an instance of WinHttpResponse, this object contains the HTTP status, the HTTP header information and the contents.

If the contents of the response is possibly very large than it is better to use a streaming variant of the submit request which has a write stream as an argument. This method will still answer a WinHttpResponse instance but the contents variable of this instance will be nil.

After you have finished with the session you should send “session release”. This is because the session object contains a handle to a Windows structure that must be freed to prevent memory leaks. The class side of WinHttpSession contains a number of convenience submit methods that handle the release message for you.

WinHttpConnection

If you need to send multiple requests to the same host it is more efficient to work with a connection object instead of a session:

| session connection responses |
session := WinHttpSession new.
responses := OrderedCollection new.
connection := session connectTo: 'en.wikipedia.org'.
[
	responses
		add: (connection submitGetRequest: '/wiki/Smalltalk');
		add: (connection submitGetRequest: '/wiki/HTTP');
		add: (connection submitGetRequest: '/wiki/Virtual_machine')
] ensure: [ connection release ].
responses

WinHttpRequest

Normally you don’t need to work with requests objects directly, you can just use the submit methods of the session or connection objects. Only when you need to set special options you need to access the request object directly. An example of this is when you want to send a request to a secure web server with an invalid server certificate. The default behavior of winhttp is to refuse the connection and throw an error. For example, if you want to accept certificates with an unknown certificate authority you need to set the “IgnoreUnknowsCA” option:

| url session connection request response |

url := 'https://www.securebutinvalid.com' sstAsUrl.
session := WinHttpSession new.
connection := session connectToUrl: url.

request := connection openGetRequest: url absolutePath.
request setSecurityOptionIgnoreUnknownCA.
request send.
response := request getResponse.

request release.
connection release.
session release.
response

Note that we think this part of the WinHttpClient API is a bit awkward. The intention is to improve this in future versions.

Proxy support

WinHttp has support for proxy’s and a number of proxy authentication methods (Basic, NTLM etc). When you instantiate a session you can specify the proxy to use or you can use the defaultProxy class methods to set a default for all sessions.

It is also possible to copy the proxy settings that are configured in Internet Explorer. Use initProxyInfoFromIE class method of the Session class for this.

For more examples on how to use WinHttpClient see the unit tests in WinHttpClientTestApp.

Using WinHttpClient in the VA Webservice stack

The application WinHttpClientWebServiceSupportApp contains classes that can replace the standard HTTP Client with WinHttpClient for calling webservices. Thanks to the flexible way that webservice support is implemented in VA Smalltalk this is quite easy to do. We just have to replace the default HTTP and HTTPS dispatchHandlers with a custom version. We can do this when we create the webservice container:

| container httpHandler secureHttpHandler |

container := SstWSContainer containerNamed: 'test' ifNone: [ SstWSContainer createContainerNamed: 'test' ].

httpHandler := WinHttpDispatchHandler new.
secureHttpHandler := WinHttpDispatchHandler new.

container handlerFactory
	register: httpHandler named: 'wsHttpClientRequestHandler' inNamespace: container handlerFactory globalNamespace;
	register: secureHttpHandler named: 'wsHttpsClientRequestHandler' inNamespace: container handlerFactory globalNamespace.
container

For a full example see the test class WinHttpWeatherForecastTest.

Secure webservices with client certificates

Supporting secure webservices that use HTTPS was very easy with WinHttpClient. But adding support for client certificates involved a bit more work. To set a client certificate for a request another Windows DLL is required: crypt32.dll. This DLL is also shipped with all the current Windows platforms. This DLL contains the functionality to handle certificates and all kinds of secure hashing, encrypting and decrypting functionality. In the future we can make all this functionality available from Smalltalk in a separate project. Currently WinHttpClient uses just a few functions from this DLL to set the correct certificate for a request.

You can tell the WinHttpDispatchHandler to use a client certificate by calling the setter method clientCertificateFilename:. The file you provide must be in one of the supported formats by Microsoft and must contain bot the certificate and the private key.

In the old situation we had the certificate and the private key in two separate pem files. WinHttp cannot handle this. We used openssl to convert the two pem files into a single pfx file:

openssl pkcs12 -export -out testcert.pfx -inkey testcertrsa.pem -certfile testcert.pem

Cloudfork-AWS needs more than just handling simple HTTP GET and POST requests, the following features are also needed:

• Setting custom request headers – S3 uses custom headers for authentication and for attaching meta-data to S3 objects. We need to be able to set these headers. This feature is also required for range requests, with these requests you can download a part of a S3 object instead of downloading the entire object.
• Access to the response headers – So we can read the meta-data of S3 objects.
• Support for PUT, HEAD and DELETE requests – Also required for S3. PUT is required for storing objects and creating buckets. DELETE is required for removing objects and HEAD for getting the object meta-data without downloading the object itself.
• HTTPS support – The AWS services can be accessed via plain HTTP or via secure HTTPS. The choice is up to the client. But the release notes of the latest releases of SimpleDB mention that HTTP support will be deprecated and that future versions will require HTTPS.
• HTTP/1.1 support – Not a must have feature but version 1.1 requests can be more efficient than version 1.0 requests because of the keep-alive feature of version 1.1. With this feature socket connections can be reused between requests.
• Streaming uploads and downloads – Also not a must have feature for most use cases. Only when large s3 objects need to be handled.
• Proxy support – Not a requirement of one of the AWS services but a feature that is often required by client configurations.

Note that most of these features are only required for S3 and not for SQS and SimpleDB. SQS and SimpleDB only use GET and POST requests and the authentication is done in the URL and not through HTTP header fields. The HTTP responses of SQS and SimpleDB always contain XML and the maximum size is about 8KB for SQS and 1MB for a SimpleDB resultset so streaming support is not required.

As far as I know there are three HTTP clients available for Squeak:

• The HTTPSocket class – This class is part of the Network-Protocols package and is part of the standard images of the latest Squeak and Pharo versions.
• SWHTTPClient – This is an extensive HTTP client library. It was originally developed for Dolphin Smalltalk and was ported to Squeak. The latest release is not fully compatible with the latest Squeak release. There are a number of class extension collisions.
• CurlPlugin – This is a Squeak plugin that uses the libcurl C library, libcurl is a well-known and powerful open source “URL transfer library” with support for HTTP, FTP and many other protocols.

HTTPSocket

This is a very simple implementation of a HTTP client in a single class. HTTP GET and POST requests are supported, access to the headers is also possible and simple proxy configurations are also supported. HTTP version 1.1 is not supported, HTTPS is also not possible.

The current version of Cloudfork-AWS does not work with HTTPSocket as a HTTP client. With the provided functionality it should be possible to support the SQS and SimpleDB API’s. But when I use HTTPSocket I get an AWS error telling me that the signature calculated is wrong. I think this is because HTTPSocket always adds the port number to the host header field. Cloudfork doesn’t do this when it calculates the signature so you get a mismatch. It is on my todo list to fix this.

SWHTTPClient

SWHTTPClient is a full featured HTTP client library. It supports HTTP/1.1, access to the header fields and the PUT, HEAD and DELETE methods. Streaming uploads and downloads are also possible. The one thing that is not supported or that I couldn’t get working is HTTPS. Perhaps it’s possible to get this working by plugging in the Cryptography package but I have no idea how.

Another issue is that SWHTTPClient is not fully compatible with the latest Squeak and Pharo releases. The package contains some class extensions that override exiting methods with different behavior. For example the String>>subStrings: method.

Cloudfork-AWS can use SWHTTPClient, all AWS features work except HTTPS. I have fixed all the incompatibilities I bumped into. The patched version of SWHTTPClient is available from the Cloudfork project page on SqueakSource.

CurlPlugin

The installation of this library is a bit more work. You need the place the correct binaries for your platform in the Squeak installation directory and load the CurlPlugin package from SqueakSource. If you load the package you may get a warning that the class CurlPlugin cannot be loaded. This is no problem, you can still use the plugin through the Curl class. The CurlPlugin class is only needed if you want to create a new version of the plugin or support a new platform.

The libcurl library that the CurlPlugin uses supports all the HTTP features we need and many more. It is one of the bests HTTP client libraries around. And it’s open source. It has an optional integration with openssl which provides the functions required for HTTPS.

The current version of the CurlPlugin doesn’t expose all the features of libcurl. Currently HEAD and DELETE requests are not supported. It is also not yet possible to set the header fields for a requests. The other methods work very well and HTTPS also works fine.

Cloudfork-AWS

For the SimpleDB and SQS services the CurlPlugin is the best HTTP client. All the required features are there and the performance is very good. SimpleDB and SQS also work with the SWHTTPClient, only without HTTPS support. If the Curl class is present in your image Cloudfork-AWS will use this class for all SimpleDB and SQS service calls, otherwise the SWHTTPClient is used.

The current CurlPlugin doesn’t support all the features required by the S3 service. For this reason the Cloudfork S3 functionality requires the SWHTTPClient.

Future work

I think the CurlPlugin has the potential to become a very good HTTP client library for Squeak and Pharo. It will also be relatively easy to maintain this library because all of the complex work of supporting the different protocols is implemented in libcurl. This C library has a very large community and is well maintained. I will try to extend the plugin and add the missing features.

I will also try to make Cloudfork-AWS compatible with HTTPSocket. This will not be the best performing solution but it can be an easy starting point.

In Amazon SimpleDB there is no concept of relations ; it is a simple storage of items having attributes (key-value pairs). The Cloudfork-ActiveItem framework can map these relations to foreignkey-like attributes but that should be used with care. Because SimpleDB is not a relational database, operations such as Joins are simply not possible. However, mapping the composition relation fits much better in the SimpleDB storage model. The notion of a SimpleDB item being a container of information is just what it is meant to be.

To illustrate how ActiveItem supports this design construct, I will give an example that models multiple-choice questions for an exam training application. A Question is a composition of 4 Choices ; one of them is the correct answer to that question.

Question class>>describe: aQuestion

  aQuestion
    hasString: #code ;
    hasText: #text ;
    ownsMany: #choices

Choice class>>describe: aChoice

  aChoice
    hasText: #text ;
    hasBoolean: #isAnswer

When saving a Question, ActiveItem will create one SimpleDB item with both the attributes of the Question and the attributes of each Choice:

code -> '010-0001'
text -> 'What language is best for writing Web applications?'
choices.1.text -> 'Java'
choices.1.isAnswer -> 'false'
choices.2.text -> 'Ruby'
choices.2.isAnswer -> 'false'
choices.3.text -> 'PHP'
choices.3.isAnswer -> 'false'
choices.4.text -> 'Smalltalk'
choices.4.isAnswer -> 'true'

By supporting composition, class Choice can be a normal ActiveItem subclass with its own attribute description. However, as you can see in the example, Choice objects do not need an id (actually the collection index is the id).

Because of limitations to the number of attributes per item (currently 256), this composition solution is not suitable for arbitrary large collections. If you expect this for your model then I suggest you use the normal hasMany: method that maps associations using “foreign-key” attributes.

So systems that make use of AWS must have the correct time and also the timezone must be correct. Otherwise the conversion to UTC will give the wrong result. A few days ago this all worked perfectly in VA Smalltalk, but tonight all AWS calls fail Last night we in The Netherlands switched to Daylight saving time (DST). VA Smalltalk doesn’t seem to handle this very well. A call to “DateAndTime now” still returns an offset from UTC as one hour instead of two. It seems that this is a known problem.

Until this problem is fixed we have to use a less than elegant solution to get things working again. We have added a “DSTMode” flag, when this flag is true we subtract an extra hour when converting to UTC. You can enable this mode by executing:


CFPlatformServiceVASTUtils enableDSTMode: true


Support for the DSTMode was built into Cloudfork version jvds.79.

SimpleDB
In short, SimpleDB items are stored in a domain which has a name. Each item has a name and a collection of attributes. Each attribute has a name and one or more values. Values can be String only ; the application must take care of conversion. SimpleDB provides a database in the cloud that supports large volumes of data that can be accessed anywhere on the Internet. Amazon Web Services (AWS) takes care of high availability, consistency, indexing and performance.

Before throwing away your current persistency, it is important to realize that SimpleDB is not a relational database. It does not provide relational consistency (constraints), does not have a schema and “records” contain String values only. Queries on items require expressions that have limited operators; no joins, no subselects (see documentation).

Use case
Besides being a simple object database, the service can be used to store reference data (e.g. zipcode tables, gps locations, currencies) or logging information (audits). Another example is storing social user profiles which typically have a variable set of properties. SimpleDB can also be used for storing metadata and references to S3 objects such as images,video and documents. Because a S3 object can contain any data, it can be used to store large attribute values that do not fit into a SimpleDB item.

Smalltalk
After subscribing to the SimpleDB services with your AWS account, you must create the credentials object:

awsCredentials := CFAWSCredentials
  newWith: '<your access key>'
  andSecret: '<your secret access key>'.

Create a SimpleDB domain
To store items we need a domain, let us create one. Every api call returns a CFAWSResponse which must be checked for errors before using its result.

sdb := CFSimpleDB newWith: awsCredentials.
response := sdb createDomain: 'zipcodes'.
response isError
  ifFalse:[domain := response result]

Add items
The variable “domain” will be an instance of CFSimpleDBDomain that has various methods to access its items. For convenience, class CFSimpleDBItem can be used to encapsulate the name of the item and its attributes (name,value pairs).

item := CFSimpleDBItem newNamed: '3768GX'.
item valueAt: 'city' put: 'Soest'.
item valueAt: 'country' put: 'Netherlands'.
domain putItem: item.

Query items using expressions
AWS SimpleDB offers two sets of api calls that support criteria-based retrieval. You can use the query syntax:

domain query: '[''country'' = ''Netherlands'']'.

Or the select expressions:

sdb select: 'select city from zipcodes'.

More details on quering using “select” and “query” can be found at the AWS SimpleDB documentation.

Query items using Block
Cloudfork has classes that support the use of normal Smalltalk blocks to define the select condition. See the documentation for all possible operators and functions and class CFSSWOperand for the Smalltalk counterpart.

domain selectAllWhere: [:each | each country = 'Netherlands'].

Delete a SimpleDB domain
Deleting a domain will also delete all its items. No warning here.

sdb deleteDomain: 'zipcodes'.

This post showed you the basic usage of the Cloudfork SimpleDB services. Browse the classes in this package to see how other SimpleDB API services are mapped to Smalltalk messages. Also have a look at the CFSimpleDBEmulator which can used for Unit testing classes that use Cloudfork-AWS SimpleDB.

If you are planning to use SimpleDB as an object database, then have a look at Cloudfork-ActiveItem. It is a framework that can help in mapping your objects to SimpleDB items and takes care of String conversions, data sharding and can handle associations similar to Rails ActiveRecord.