Smalltalk in The Cloud

The Cloudfork-AWS project makes it easy to use the Amazon Simple Queue Service (SQS) from Smalltalk. SQS is a highly scalable and simple to use messaging system. Basically it lets you send and receive messages via one or more queues you define. It is available via a REST and a SOAP based HTTP interface. Cloudfork contains the CFSimpleQueueService class that makes the generic SQS calls available as Smalltalk methods. Calls that are related to a queue are implemented in the CFSimpleQueue class.

It is important to remember that SQS is not a transactional messaging system. This means that messages you send are immediately available on the queue without requiring a commit. For receiving messages SQS uses the concept of a Visibility Timeout. The timeout value in seconds is the amount of time you have to processes the message after the receive. During this time the message is invisible to other clients that receive messages from the queue. If you don’t delete the message before the timeout expires the message will become visible to other clients. So you should think carefully about how to set the visibility timeout or better: design your system in such a way that the occasional occurrence that a message is processed more than once doesn’t cause serious problems.

Use case
There are many different situations in which a messaging system can improve the scalability flexibility and or robustness of a system. As an example think of a web application that needs to send out emails with a PDF report as an attachment. You can generate the PDF and send the mail directly from your web application. But if creating the report involves a lot of data processing this will seriously limit the scalability of the web application. In these situations it is better to send out a CreateAndMailReport message and let some other process handle the processing. You can start with a single create report process, if your website becomes popular you can easily add as many report processes on as many servers as you need without changing a single line of code.

With the cloud computing facilities of Amazon you can completely automate this process. Create an EC2 image which runs one or more create report processes and create a monitor process that checks the number of messages in the queue. If the number of messages in the queue increases this monitor process can startup extra EC2 report instances. If the queue is empty the monitor process can stop the EC2 instances. I use exactly this construction for transcoding media files (audio and video). Sometimes I need to process a lot of media files in a short period of time. I use a maximum of eight EC2 images to handle these peaks. Most of the time I have just one transcoding EC2 instance running during working hours and zero instances during the night.

Smalltalk
Ok, that was a short intro of SQS, Amazon provides excellent documentation where you can find all the details. Now let’s turn to some Smalltalk code. The assumption is that you have an AWS account and that account is subscribed to the SQS service.

Sending messages
Messages in SQS must be Strings with a maximum size of 8K. The method below creates a new queue and sends one message to this queue. If the queue already exists then the createQueue method does nothing.

sendMessage: sqs

	| name qurl |
	
	name := 'cloudfork-example-q'.

	qurl := (sqs createQueue: name) result.
	queue := sqs openQueue: qurl.
	
	queue sendMessage: 'Cloudfork says Hi!'

Receiving messages
The code below shows how a receive loop can be implemented. This loop will run forever and call handleMessage: for all messages received. With SQS you have to use polling to get new messages. If no message is available nil is returned as the result. In this case the process sleeps for 30 seconds before trying again. This loop should be run as a background process otherwise it will lock up the user interface.

startReceiveLoopFor: aQueue

	| message response |
	
	[ response := aQueue receiveMessage.
	response isError
		ifTrue: [ self error: 'error receiving message']
		ifFalse: [
			message := response result.
			message isNil
				ifTrue: [
					Delay forSeconds: 30 ]
				ifFalse: [
					[ self handleMessage: message ] 
						ensure: [ aQueue deleteMessage: message ]]]] repeat

Note that the error handling is very basic. The loop will keep running if an error occurs during the handling of a message. But the loop will end when the receiveMessage method returns an error.

This entry was posted on February 5, 2009 at 22:25 and is filed under Cloudfork. You can subscribe via RSS 2.0 feed to this post's comments.

Tags: aws, sqs, squeak