Cloud Track Record

Products in the Cloud

SmartaTech developed its own product, URLPoke, running in the AWS (Amazon) cloud. It ran on three-plus servers: web (rails), database (mysql) and worker-bee (groovy).

We have experience setting up this product using the AWS API in such a way that the number of workers scales up and down depending on the load experienced. The method used to deploy the system in the first version was a combination of the AWS API and bash scripts. It was quite a tricky thing to set up, but many lessons were learnt along the way. It was running successfully for several years in the cloud, with over a hundred users, before being discontinued.

We saw this as a good way to experiment and learn about how to manipulate applications in the Cloud, and it really only emphasizes how accessible this facility is for rolling your own deployments.

Lessons learnt

• End to end thinking. Although we used three different technology bases (Rails, MySql, Groovy/Java) to write this application, having to deploy it all yourself onto remote infrastructure in a repeatable manner does focus the mind, and the discipline of configuration management.
  
• Scaling works. However much you read about "the Cloud is great for scaling up and down use of server resources", it is only becomes real when you try it yourself. Seeing the server usage adapt to load (where load was derived from AWS's monitoring of CPU usage) was a great turning point.
  
• Even simple programs are complicated to deploy. The stark truth is that there is a lot to think about when pushing applications onto infrastructure. In version 1 we did not try to deploy onto totally vanilla OSes: we utilized images built previously, adding on application-specific configuration. Even this is not trivial. 
  

Cloud Orchestration

For version 2 of URLPoke we wanted to improve on the largely hand-crafted and bash-script driven approach to using the AWS Cloud.

Therefore we iimplemented automated deployment using Chef as the orchestration engine for deploying the application to base OSs rather than pre-crafted images.

The Chef approach (scripts based on the Ruby language), with its rich library of cookbooks (e.g. for installing Apache, MySql, RVM and so on) is a truly great thing.

Version 2 ran for two years, and using the Chef server allowed us to upgrade individual nodes (e.g. the MySql box is a node) with new versions of software when required. Being in the Cloud, we could specify that this is a staging version, fire up a few staging servers and test it in the Cloud before pushing live.

Lessons learnt

• Chef is Great. It was not easy to get our heads around Chef, but once you have done that you have a powerful tool and skill. This is something that everyone needs to learn!
• There is a lot to configure. There really is a lot to ensure you have installed and configured if you are working from a base OS every time. It si excellent discipline to have all these elements under control and understood.
  
• DevOps is Powerful. The bottom line is that development and infrastructure have to go together. The artificial separation (caused by company IT departments getting very big and the need for management structures) has removed the end-to-end thinking that every developer needs to have, about how the application will be deployed. Doing it yourself, in a Cloud environment, and having to make deployments repeatable and fully configured (ie no black magic once the machines have been built) and parameterized means that they are much more reliable.