A Lesson on Failure: “Going Cloud”

About a year ago, when I knew very little about Cloud Computing or Amazon Web Services, I was assigned, along with a few colleagues from the consulting company I worked for, to take an existing codebase from “alpha” to “production” and make sure it deployed smoothly to the Amazon cloud.

The customer wanted to go live in under three months and expected to handle tens of thousands of visitors clicking on banner ads and generating revenue. More likely, they were hoping for a good exit: handing the hot potato to someone else while walking away with a tidy profit. There is a term for these people, but this isn’t a meme post, so I’ll leave it there.

Things started off reasonably well. We worked through a long list of functional issues and ended up fixing more than 100 bugs and glitches. That was about all we could do, and it took us many long hours to get there.

We didn’t concern ourselves with the cloud configuration. We knew almost nothing about it, and the customer guarded the “keys to the kingdom” closely, approving instance and resource setups only case by case. They probably felt they were overpaying their Eastern European contractors (us), so I understand why they watched the Amazon bill so carefully. It was their home, their needs and their rules, and that was fine with us.

Overall, things went well from our side until performance testing began, but after that, everything fell apart. Within about a month, the project was taken away from us and then passed to different contractors, although I don’t think the product ever launched. These days the website domain is one of countless spam domains, abandoned and possibly resold a few times already.

What did the programmers and QA engineers see when things started going wrong?

The performance tests were returning poor results. We had a “performance environment” that closely mirrored production configuration, so we expected test results to reflect real-world behavior.

The metric falling short was Queries Per Second (QPS), a target previously agreed with the customer in our contract. Qualitatively, pages loaded slowly and timeouts were frequent. Simply navigating the site was painful because it was so sluggish and unpredictable.

We were better software engineers than lawyers (oh, the irony), so all we could do was try to improve the software stack:

• Backend: The framework was Python Tornado, so we converted all “sync” operations to “async”. We were hunting for outliers that might be blocking the main server thread, but never properly identified any. The result was inconclusive: some tasks improved, others got worse.
• Database: The database was MongoDB, so we spent a lot of time optimizing queries and removing unneeded indexes. Again, there was no meaningful improvement.

By the time the customer walked away, all of us were exhausted and frustrated.

It took me about a year to understand what actually happened. It wasn’t the software stack, but the customer’s “Cloud” (AWS) setup. I only reached this conclusion after working with AWS day to day and while preparing for the AWS certification I’m currently pursuing.

In short, the customer relied heavily on the AWS free tier. They used t2.micro instances for development and testing, and most likely t2.small or t2.large for the Mongo database. I don’t remember the production web instance setup, but we probably had no proper load balancer. Load may have been distributed through DNS round robin: each instance had an elastic IP and Route 53 handled the balancing. Either way, all frontend instances used a single database node as their backend.

Here is the detailed explanation:

• Burstable “T” instances: These are marketed as general purpose. Their baseline performance is limited, and extra capacity comes from a credit system: credits accrue while the instance is idle and are spent during load spikes. This works well for many non-production scenarios, but not when sustainable performance is required.
• EBS storage: When creating instances with default storage settings (magnetic or general purpose), the volumes use the same credit model. During normal operation, Amazon throttles baseline IOPS (input/output operations per second) to a value well below what the underlying storage could deliver (details).
• Credit exhaustion: Our performance tests burned through all accumulated EBS credits within the first few minutes, capping storage performance at a low baseline. On the database instance this appeared as high IO wait (the wa field in top). With our limited knowledge, we blamed poor application code rather than something external, i.e. the AWS-imposed limit.
• MongoDB’s access pattern: MongoDB has an unpredictable storage access pattern by design, so there was likely never any burst capacity (idle-time credits) available in the first place. We often saw sluggish performance during normal daily work with no obvious cause.

Two things might have helped:

1. MongoDB sharding on every frontend instance. This could have spread I/O across the whole cluster and likely mitigated the single-point-of-failure database design. On the downside, it might have caused memory swapping on t2.micro instances, cancelling out any benefit.
2. Provisioned IOPS for EBS storage. A dedicated DevOps engineer, which we didn’t have, could have monitored the “average queue length” metric and recreated the volume with the right provisioning to balance performance and cost.

Neither happened. Sharding was only mentioned when we were already in a downward spiral and it was too late to do anything meaningful.

When moving to the cloud, put a Cloud Architect on the team. It may be the single biggest difference between success and failure. Also, a cloud provider’s free tier is free for a reason, and that reason may be less obvious than what’s written in the marketing material.

Update: Two days after writing this post, I passed the AWS Certified Solutions Architect (Associate) exam. Please always work with an expert when setting up production environments.