Cloud Integration – One Good Start

Back in the early 2000s, many businesses offered web hosting or ran their services straight from bare-metal servers. The “cloud” only gained real traction within the last 5 to 10 years, and plenty of those older businesses are still around today.

What did it take (and in many cases still takes) to provide services from a colocated or on-premises bare-metal server?

• Hardware first. The physical server had to be bought before anything else, and enclosure size requirements sometimes ruled out consumer-grade hardware.
• Manual OS installation. The operating system had to be installed by hand from installation media.
• Tedious configuration. Setup was a slow process, and many details kept getting fixed in the days after going live.
• Hard-to-handle problems. Unplanned issues, failures, and upgrades (hardware and software) were difficult to deal with and sometimes meant taking the whole server down. Meeting any SLAs was often a matter of luck.

Despite all this, things often worked just fine.

The cloud introduced a different paradigm by fully separating the service from the hardware it runs on. Servers could be created on demand, and uptime could be preserved through the platform’s built-in failover features. Still, it wasn’t all smooth sailing: service performance was often much slower than the bare-metal equivalent. That leads to the problem at hand: how do you integrate bare-metal servers, colocated or on-premises, with cloud services?

First, why supplement physical servers with cloud services at all? Going back to the list above:

• OS installation: It may not be fully automated in every scenario, but you can keep versioned images that bring up an operating system with minimal intervention.
• Configuration: A tool like Chef can manage it automatically. The Chef Server can live in the cloud and connect to existing infrastructure over a VPN.
• Backups: Data backups can be stored in the cloud alongside tools that restore anything, at any time, with minimal human involvement.

These changes can mean quieter nights, far less manual work, and SLAs that become predictable.

The AWS services relevant to this scenario are S3, EC2, CodeCommit, and OpsWorks.

Backups can be stored in one or more S3 buckets. Options include:

• A specialized tool such as Amanda, which integrates well with AWS.
• A classic approach combining tar and find to handle incrementals.
• Mirroring data into a versioned S3 bucket.

There are two options:

• OpsWorks, where Amazon sets up a Chef Server for you. This is simple, but monthly costs can climb quickly.
• A standard EC2 instance, where you can install a Chef Server yourself. Even a t2.micro from the free tier will do.

Whichever you choose, the Chef Server must be connected to the on-premises or colocated servers through a VPN solution (for example, OpenVPN).

Development files (cookbooks, node attributes, data bags) can be stored easily in AWS CodeCommit, a git repository that should cost nothing under typical usage. Alternatively, you can host a git repository on a separate EC2 instance, distinct from the Chef Server, to avoid a single point of failure.

This is only the beginning of the journey to the cloud. Thanks for reading!