Thoughts from the DevOps world

Down the Rabbit Hole: Modern Image Formats

2025-07-20T15:42:34+00:00

While browsing articles about website optimization, I came across a tip to use a modern image format like WebP. The idea is that WebP files are smaller, so pages load faster. At the time, I didn’t really understand why that would work, so I started digging into it, and I ended up going down a rabbit hole.

1. Introduction

Some background: the well-known image formats JPEG, PNG and GIF have been around for 25 years or more. They were designed when storage and processing power were expensive, so each one was built for a specific purpose. JPEG became the standard for everyday photos, PNG for icons and computer-generated images, and GIF for short animations (once the patent on its compression algorithm expired in the early 2000s).

By the early 2010s, computing had become cheap, and older ideas for better image compression finally became practical. New standards were also introduced, mostly without people noticing. Engineers also began to use hardware built into phones and other portable devices to speed up encoding and decoding of images and video. The result was a confusing patchwork of new image and video formats.

In the early 2020s, browsers chose which of these formats to support and which to ignore for the time being. Operating systems and image software, meanwhile, supported whatever suited their own users. HEIC/HEIF is a good example. It is the default image format on Apple devices, which have dedicated hardware to handle it. On other systems it can require paid plugins, or may not work at all. So most apps that display images have to quietly convert whatever users upload into a more widely supported format.

Today, only two modern formats have broad browser support: WebP and AVIF. Both aim to replace all the older formats, offering good quality at small file sizes for every kind of image, from photos to drawings, icons and short animations. AVIF goes a step further by also supporting HDR (high dynamic range).

To sketch out some context, the well-known imge formats such as “jpeg”, “png” or “gif” are with us for 25 years or more; they were put together in a time when both storage and cpu processing power were premiums rather than commodities. That meant accepting compromises and precise product-market fits: “jpeg” became the universal format for everyday pictures, “png” for icons or computer generated images and “gif” was mostly delegated to short animated sequences once the patent covering the corresponding compression algorithm expired in the early 2000s.

2. Exploring New Image Formats

Setting the background aside, using these formats on websites and web apps is fairly simple. You can start by saving your image in the new format (instead of the usual PNG or JPEG) from your image editing software. After that, you upload the pictures and add <img> tags as you normally would.

There are a few catches, though:

Tool support can be patchy. Your image editor may not be able to save WebP or AVIF files at all, or it may save them without the best balance of quality and file size.
Not every browser supports them. Depending on your audience, some users may be on browsers that don’t support AVIF, or browsers that only support WebP. Sticking with the older, reliable formats is always a valid choice if it suits your use case better.
Wider support means more storage. To cover everyone, you need to keep at least two copies of each image: one in the modern format and one in an older, widely supported format. Your users get a better experience, but you pay for the extra storage.

3. Implementation in Web Context

In practice, there are a few ways to support a wide audience:

Use the HTML5 <picture> tag with an <img> fallback. The browser picks the first format it supports, and the <img> tag points to the widely supported format as a fallback. The <picture> tag can also choose an image to suit the user’s screen size. The catch is browser compatibility: some browsers may load both the new and old image, which would make page loading slower instead of faster.
```
<picture>
  <source srcset="flowers.webp" type="image/webp">
  <img src="flowers.jpg" alt="Flowers">
</picture>
```
Use a JavaScript fallback with an onerror handler. The <img> tag tries the new format first. If that fails, the handler loads the older format instead. The right image does eventually appear, but the experience will probably be sluggish on older browsers.
```
<img src="flowers.webp" alt="Flowers" onerror="this.onerror=null;this.src='flowers.jpg';">
```
Rewrite the request on the server, based on the browser’s headers. This works in almost all situations: the browser always receives a format it can display, and there’s no risk of it loading two copies of the image. The only downside is that if a user saves the image, the file extension may not match its real format.
```
# Frontend:
<img src="flowers.jpg">

# Backend .htaccess file for Apache httpd:
RewriteEngine On
RewriteCond %{HTTP_ACCEPT} image/webp
RewriteRule "^flowers.jpg" "flowers.webp" [L]

# Nginx configuration:
if ($http_accept ~* "image/webp") {
    rewrite ^flowers.jpg flowers.webp last;
}
```

Generate images dynamically in the right format. For this you need to check both the browser’s Accept header and whether your server can produce the format. The result isn’t always a smaller file, depending on the image content, so test it before committing to this approach. Here is a PHP example using GD:

$webp_available = ((imagetypes() & IMG_WEBP) > 0)
    && is_array($_SERVER)
    && array_key_exists('HTTP_ACCEPT', $_SERVER)
    && str_contains($_SERVER['HTTP_ACCEPT'], 'image/webp');

header("Content-Type: image/" . ($webp_available ? 'webp' : 'png'));
header("Cache-Control: no-store, no-cache, must-revalidate, max-age=0");
header("Pragma: no-cache");

if ($webp_available) {
    ImageWEBP($image, null, 100);
} else {
    ImagePNG($image);
}

4. Server Side: Heavy Lifting

If your image editor doesn’t make good use of the new formats, or saving multiple copies of every picture becomes a chore, giving up can seem like the natural choice. There is help available on the server side, though, depending on your platform:

Content management systems often have plugins that do this for you. They create the extra copies automatically. Plugins like this are readily available for WordPress, for example.

You can convert images in bulk with a few Linux shell commands:

# Install the WebP and AVIF command-line tools (run as root)
dnf install libwebp-tools libavif-tools

# Create WebP copies of all PNG images in the current directory
find . -maxdepth 1 -type f -name "*.png" -exec sh -c 'cwebp -lossless "$0" -o "${0%.png}.webp"' {} \;

# Create AVIF copies of all JPEG images, including subdirectories
find . -type f \( -name "*.jpg" -o -name "*.jpeg" \) -exec sh -c 'avifenc "$0" "${0%.*}.avif"' {} \;

5. Conclusion

This exploration was an eye-opener, for me at least. My personal view is that for the average website, adopting a new image format brings only a small, not very consequential improvement. There are other use cases where it matters more, though:

CDN services (content delivery networks) can convert images transparently, and at their scale the bandwidth savings add up.
Photo-sharing sites and social media platforms can use AVIF for its HDR support, giving better-looking images to users with compatible screens, such as recent Apple devices or high-end desktop monitors.

Migrating from Python 2 to Python 3

2018-05-13T19:39:46+00:00

Modern Python has powerful features such as asynchronous programming (using async and await) and memory-efficient data processing with generators and iterators. These capabilities, found mainly in Python 3, can noticeably improve both performance and maintainability. Taking advantage of them often means migrating existing Python 2 projects, and the effort involved ranges from a simple conversion to a major refactor. To help you plan, here are some of the most important differences between the two versions.

1. Print

This one is easy. Most of the time it’s just a matter of changing:

print "abcd"

to:

print("abcd")

2. dict.keys() and dict.items()

This works in Python 2:

x = {1:2, 3:4, 5:6}
k = x.keys()
z = k[0:2]
# [1, 3]

In Python 3, keys() and items() return view objects that can’t be indexed or sliced, so k[0:2] raises an error. A quick (though not optimal) workaround is to wrap the result in list():

x = {1:2, 3:4, 5:6}
k = list(x.keys())
z = k[0:2]
# [1, 3]

3. range() and xrange()

In Python 2, xrange() is a lazy sequence generator, typically used in for loops, while range() builds a full list:

a = [i for i in xrange(23)]
# [0, 1, ... , 22]
b = range(23)
# [0, 1, ... , 22]

In Python 3, range() takes over the role of xrange(), and the old list-returning range() no longer exists. Where you need an actual list, a list comprehension (or list(range(...))) will do the job.

4. Exceptions

Exception handling in Python 2 looks cluttered:

try:
    raise IOError, "file error";
except IOError, err:
    print err

Python 3 is much cleaner:

try:
    raise IOError("file error")
except IOError as err:
    print(err)

5. Rounding and Division

These changes can break programs in surprising ways:

a = round(122.5)
b = 7/5

# Python 2:
# a = 123, b = 1

# Python 3:
# a = 122, b = 1.4

In short, Python 2 inspects the operand types and performs integer or floating-point division accordingly. Python 3 is more predictable: / always performs floating-point division.

Rounding is harder to reason about. Python 2 rounds halves away from zero, whereas Python 3 rounds halves to the nearest even number (banker’s rounding), which is why round(122.5) gives 122.

There are many more differences. For example, if your code uses urllib, every call site must be updated to the new module structure. On a more encouraging note, an automated tool can handle much of this tedious work: 2to3.

Hero Culture is Bad

2018-01-28T13:23:32+00:00

Years ago, in the country where I grew up and back when I still watched TV, a pattern kept repeating: TV stars would take their shows to rival stations, only to find themselves in a worse position a few months later.

They were often offered better contracts and sometimes even managed to bring part of their original crew along. Yet they rarely held on to their audiences for more than a short while, and some were pushed out of television altogether.

Tabloids occasionally asked how this could possibly happen, but never offered real answers (not that they had much insight to give). I doubt their readers could have made sense of it either.

I couldn’t make sense of it back then, either. I’d heard the more thoughtful explanations, such as “the shows were simply better before the move” or “some stations have more money and can afford not to cut corners.” But that’s only the surface. It took changing jobs a couple of times myself to understand what success really depends on. In short:

It isn’t about having some special talent.
It isn’t about single-handedly pulling the company uphill.
It’s about the entire environment that makes you succeed or fail, starting with how clean the toilets are at your workplace.

How does the environment make or break people?

The most basic example is education: whether you get a good one depends on your family, neighborhood, city and society as a whole. You could argue that business is different, since people are selected through interviews rather than born into their position. I’d argue the same constraints apply everywhere: location, money and the human factor.

Here’s an example from one of my earlier jobs. At the time, the only way to show slides in a meeting was with a projector. It wasn’t easy: you had to get it from IT, wrestle with the setup, and only then start your actual work. Afterwards you had to reverse the whole process so someone else could use it in another room. Finance considered this the optimal approach, because fitting every meeting room with a TV or a fixed projector would have cost far more.

That money-saving strategy had two problems:

The setup was cumbersome. Booking the projector, collecting it from IT, configuring it and returning it ate up 20 minutes or more of a meeting that lasted an hour at most. A real bottom-line win, I’d say.
The projectors wore out. Their heating and cooling cycles were incompatible with this process, so within a few months the lenses degraded, the image lost its sharpness and the devices were no longer fit for purpose. Another bottom-line win.

Challenges like these create workplace heroes: people who seem to succeed against everything thrown at them, but who aren’t really winning. They’re survivors, the ones who appear to float above the water.

After all these years, my conclusion is that you can’t simply transplant successful people into such environments and expect them to keep succeeding. Even if they become local heroes, their real impact will be severely limited.

Do heroes always win?

The workplace I mentioned had a hero culture by any definition. A few engineers worked their tails off, fighting company procedures, incompetence and, ultimately, sheer craziness. But good code alone doesn’t get you anywhere, as they found out the hard way. Such an environment simply isn’t conducive to good performance.

These heroes eventually thrived professionally once they landed in genuinely good environments, much like the TV stations I described at the start, the ones that kept drawing huge audiences whether or not particular anchors were on screen.

So what was the secret of the successful stations? They had hero-proofed their work environments.

What does a hero-proof workplace look like?

Culture helps to a degree: successful companies today tend to have flat structures and open-door policies. That’s the visible part. The less visible part is that in such companies, nobody can succeed alone, and everybody knows it.

And yes, it really does start with clean toilets. Too few bathrooms, or mostly dirty ones, are among the most effective mood destroyers in any organization. It works like Maslow’s hierarchy of needs: the basics must be in place before anything else, and the quality of your training programs only starts to matter much later. The stations that kept their audiences after “losing” certain people were simply better at covering the basics. In essence, they:

ran a better hiring process for support staff;
put the right tools in the right hands;
held everyone to a high standard, with no exceptions.

By offering audiences a better product, they could charge advertisers more, spend more time perfecting details and cut fewer corners than their competitors. They kept a virtuous cycle going. Yes, it really is that simple.

Using Linux Network Namespaces

2018-01-20T19:55:57+00:00

You rarely get to work with Linux namespaces directly. Usually they sit behind the scenes in container tooling or hosting platforms that enforce hard resource limits for customers. But sometimes containers are too heavy for the job. I once needed two instances of the same service to talk to each other over the network.

Why not just have them listen on different addresses or ports? Because it isn’t always that simple. If you can’t change the code and only want to replicate a certain behavior, there may be no way to make an instance use a different port. If the protocol also relies on broadcasting, it gets harder still, since you often don’t have two IP addresses on separate interfaces attached to the same network. Virtualization or containers would solve this, but they’re overkill here. A lightweight alternative is to manipulate Linux network namespaces.

Note: Run all commands below as root.

1. Unshare

Open two terminals. Call one the “primary” and the other the “secondary”, and keep track of which is which. In the secondary, run:

# unshare -n /bin/bash
# echo $$
xxxxxx

Note the PID that is printed, since you need it in the next step.

2. Set Up the Network

In the primary, create a virtual Ethernet pair and configure one end, using the PID from step 1:

# ip link add name veth0 type veth peer name veth1 netns xxxxxx
# ip address add 192.168.10.10/24 dev veth0
# ip link set veth0 up

Then, in the secondary, configure the other end and test the link:

# ip address add 192.168.10.15/24 dev veth1
# ip link set veth1 up
# ping 192.168.10.10
...

3. Set Up the Services

I won’t go into detail here, since the goal was only to get a working communication path. It looks easy, and it is. Hope it helps!

After 6 months at Googamaz...book

2017-10-15T21:32:07+00:00

Some intense months later, I’m back with fresh ideas and stories to share. Secrets follow. Well, not really, but I’ve put everything in a list to make it easier to digest.

1. Getting in is hard, staying in seems even harder

Once the commercials are over, the movie begins: the interview and then the orientation don’t prepare you for the environment you’ll be swimming in when the daily work starts. On my path to enlightenment I realized that the kitchen staff, the people who cook that good food for you three times a day, are likely evaluated on par with the people they feed. At the end of the day, nobody gets a free meal.

I also found out that the photos floating around the internet, with colorful interiors, pool tables, arcades and pinball machines straight out of the ’80s, are real. I also learned that an unhealthy balance between productivity and time spent having fun in the games room or napping in the library will land you on a PIP (performance improvement plan) before your first anniversary. Failing it is obviously your ticket out, but some people choose to leave on their own terms. For the company the outcome is the same, though: a (possibly lifetime) “no rehire” flag in your file.

From the outside, people leave or are let go, and it’s impossible to tell the two apart. There seem to be “sweet spots” for leaving at certain intervals, like just before the 6-month mark for people who’d rather not pay too much tax to some country or, who knows, maybe it’s just a quick divorce. Another sweet spot seems to be around 1 year, when the first batch of shares vests and the payout can be significant. Seeing many people stay beyond their 5th year or so is encouraging, though.

2. Recruiters tend to avoid you

Changing my position on my LinkedIn profile made the profile-visits counter drop by more than 30%, while InMails and connection requests went from nearly daily to maybe once every few weeks. Simply put, very few companies can offer compensation like the big ones everybody talks about. The big companies also run their own recruiting pipelines and don’t work with third parties.

On a side note, some recruiters simply cold-mail every candidate in their internal database who matches certain keywords, without checking whether those people are still on the market for that kind of role. I don’t have figures to tell whether that strategy actually works for them, but I now ignore everything from recruiters with less than 5 years of experience, no exceptions.

3. The Ulysses syndrome is real

The path from “wow, I’m so lucky” to “what the *** did I get myself into” is short, really short. In the first month or so you might consider writing angry emails to everyone who didn’t believe in you or refused to give you a position you badly wanted at some point in your career. Of course you don’t, because you realize that people, starting with you, are constantly evolving and, by definition, focused on their own needs. What others can put on the table isn’t always their top priority.

I’m kidding: you don’t do it because the honeymoon ends faster than you’d like, and you find yourself in a different country, in an environment governed by different rules, away from familiar places and intimately close to a very demanding workplace. If your family is with you, the challenge is multiplied, as they may struggle to adjust to local life too. This is actually the leading cause of people leaving companies in their first year after relocating to another country, or even across the ocean.

4. There is light at the end of the tunnel

At some point the roots start to grow and the new environment no longer feels menacing. You start meeting people and doing work that seems to make other people’s lives easier. The prospect of your first formal performance review ending in a PIP slowly fades. Maybe the whole future of humanity no longer looks so grim after all, and neither does the outlook for your career.

Private Yum Repository with Amazon S3

2017-03-20T22:26:24+00:00

Setting up a public yum repository with S3 is straightforward, but making it private takes more effort. Access control has to be handled by a plugin that fetches files from the S3 bucket through the API, typically using stored credentials. On EC2 instances with an appropriate IAM role, you can skip storing credentials, but in any other environment that isn’t an option.

Either way, the first steps are the same for public and private setups.

1. Create the repository directory structure

Use the createrepo tool, which is available on both RedHat-based and Debian-based systems (e.g. Fedora/CentOS or Ubuntu). Running it creates a repodata subdirectory containing several files with metadata parsed from your RPM packages.

2. Sync the local repository with the S3 bucket

With the AWS Command Line Interface installed, this takes a single command:

    $ aws s3 sync ~/localrepo s3://yumrepobucket/remoterepo --delete

Running this from cron is trickier, because the credentials may not be picked up correctly. Setting the AWS_CONFIG_FILE environment variable should resolve that.

3. Install a yum plugin

You could write your own plugin if you know Python, but grabbing an existing one from GitHub is much quicker.

The one that worked best for me was yum-s3-iam. Installation was simple:

Copy s3iam.py to /usr/lib/yum-plugins/
Copy s3iam.conf to /etc/yum/pluginconf.d/
Copy a customized version of s3iam.repo to /etc/yum.repos.d/

Although the documentation doesn’t mention it, you can add the key_id and secret_key parameters to the .repo file. If you’re on EC2, configuring an IAM role instead should be enough. For security, whichever identity you use (role or user, if a role isn’t possible) should have read-only access to the bucket and nothing more.

That’s it: you now have a private yum repository hosted on an S3 bucket. Enjoy!

Cloud Integration – One Good Start

2017-03-08T15:38:40+00:00

Back in the early 2000s, many businesses offered web hosting or ran their services straight from bare-metal servers. The “cloud” only gained real traction within the last 5 to 10 years, and plenty of those older businesses are still around today.

What did it take (and in many cases still takes) to provide services from a colocated or on-premises bare-metal server?

Hardware first. The physical server had to be bought before anything else, and enclosure size requirements sometimes ruled out consumer-grade hardware.
Manual OS installation. The operating system had to be installed by hand from installation media.
Tedious configuration. Setup was a slow process, and many details kept getting fixed in the days after going live.
Hard-to-handle problems. Unplanned issues, failures, and upgrades (hardware and software) were difficult to deal with and sometimes meant taking the whole server down. Meeting any SLAs was often a matter of luck.

Despite all this, things often worked just fine.

Cloud Integration

The cloud introduced a different paradigm by fully separating the service from the hardware it runs on. Servers could be created on demand, and uptime could be preserved through the platform’s built-in failover features. Still, it wasn’t all smooth sailing: service performance was often much slower than the bare-metal equivalent. That leads to the problem at hand: how do you integrate bare-metal servers, colocated or on-premises, with cloud services?

First, why supplement physical servers with cloud services at all? Going back to the list above:

OS installation: It may not be fully automated in every scenario, but you can keep versioned images that bring up an operating system with minimal intervention.
Configuration: A tool like Chef can manage it automatically. The Chef Server can live in the cloud and connect to existing infrastructure over a VPN.
Backups: Data backups can be stored in the cloud alongside tools that restore anything, at any time, with minimal human involvement.

These changes can mean quieter nights, far less manual work, and SLAs that become predictable.

Using Amazon Web Services

The AWS services relevant to this scenario are S3, EC2, CodeCommit, and OpsWorks.

1. Backups

Backups can be stored in one or more S3 buckets. Options include:

A specialized tool such as Amanda, which integrates well with AWS.
A classic approach combining tar and find to handle incrementals.
Mirroring data into a versioned S3 bucket.

2. Configuration

There are two options:

OpsWorks, where Amazon sets up a Chef Server for you. This is simple, but monthly costs can climb quickly.
A standard EC2 instance, where you can install a Chef Server yourself. Even a t2.micro from the free tier will do.

Whichever you choose, the Chef Server must be connected to the on-premises or colocated servers through a VPN solution (for example, OpenVPN).

Development files (cookbooks, node attributes, data bags) can be stored easily in AWS CodeCommit, a git repository that should cost nothing under typical usage. Alternatively, you can host a git repository on a separate EC2 instance, distinct from the Chef Server, to avoid a single point of failure.

Conclusion

This is only the beginning of the journey to the cloud. Thanks for reading!

Leaving in Anger? No.

2017-02-24T15:57:32+00:00

Suppose you ignored my advice from a few months ago and actually landed that job. That’s a real achievement, but it isn’t what I want to write about. In most cases, what you leave behind matters more than what lies ahead of you.

Your current job probably doesn’t compare with the likes of Googamazbook, and that’s fine. Or maybe you feel you didn’t accomplish everything you hoped for when you signed up for the role you’re about to leave. That’s fine too. Let’s go through some common frustrations.

1. Promotions (or the lack of them)

You feel you deserved more, and you were passed over for reasons that seemed unclear or even unprofessional. That may well be true, but it isn’t always about you as an individual. It’s about how companies work.

In the companies I’ve worked for, most employees sat somewhere in the middle of a Gaussian curve when ranked by seniority. On the left were people in their first jobs, such as recent graduates, who tended to move up quickly toward the bulk in the middle. On the right were senior managers and people with “principal” in their titles. Middle management sat in the middle too. The name says it all.

In short, there isn’t much room for natural movement if that shape is to be preserved, and it has to be preserved if the company wants to survive in the market. Add the people hired from outside, and there are always more candidates than seats at the front of the room. That’s why you need to prepare to secure a promotion: you have to look better than the others competing for the same seat. Professionally better? Probably not, but that’s how it works.

2. Raises (smaller than expected)

You think your latest 3% raise was dreadful and didn’t reflect your performance. Complaining openly about it just before you leave will fix… well, nothing. Companies operate by different rules.

From the company’s perspective, pay should reflect an employee’s market value: what no other employer would beat, plus the risk your employer is willing to accept. That figure is only loosely tied to the internal ranking described above, because some people care more about their title than their salary. Companies know this and act accordingly.

Overall, companies pay the minimum needed to keep a particular person. Normal companies are for-profit businesses and follow market rules. Public-sector organizations are different. They pay in close relation to internal rank, because individual performance almost never brings in more business, only bigger budgets.

3. Bosses (poor management quality)

Bosses are judged within the company’s internal management policy. Very few companies act on individual complaints from the minions, but all of them watch trends, statistics and overall scores gathered through anonymous surveys.

Since your direct boss is your main point of contact with the company, they end up absorbing everything positive and negative the company projects. They rarely decide how promotions and pay adjustments are handed out. They can propose changes and fill in positive feedback on forms, but as individuals they are judged on end results alone, and those results are always capped by the company’s budget.

4. Culture (meaningless words)

You may leave feeling disgusted by the company culture, but the culture itself usually isn’t the cause. Many companies paint slogans on walls and doors so that people won’t forget them.

The unspoken problem is that culture isn’t passed on by wall messages. It spreads more subtly, from managers to employees and between colleagues of different tenures. Periods of rapid expansion promote managers who aren’t properly grounded in the founders’ vision, and before long ideas like “the sole purpose of this company is to fill the owner’s bank accounts” become universally accepted. Then again, making money is one of the core purposes of any company.

5. Conclusion

If any of this sounds familiar, remember that it isn’t about one person casting spells from an ivory tower, and you weren’t singled out. And if things really don’t work out, leaving is sometimes the only option.

So why not vent your anger on the way out? Because it only upsets individuals instead of fixing procedures or redistributing budgets. You may need one of those individuals as a good reference when you pass the onsite at Googamazbook, or even to get considered for an interview in the first place. Be human, and respect the humanity of others.

FOSDEM 2017

2017-02-09T15:07:07+00:00

FOSDEM is an annual event for open source software developers, held in Brussels on the first weekend of February.

This year’s event was the 5th FOSDEM I have attended since 2010. Over that time I have watched it evolve: the talks followed the industry’s shifting focus, and some topics disappeared to make room for newer ones. Many of those newcomers never gained real traction and faded away as well.

The biggest shift I have noticed is a move away from abstract or general subjects (such as Linux Kernel discussions or C/C++ performance tricks) and towards finished products and near-mature technologies applied to achieve concrete results.

The quality of the sessions varied. Even when the slides were genuinely interesting, some speakers could have used better presentation skills, and their job title made no difference. A few good ideas were spoiled because the presenter could not keep the audience focused on the topic.

I also noticed that all the non-technical talks and discussions had been moved to the smaller “developer” and “community” rooms. Unless you grabbed a seat early in the morning, you could not get in, and the queues at the doors grew long because interest often exceeded the room capacity.

The event as a whole was busier than last year’s, continuing a trend of several years. The hallways and common areas were packed at times, and lunch took a long while, with 20-30 people queuing at every vendor.

I had also hoped the closing keynote would be interesting and innovative: a peek into the future, a look back at the previous year’s events, or achievements that made the world a better place. Unfortunately, it was a dull talk that many attendees stopped following before the end. Unsurprisingly, no questions were taken from the audience.

That was FOSDEM 2017. I expect the organizers to spread the topics over more days or introduce some form of registration, because the venue, large as it is, is reaching its limits.

XFS Corruption: Repairing

2017-01-24T18:48:57+00:00

Note: The go-to tool for XFS repairs is, unsurprisingly, xfs_repair. Getting it to run, however, can sometimes be the hard part.

Introduction

How does filesystem corruption happen? There are several likely causes:

Kernel bugs: Rare, but they have happened many times in the past and will happen again. Little can be done beyond applying patches and keeping the kernel up to date.
Memory issues: For example, memory errors that propagate into the filesystem’s control structures. ECC memory mitigates this, but it can never be fully ruled out.
Underlying storage problems: Quite unlikely, but still possible.
Hitting the reset button on a running server: Journaling filesystems can almost always recover from this.
RAID controller issues: Possibly the leading cause, and not easy to mitigate, even though a firmware upgrade is sometimes available.

Any filesystem type can be affected by any of these causes. XFS simply appears more often in corruption-related search results because it is so widely used.

Disaster?

You wake up in the morning, happy, with the whole day planned out. Then an error pops up on your pager:

“Corruption of in-memory data detected. Shutting down filesystem”

This one is bad. If the filesystem holds databases or web server files, things will go south within minutes: services will go down or stop serving users. That is why smart people replicate their services, use load balancers, or run a hot-replica architecture.

Repairing

The only complicated part is that you need to be on the server’s real keyboard and video (for example via a KVM switch). That may mean a trip to the server’s physical location.

Assuming that requirement is met, the repair itself is straightforward:

Boot into single-user mode. For example, edit the kernel line in GRUB and add the keyword single at the very end. This gives you a root console with no password prompt and no services running.
Unmount the corrupted filesystem using umount.
Run xfs_repair against that filesystem.

Minutes later, since xfs_repair is usually quite fast, a normal reboot should bring the system back to its former glory. Some files may be missing, and some content or database tables may be corrupted, but that’s life. At least everything starts up normally again for now.

Conclusion

Easy? It is. Well, until the next time…

Angry Birds: Following on your Passion

2017-01-15T10:55:57+00:00

This is meant to be a text about following your passion. Some people set out to save the world, while others go for seemingly simpler things, like popping green pigs. Don’t get me wrong: it’s not about the purpose per se, but about the way you get there. It always comes down to work, and a lot of it.

Looking at the screenshot below, you could easily conclude that at some point I became very, very efficient in the pig-popping business. After all, some of these golden eggs only unlock after winning 3 stars in around 60 different levels.

Note: The original Angry Birds game currently has about 500 different levels.

How many hours did I spend on this game? Many? A lot? Days? Nights? Did I do anything else? (Is that even a valid passion?)

I know what everybody is thinking: that I either have no life, or that I cheated by tampering with the data on my phone’s storage. Both are false, by the way. I’m also not sure you could reach the state shown below by cheating. Of course, you can’t complete the Master League without power-ups, but Rovio has a business to run, so that’s quite understandable.

What if I told you I never spent a single penny on any of the in-game purchases? If you don’t use power-ups in the regular levels, they pile up, because the game hands some out from time to time. Everything up to the Silver League can be won comfortably without power-ups. The Gold League is hard to win without them, and the Master League is hard to win even with plenty. As for tickets, you can collect more than you’ll ever need through good old patience: open the game every day and don’t play any level for about 2 months.

But let’s take a step back. What we’re really talking about here is:

Passion
A strong desire to win
Starting early
Many hours of work
Patience
Resource management

I hope you now see that this text isn’t really about Angry Birds. It’s about following your passion all the way to the end. Thanks for reading!

The "Debugging" Interview – a Few Pointers

2016-12-28T10:32:57+00:00

The long interview day is almost over. Googamazbook has drawn out the best and the worst of you (well, neither, but I’m trying to sneak in some literature). The last interviewer walks in, smirks, and says:

Time for the easy interview, heh?

You have every reason to worry that you’re one step from failure. Why didn’t you spend the day by the pool in the basement of that many-starred hotel they booked for you? Without further ado, the questions start coming.

Question 1

How do you figure out whether a process is CPU bound or I/O bound?

This is trickier than it looks. Two variables give four possibilities:

The process is CPU bound.
The process is I/O bound.
The process is both CPU and I/O bound.
The process is neither.

In the real world, most processes fall into the last category. They spend nearly their whole life in the S state (sleeping).

Processes also rarely run one operation, or one fixed set of operations, and then exit. Over its lifetime, a process can be CPU bound at some moments and I/O bound at others, even if those moments are a tiny fraction of its run time. Watching top or ps for long stretches in state R (running) or D (uninterruptible sleep / I/O wait) may therefore tell you nothing useful.

The conclusion: if you treat the process as a black box, you cannot answer the question. With access to the source code, you can find the relevant code blocks or add tracing. Otherwise, the best you can say is one of the following:

The process uses CPU in a way that would probably benefit from more computing power, at least for some operations.
The process reads or writes large amounts of data and would probably benefit from faster storage.

The real trick is verifying either statement for an arbitrary black-box process. It isn’t hard, but you have to look under the operating system’s hood:

High CPU usage shows up in /proc/pid/sched. Check nr_involuntary_switches and nr_voluntary_switches. If involuntary switches outnumber voluntary ones, the kernel is forcibly taking the process off the CPU, which suggests a CPU-intensive process.
High I/O also shows up in /proc/pid/sched, in the se.statistics.iowait_sum attribute. This is an absolute figure, so correlate it with the process’s running time and compare it with other processes. Monitoring with ps or top is still the best approach for I/O.

Done? Yes. On to the next one.

Question 2

How do you debug an unresponsive service?

This is too general. First ask which service it is, and whether the node is still reachable some other way (for example, over ssh). The scenario usually involves a single unusable service on a node that is up and accessible. Otherwise, “reboot the node” would be the answer, and the bar can’t be that low for an interview like this.

A typical exchange goes like this:

What service is it? HTTP.
Is it accessed by IP or by name? By name, via DNS.
Does DNS resolve the name to an IP? Yes.
Is that the IP of the node with the unresponsive service? Yes.
On the node itself, is the service listening on the expected port? Yes.
Is any firewall filtering that port? No.

All the trivial explanations are now ruled out, so it’s time for the big guns. The scenario continues:

Is the service up and running? Yes.
How many processes does the service have? About 50.
(Getting smart) What is the service’s process limit? Around that number.
(Eureka!) The service is overloaded, so raise the limits and do a graceful restart. After doubling the limit, the service is still unresponsive.

You can smell failure, as many candidates have before you. That’s a pity, because you could simply inspect individual processes and see where they’re blocked.

Run strace -p on individual processes to see what they’re doing. They’re blocked in a read() call on some file descriptor.
(Past the mountain’s peak) Check ls -la /proc/pid/fd/xx. It’s a file under /www/service.
How is that path mounted? It’s an NFS directory.
Is the NFS server reachable, e.g. via ping? Yes.
Go to that server and run top, ps, and so on. top reports 99% wa.
(The cause is identified, now for the fix) Run ps and look for processes in state D. /usr/local/bin/maintenance.sh is in state D.
What does it do? It should prune some logs, but because of a bug it keeps repeating the job.
Try kill -9, remove it from cron, and chmod a-x the script. That fixed it.

Crazy, wasn’t it? I made this scenario up myself, but it is a plausible interview exercise. A few pointers to remember:

The solution is never trivial (no client error, no external service error).
The problem isn’t immediately visible with the usual tools (ps, top, df).
The cause is almost never on the node where the symptoms appear, and the relationship between the nodes is rarely easy to work out.
System call tracing is almost always needed to uncover that relationship.

On to the next question.

Question 3

How would you deal with a “disk full” situation?

First, two kinds of resources can run out and cause a “disk full” condition:

File system blocks (the actual storage of contents).
Inodes (metadata).

So diagnosing the problem takes two commands:

# df
# df -i

The approach depends on the cause. Running out of space can usually be traced to one or a few very large files, while running out of inodes is caused by huge numbers of small files.

Finding the largest files is a job for find:

# find / -type f -size +10M -printf '%k\t%p\n'

This prints the on-disk size (in KB) and path of every file larger than 10 MB. There are usually a few outliers of several gigabytes or more. Deleting them is easy but doesn’t always fix the problem, because a program may still have them open and be writing into them.

To check, use lsof:

# lsof | grep file_name

If you can’t stop or restart the program after deleting the file, truncate may help:

# truncate -s 0 file_name

You may also meet the opposite case: a very large file was deleted but is still held open by a program. The find command above won’t show it, but lsof will, and truncate then finishes the job:

# lsof | grep "deleted"
...
# truncate -s 0 /proc/pid/fd/xx

Inode exhaustion is harder to fix. There are no outliers, just hundreds of thousands of files that may need deleting. Locating them takes some detective work: figure out what software runs on the node and whether anything changed in the last few days. find helps again:

# find / -mtime -1 -print

This lists files created or modified in the last day. For a more complex deletion example, see the second question in this interview series.

That’s all for today. Thanks for reading!

The "Networking" Interview - a few Pointers

2016-12-21T11:07:57+00:00

You’re sitting in a tiny room at Googamazbook. The interviewer comes in, says hi (if you’re lucky), introduces themselves (they act like they’re doing you a favor) and starts with what seems like an easy question:

What happens when you run telnet www.brainware.ro 80? Please go through all the layers.

Warmed up by the systems interview pointers, you start with /etc/hosts and /etc/resolv.conf: name resolution first, then the actual connection to the HTTP port. If your knowledge stops there, keep reading. If it doesn’t, keep reading anyway, in case something was missed.

1. Name Resolution

Name resolution works by sending a UDP packet to each resolver listed in /etc/resolv.conf. The UDP header is small and contains:

Source port and destination port;
Packet length (header + payload);
Checksum.

The checksum is computed over a so-called UDP pseudoheader (which includes data from the protocol above it, such as the source and destination IP addresses) plus the payload. This is a tricky protocol detail.

The IP address needed for the next stage is taken from the first response received (the exact response format is out of scope for an interview).

2. TCP Connection

Everybody knows the SYN / SYN-ACK / ACK sequence. But what is actually exchanged when the connection is initiated?

Sequence numbers;
MSS (maximum segment size: if a TCP payload larger than the MSS arrives in a single packet, the other side drops it);
Receive window size (how much data the other side is able to receive).

From here, questions can escalate quickly, for example:

How does TCP detect that packets were lost in transit? It doesn’t directly; a retransmission timer is used. The other side may also send packets from time to time with the ACK field set to the last sequence number received.
What is silly window syndrome? The receiver keeps advertising tiny window sizes while the sender tries to push out as much data as it can. It is usually mitigated by both the sender (delaying data) and the receiver.
What is TCP Cork? It delays data for up to a timer value (around 200 milliseconds) so that larger payloads can be sent when appropriate and possible.
What are SYN cookies? A defense against a SYN flood. The initial sequence number is a specially crafted value that encodes the MSS and a connection signature.
What is a congestion window? The amount of data one side may send without waiting for an ACK; it is dynamically resized up and down. In practice it is represented as a “window” over a circular buffer.
Why are there many TIME_WAIT connections in netstat that can’t be closed or removed? This comes from the inner workings of TCP: connections can’t be guaranteed to close cleanly before a timeout. During those few minutes, any stray ACK packets belonging to the connection can still be received and handled properly.

The discussion will soon move down to layer 3 (IP).

3. IP Connection

As soon as you mention MSS, the interviewer will probably ask how it differs from MTU. They are distinct concepts: one applies to the TCP payload, the other to the total IP packet size. Exceeding one causes packets to be dropped; exceeding the other causes fragmentation. MSS is communicated, while MTU is discovered.

An important topic is MTU discovery. On both IPv4 and IPv6, this is done by sending maximum-size packets and listening for “fragmentation needed” ICMP replies. With IPv4, the “do not fragment” flag must be set for this to work. The ICMP responses (if any) carry the MTU accepted by that node on the route.

Some points worth mentioning about MTU:

The standard Ethernet value is 1500 bytes;
Every transmission protocol (e.g. WLAN, PPPoE, FDDI, Token ring) has its own maximum value;
IPv6 forbids fragmentation by intermediate nodes (only the source may fragment data as needed).

Mentioning “Ethernet” leads the discussion to layer 2.

4. Data Link

This layer is simpler to describe: the payload (most likely an IP datagram, but possibly ARP) is encapsulated in a frame with a header and a footer. The header usually holds the source and destination MAC addresses, an optional VLAN field, and a field indicating either the length or the payload type. The footer holds the checksum.

The conversation may then turn to ARP, which is simply an IP-to-MAC resolution protocol. It works by broadcasting messages like “who has ip_address?” and answering with unicast messages like “ip_address is at mac_address”. Every node keeps a cache of the associations it knows about; entries expire within minutes, so ARP messages are sent regularly. Such a simple protocol is open to attacks, but that’s getting a bit far afield.

Very few candidates get asked about the next layer, the physical link.

5. Physical Link

Layer 1: it can’t get any lower than this. With an Ethernet connection, the 1524 bytes (maximum MTU plus the Ethernet header) are placed between a header (the preamble) and a footer (the frame check sequence). These are handled by the firmware of the Ethernet device.

Is that all? Not necessarily. There may still be time for variations.

6. More Questions

You still have 10 minutes left, and the interviewer wants to end on a high note:

How does DHCP work?

You can’t help thinking of Dora the Explorer, and that’s actually a good thing: it means you read the docs! There are 4 steps before the client gets an IP address, two initiated by the client and two by the server:

Discover;
Offer;
Request;
Acknowledge.

Communication uses UDP on port 67 (server) and 68 (client). The client has no IP address yet, so 0.0.0.0 is used as the source address. The first packet is a broadcast, which means any device on the local network can receive it and possibly return an offer. The remaining 3 packets are unicast (sent to a specific MAC address).

Follow-up:

Why 4 packets? Wouldn’t the first 2 be enough?

Two would be enough if the protocol guaranteed that only a single DHCP server exists on the network. That isn’t the case, so the client must choose one of the offers to proceed with.

As the next interviewer knocks on the door, the current one still has one last question:

How does traceroute work?

It’s simple: it sends packets with increasing TTL values and records the responses. All the usual protocols (ICMP, TCP, UDP) are supported. The traceroute program also offers handy features such as MTU discovery and gateway testing. There is also a simpler tool worth checking out: tracepath.

Even at this point the interview may not be over, but these pointers should help you make a good impression, or at least there’s hope. Thanks for reading!

Crazy DevOps Interview Questions (4)

2016-12-06T20:10:57+00:00

Note: This is the fourth article of the series. The others are: part 1, part 2 and part 3.

Question 1

You have a shared document open when the phone rings. The interviewer, speaking with a thick accent, asks:

“What does ls * do?”

It sounds far too easy to be true. You say it lists all the files in the current directory. The interviewer asks a couple of follow-ups about how it works, and you explain that the star is passed to ls as an argument, which somehow makes the binary walk the whole directory and print its contents. Simple!

The interviewer thanks you and hangs up, and you’re left grinning. That lasts until the recruiter calls to suggest you try again in a year or two.

So what actually happens when you run it in a Linux shell? You’d see something like this:

$ ls *
1file 2file 3file

4dir:
a b c d

5file 6file

7dir:
e f g h

8file 9file

Notice that ls descended one level into the subdirectories. That would be impossible if it only listed the current directory, so something else must be going on.

There is no magic here. The shell expands the star into a list of matching files and directories before ls runs, then hands that list to ls as arguments. ls then processes each argument in turn, listing files as they are and showing the contents of directories. This is the “trick” the interviewer is after, but a few more details are worth mentioning:

Hidden names are skipped. The star expands only to names that don’t begin with a dot. Dot-files aren’t truly invisible. By convention, dot-prefixed names are reserved for local configuration files.
The argument list has a size limit. If the expanded list has too many entries or is too long, you get a “too many files” style error. The limit is high and varies between systems, but it is finite. You could hit it if, say, a program writes status files into a single directory for a long time. The GNU Core Utilities FAQ explains this in more detail.
Empty directories behave differently. If there is nothing for the star to match, the shell leaves it alone and passes a literal * to ls, which then reports a “not found” error.

If you answered along these lines, expect follow-up questions about handling “too many files” errors. These are usually solved with find, which brings us to the next question.

Question 2

The interviewer presents this scenario:

“I tried to delete everything in the current folder with rm -f * and got an ‘argument list too long’ error. I really need those files gone.”

As noted above, find is the right tool because it enumerates files one at a time instead of building one giant argument list. For example:

$ find . -type f -print

On its own this not very useful, but it is easy to turn into a scalable equivalent of rm -f *:

$ find . -maxdepth 1 -type f -delete

If you’d rather call rm explicitly:

$ find . -maxdepth 1 -type f -exec rm -f {} \;

Or you can pipe the results to xargs:

$ find . -maxdepth 1 -type f -print0 | xargs -0 rm -f

Note: -print0 separates file names with a null character, and xargs -0 reads them back in that format. This is necessary for file names containing unusual characters such as spaces, tabs or newlines.

Question 3

Having solved a hard problem, you get an easier one to finish the interview: rename every .htm file in a directory tree to .html. Sounds easy, right?

Your first instinct is correct: use find to locate the files.

$ find . -type f -name "*.htm" -print

Now what? You could pipe the output into a script, but the interviewer probably wants a one-liner. Here is the simplest approach:

$ find . -type f -name "*.htm" -exec mv -f {} {}l \;

Did you spot the trick? find replaces {} with the current file name, so {}l is the same name with an l appended.

The follow-up is harder: do the reverse and rename every .html file to .htm. This calls for some shell string manipulation using the % operator:

$ find . -type f -name "*.html" -exec sh -c 'f="{}"; mv -f $f ${f%l}' \;

Notes on the command above:

-exec launches a shell with an inline script, and find substitutes {} with the file name.
The script is in single quotes so the calling shell doesn’t expand $f before the inline script runs. Without them, the assignment just before it would effectively be ignored. This is a tricky point. Try the following and compare what gets printed:
```
$ a="outside"
$ sh -c 'a="inside"; echo $a'
$ sh -c "a=\"inside\"; echo $a"
```
${f%l} looks odd, but it simply strips the trailing l from the value stored in $f.

I’m curious how many people could solve this in an interview without having worked through all these pitfalls first. Not many, I suspect.

That’s all for today. Thanks for reading!

The "Systems" Interview - a Few Pointers

2016-12-03T13:20:57+00:00

Note: This article is about Systems Programming on Linux platforms.

This kind of interview checks what you know about what happens beneath the command-line surface. Down there, processes are created and terminated, output is collected, and system calls are made. Things get complicated quickly for anyone who is unprepared.

Question: What happens when you run…?

$ ls

This classic interview question has been asked for more than 15 years. It may not come up much anymore (as of 2016), but the answer is still instructive. The question is not about the file listing that appears on screen, but about everything that happens to produce it.

Step by step

The shell parses the command line and identifies the command and its arguments (none here).
The shell works out whether the command is:
- a built-in (internal) command,
- an alias, or
- an external binary that must be located.
On most systems, ls is an alias for the real binary plus an option such as --color=auto.
As a safety step, the current directory is validated with stat(".") before anything is executed.
The shell searches for the binary by walking through each directory in the PATH environment variable. It appends the command name to each one and calls stat() on the result. ls is usually in /bin, so the search stops once /bin/ls is confirmed as executable.
The shell creates a new process with fork(), then calls an exec()-family function to run the binary with the arguments determined earlier.
Boilerplate startup code runs: shared libraries are loaded, memory is allocated, signal masks are set, signal handlers are installed, and so on.
Now the interesting part begins:
- ls reads /proc/filesystems and loads the current locale.
- It opens the current directory with opendir().
- It enumerates the entries with readdir() and closes the directory with closedir() (more memory may be allocated along the way).
- It writes the directory contents to standard output with write(1, ...).
Finally, ls releases its memory and exits.

It’s more involved than it first looks, isn’t it?

Follow-up: What about…?

$ ls | tee list.txt

The basic idea is simple: the standard output of ls is piped into the standard input of tee.

NB: stderr is not redirected in this case, so error messages will most likely go straight to the console.

What happens behind the scenes?

The shell still parses and executes everything. The earlier details still apply: built-in vs. alias vs. external command, and a stat() in each PATH directory until the binary is found. The difference is that the shell now identifies and launches two external commands (ls and tee), each in its own process.

The “magic” happens before fork() and exec(). Roughly:

The shell asks the kernel for an unnamed pipe with the pipe() system call, which returns a read end and a write end.
The write end of the pipe is duplicated with dup2() onto the stdout of the process that will run ls.
The read end of the same pipe is duplicated with dup2() onto the stdin of the process that will run tee.
The shell collects the exit codes of both child processes. Even if the first one produces no output, the second one still runs, with empty input.

Digging deeper:

An interviewer who knows this area well will probe how fork() and exec() behave at the kernel level. A strong candidate should be able to explain the following.

fork() duplicates the current process. Logically, nearly everything is copied: memory, file descriptors, and kernel resources such as references to shared memory segments or semaphores. Only the PID differs. If the process has several threads, only the thread that calls fork() is duplicated, which can cause many hard-to-find bugs.
Copy-on-write: the memory is not actually copied up front. Pages are marked read-only, and if either parent or child tries to modify one, a page fault occurs. The kernel then quietly duplicates the page and lets the write proceed.
exec() replaces the process image with that of the new binary, and the kernel frees the previously allocated memory. File descriptors without the close-on-exec flag (O_CLOEXEC) stay open and are available to the new program. By default, stdin, stdout, and stderr do not have this flag set.

Closing thoughts

As you can see, this kind of interview can get tough quickly. Fortunately, only a few companies run such “systems interviews,” and only for certain roles. If you have one coming up and spent a few minutes reading this, you’re likely better prepared than before. At least, that’s the hope. Thanks for reading!

Amazon CloudFormation (an Introduction)

2016-11-20T14:22:07+00:00

CloudFormation is Amazon Web Services’ core tool for expressing infrastructure as code. Infrastructure is described in either JSON or YAML (the keywords are nearly identical), and a template designer can help assemble resources and their dependencies. The specifics of each resource still have to be written out by hand.

From what I’ve seen, there is no limit on the kinds of resources you can define. Every resource type AWS offers can be coded, created and provisioned. You can set an explicit creation order (e.g. resource A before resource B) on top of the implicit order CloudFormation works out itself (e.g. an IAM role attached to an EC2 instance is always created first). Rollback is clean too: deleting a stack removes everything the stack created, and nothing is left behind.

A Limitation: Pre-existing Resources

That same design has a downside: working with resources that already exist is awkward. For example, if you define a policy for an existing S3 bucket, it completely overwrites any policy already on the bucket, and deleting the stack leaves the bucket with no policy at all. There are workarounds using Lambda functions, which are not covered here.

Template Structure

A stack file, in JSON or YAML, has three main sections:

Parameters: the inputs, e.g. the SSH key pair to use with newly provisioned EC2 servers.
Resources: the contents of the stack (EC2 servers, security groups, Route 53 DNS entries, RDS databases, etc.).
Outputs: typically public IP addresses, but possibly other values depending on the resources created.

Note: According to the template anatomy documentation, Resources is the only section that is strictly required. A template with no inputs and no outputs seems very limiting, though.

Constants (AMI IDs, node names, etc.) can be placed in a Mappings section. Conditions let you make decisions, such as creating or skipping certain resources, based on things like the AWS Region or user-supplied Parameters.

Programmatic Constructs

As a coder at heart, I was drawn to the programmatic features CloudFormation offers. Here are a few JSON examples.

Using user inputs (Ref)

{
  "Parameters" : {
    "KeyName": {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
    },
    "InstanceType" : {
      "Description" : "EC2 instances type",
      "Type" : "String",
      "Default" : "t2.micro",
      "AllowedValues" : [ "t2.micro", "m4.large", "c4.large", "r3.2xlarge" ],
      "ConstraintDescription" : "must be a valid EC2 instance type."
    },
  }
  "Resources": {
    "MyEC2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "InstanceType" : { "Ref" : "InstanceType" },
        "KeyName" : { "Ref" : "KeyName" },
      }
    }
  }
}

Looking up constants (Fn::FindInMap)

{
  "Mappings": {
    "InstanceValues" : {
      "Names": { "MyInstanceName" : "myec2.environment.local" }
    }
  }
  "Resources": {
    "MyEC2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "Tags" : [ {
          "Key" : "Name",
          "Value" : { "Fn::FindInMap" : [ "InstanceValues", "Names" , "MyInstanceName" ] }
        } ],
      }
    }
  }
}

Joining text (Fn::Join)

{
  "Parameters": {
    "S3BucketName": {
      "Description" : "Name of an existing S3 bucket",
      "Type": "String"
    },
  }
  "Resources" : {
    "EC2RolePolicy": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "MyEC2RolePolicy",
        "PolicyDocument": {
          "Statement": [ {
            "Effect": "Allow",
            "Action"   : [
              "s3:GetObject"
            ],
            "Resource": { "Fn::Join": ["", ["arn:aws:s3:::", { "Ref" : "S3BucketName" }, "/*"] ] }
          } ]
        },
      }
    }
  }
}

Getting the ARN for another resource (Fn::GetAtt)

{
  "Resources": {
    "EC2Role": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [{
            "Effect": "Allow",
            "Principal": { "Service": "ec2.amazonaws.com" },
            "Action": [ "sts:AssumeRole" ]
          }]
        },
        "Path": "/"
      }
    },
    "MyBucketPolicy" : {
      "DependsOn": "EC2Role",
      "Type": "AWS::S3::BucketPolicy",
      "Properties": {
        "Bucket": {"Ref" : "MyBucketName"},
        "PolicyDocument": {
          "Statement": [{
            "Action": ["s3:GetObject"],
            "Effect": "Allow",
            "Resource": { "Fn::Join": ["", ["arn:aws:s3:::", { "Ref" : "S3BucketName" } , "/*" ]]},
            "Principal": { "AWS": [ { "Fn::GetAtt" : [ "EC2Role" , "Arn" ] } ] }
          }]
        }
      }
    },
  }
}

More functions are available in CloudFormation templates; see the intrinsic function reference.

Integrating with Configuration Management Tools

There are two main points of contact with tools such as Chef, Ansible, Puppet or Salt:

Instance user data: the bash provisioning script passed to newly created instances. With the constructs above, you can inject node names, fixed IP addresses, S3 bucket names, public SSH keys and more.
Security groups: you can open ports for configuration management engines that use protocols other than SSH (e.g. HTTPS for Chef).

The recommended practice is to keep CloudFormation focused on defining infrastructure, including installing and configuring base system packages, and then hand control over to your configuration management tool of choice. That tool should take care of installing and configuring the services themselves.

That’s all for now. Thanks for reading!

Note: This article was written by an AWS Certified Solutions Architect (Associate). Always work with an expert when setting up production environments.

When There’s no Route Forward (Or So You Fear)

2016-11-14T20:08:07+00:00

Note: This is a reflection on project work done at one of my previous employers, covering a period from late 2000s to early 2010s.

Introduction

Agile, automation, DevOps and “continuous everything” are on everyone’s lips these days, yet very few people stop to ask why the industry moved this way. For many years, software projects followed a then- familiar routine:

Thorough upfront planning, repeated before each major milestone.
Long stretches of development behind closed doors.
Mostly manual QA, with some automation likely built on a custom testing framework that one of the developers wrote from scratch.
Rare releases, perhaps once a year or even every two years, preceded by careful testing and code freezes that sometimes lasted months.

We now know this was far from ideal, but it worked for a long time because it matched what the world expected. The constraints of that era explain a lot:

Hardware limits: multi-core CPUs only reached consumers about 10 years ago (mid 2000s).
Unreliable connectivity: I was an early broadband adopter in my country 14 years ago (2002), when “broadband” meant roughly three times dial-up speed or about 16 KB/s.
Slow provisioning: setting up test environments meant manual installs (yes, from CDs or DVDs), and only after management approved the hardware expense.
Immature virtualization: VMs had existed since early 2000s, but until recently VMs were barely easier to configure than physical machines, and their performance was underwhelming.

This was how things were done across our project and most others in the company. Nobody questioned the process until after we actually went live with the product and that’s when things began to fall apart.

Disaster?

For roughly two years, things went from bad to worse. The tension between the development and operations teams escalated into something close to open war, and management’s attempted fixes alienated people on both sides. In the end, most members of both teams left for other companies.

With a few years of distance now, it’s clear that with the way things were going the collapse was inevitable, but then it could also have been avoided. So what went wrong?

Feature planning: Before every major milestone, we spent 3-4 consecutive days discussing features, and we developers built most of what was agreed. About a year later, when the updated product shipped, the problems started to show. Some features weren’t needed at all, because Ops had requested them without knowing how the product would really work. Others were “buggy”: not from careless coding, but because the product behaved the way the developer imagined, not the way the operations person expected.
Test automation: We blamed the lack of QA time for the buggy releases and tried to introduce automation. We built a full high-level testing engine that ran command-line calls and checked the responses. It could never cover the product’s complexity, and by the time it was mostly finished, the product had moved on. Eventually QA spent most of its time maintaining the engine instead of testing new features.
Performance problems: We initially took the blame for several performance issues. Later root-cause analysis pointed elsewhere: reporting scripts that dumped entire databases at once, and a badly configured monitoring setup that effectively amounted to a denial-of-service attack. Different members of the operations team had installed these without telling anyone.
Open conflict: At the peak, emails flew daily claiming that our software was buggy and unreliable. Meanwhile we were helping to fight fires and growing more frustrated, convinced we were doing our best to no avail. Looking back, I think everyone had a point, but at the time there was no quick or easy path to peace.

Management response

Eventually management stepped in and made things worse. They decided the root cause was our project management process, so we were told to move from “Waterfall” to Agile, with Sprints and regular Scrum meetings.

This was a step in the right direction in principle, but the execution was poor. We were already in firefighting mode, constantly reviewing, fixing and working around problems, and now we spent an hour or more every day in stand-ups that added little value. As the development team shrank with departures, this became the new normal.

What did Agile actually change for us?

We had Sprints, but the workload stayed the same. Priorities may have become a little clearer, but tasks were still handled whenever a developer was free, and the backlog kept growing.
We released no more features than before, because Operations “could not handle them”. The goal of a deliverable at the end of each Sprint was never met. We did ship frequent emergency fixes, but they had nothing to do with Sprint timing.
We still held week-long planning sessions about the next year’s tasks, which undermined the whole point of Agile.

In short, Agile did little for us, given how we adopted it and how we kept interacting with Operations. The development team took the change as an undeserved punishment, and most people resigned within the following year.

In an alternate reality

What should have been done? I’m convinced now that we should have adopted DevOps. With the right mindset, many things were within reach:

Merge Development and Operations so that they shared ownership of the production environment, in the literal sense of being in the same office and under the same manager.
Set up a Continuous Integration pipeline to free up QA time.
Build a fast, frequent, automated deployment process.
Replace manually configured servers and patchy scripting with proper configuration management.

We could have saved the project if management had made better decisions. Instead, everyone fell into “saving face” and “shifting blame” modes, and they probably just didn’t know of better ways of working.

Until our very last day, we developers believed that writing the best possible code was the key to a good product. We also felt that even if we did our jobs well, management and Operations might sabotage the result, so the fight couldn’t be won. A few years on, I’ve learned that good code can only take you so far.

That’s it for today. Thanks for reading!

AWSome Day Bucharest 2016

2016-11-08T12:46:57+00:00

Note: You can find my comments on the similar event held in 2015 here.

Another year, another AWS event. The venue changed and more speakers were brought in, but unfortunately they were of lower quality than Sebastien Stormacq, the sole speaker in 2015. Registration doubled this year, with roughly 500 people queuing for the early-morning check-in.

Two Tracks

The biggest change from last year was that the content was split into two tracks, a “technical” room and a “business” room, each with their own focus.

Technical track topics:

Storage and content delivery (S3, CloudFront)
Compute and networking (EC2, VPC)
Managed services and databases (RDS)
Elasticity and management services (ELB)

Business track topics (centered on the benefits of moving to the Cloud, such as flexibility and cost savings):

Benefits of Cloud Computing and defining a Cloud Strategy
AWS Security
Cloud Economics (TCO)
AWS partner programs

My Experience in the Business Track

I have attended the Business Track because I already knew the presented technical content well enough (I’m AWS CSA(A) certified), so my choice came down to curiosity. This led to a couple of unsettling observations.

1. Very low interest in the business track. Business attendees were outnumbered by technical ones by more than 15 to 1. Beyond the small turnout, the lack of engagement was obvious: many people were on their phones (possibly answering e-mails), and very few questions were asked at the end of each session. One attendee in my row, dressed in a full suit and tie, simply fell asleep.

2. No apparent plans for a nearby Edge Location. Amazon doesn’t seem in a hurry to expand its offering by opening an Edge Location closer, latency-wise, to Eastern (South-Eastern) Europe. Serving presentation content from 60-80 ms away may not be a problem, but moving internal real-time applications to the Cloud with that latency could make the user experience worse than a previous local data center solution, even if the Cloud is better on reliability and cost.

Conclusion

Until these two issues are somehow fixed, AWS technical expertise in my country will continue to be sold by the hour to external customers.

How to Fail a Big Company Interview (a Short Guide)

2016-11-02T12:09:07+00:00

So you’ve landed this incoming interview with Googamazbook, and you’re determined to fail it. Or maybe you’re not determined, you just don’t trust the 99.9%+ rejection rate these companies are known for and would like some say in how you fail. Either way, you’re in the right place.

Stage 1: The Recruiter Screening

The recruiter screening is surprisingly hard to fail, especially if school had trained you to give the expected, correct answers. Some people do stumble here because they don’t “speak the same language” as the company’s current employees, but they’re a minority.

So if:

you’re set on failing the recruiter screening;
you have plenty of experience;
you were good at quizzes in school…

…you’re practically guaranteed to pass. Sure, you could invent wrong answers, but that’s cheap. Let’s skip ahead.

Stage 2: The Phone Interview

Now you’re talking to an engineer, maybe two. No more quizzes, just problems to solve.

Failing here is easier: solve the problem using a different method than the one your interviewer had in mind. You could offer a brute-force solution as a quick exit, but if you take pride in doing things well, that’s a real struggle. You’ll probably find the optimal solution and fail at failing. Fortunately, a few things can help:

Call quality: If you can’t really understand the interviewer, failure is almost guaranteed. Even if you catch the rough ideas, you’ll miss on nuance and tone. Pray for a bad line, and firmly refuse any callback.
Interviewer benevolence: Sometimes you get a genuinely kind interviewer who wants to help you solve the problem. That makes failing very hard. You need to hope for someone with a radically different attitude.
Open-ended questions: Dive straight into the immediate problem as you see it. Never ask what’s actually needed or expected. If you ask clarifying questions first, you’re likely doomed to pass.
Questions at the end: Ask only about perks and work/life balance. It’s no guaranteed fail, but you’re on the right track.

Stage 3: The Onsite

Against your wishes, you’ve made it to the onsite. Time to bring out the big guns.

The lazy way to fail is not showing up. If they flew you to a new city, you could enjoy a free mini-vacation with most expenses covered. But you’re not that lazy: you show up on time, sober and prepared for the long day.

The classic way to fail is not solving the interviewer’s problem, whether it’s a coding challenge or a debugging task. But some problems seem easy or average, which makes failing tough. A few ideas:

Debugging scenario: Stop as soon as you’ve found the issue the interviewer had in mind. They expect you to carry it through to a conclusion, such as a definitive fix. Leaving it without closure is an easy way to fail.
Architecture design scenario: Propose round-robin DNS as the load-balancing strategy for your service. It’s a lightning-fast failure, so make a note of it. At some companies it may even shorten your interview day. More free time!
Logging: If logging comes up, stick to internal (local) logging and refuse to consider any alternatives.

If nothing works, trust the stats: at this stage, more than 90% of candidates don’t get an offer, so put your hopes to that.

Stage 4: The References

A few days or weeks after the onsite, the recruiter asks for several people to vouch for you. That’s bad news, because even a lukewarm reference from someone who barely knows you could push you to an offer.

Your best bet is to develop a multiple personality disorder, get phone numbers from another country, and answer the reference calls yourself, though that only works if you’re good at faking voices. Otherwise, try convincing someone to say something along the lines of “Who? Don’t get me started on that lazy pr1ck!” That should do it.

Stage 5: The Offer

Tough luck: at this point, you have to turn the offer down. But you won’t get here by following the advice in this article, so don’t give it too much thought.

That’s all for today. Thanks for reading!

NB: This text is not necessarily based on my personal experiences, so please take any advice here with a grain of salt.

Pierre Gauthier’s Google Interview: My Opinion

2016-10-18T15:11:07+00:00

The original post, published in April 2016, is available here (original blog post by Pierre Gauthier). Business Insider had recently picked it up (link) and sparked a lot of debate. My thoughts are below.

First, the questions are typical of a recruiter screening stage at any large company. Some candidates know the answers, while others lack even the basic concepts and struggle through the whole test. At its core, the screening exists to reduce the “load” on later “interview layers”: phone interviews with engineers, followed by on-site interviews. No company can afford to have a highly trained engineer run this kind of test. Engineer time is expensive, it takes a mental toll, and it is likely to breed frustration given how large a share of candidates turn out to be unsuitable.

Beyond the technical assessment, the screening also serves to catch a possible culture mismatch early. Does the candidate speak the same language? Not in the sense of English, but in the sense of whether they use the same words to describe technical concepts. Knowledge can be expressed in many forms, and not all of them are useful to everyone. For example, the 0x02 SYN representation is certainly handy for low-level debugging, but it is of no use in a normal conversation, even with an engineer who knows the TCP stack well.

Pierre seems to have failed the second layer of such screening, but also another one as well. There is a concept called “technical humility”, which is roughly the opposite of an inflated ego. Google in particular appears to have no use for such egos and places a very high value on technical humility: knowing when you don’t know enough, listening, respecting colleagues as human beings, and striving to become a better person and a better engineer every day.

Later edit: Business Insider claims that Pierre gave a better answer to one of the questions, the one about counting bits in an array of integers:

At one point, on a question about how to most efficiently count data, he suggests a method that might be faster than the answer the recruiter wants, but the recruiter rejects it.

This is actually incorrect. The recruiter expected a “theoretical” answer, independent of any optimization a particular CPU may or may not offer. Counting bits by right-shifting is the brute-force solution to this problem, and it is roughly 8 times more expensive than using an 8-bit lookup table, or k times more expensive than a k-bit lookup table, assuming you have enough memory and the right computing architecture to implement it.

In any case, this looks to me like a symptom of the “entrenching condition” I described in this earlier post.

Crazy DevOps Interview Questions (3)

2016-09-11T14:32:57+00:00

Note: This is the third article of the series. The others are: part 1, part 2 and part 4.

Question 1

The interviewer hands you this ls -la listing:

ls -la

total 108
dr-xr-x---. 7 root root 4096 Sep 5 07:16 .
dr-xr-xr-x. 22 root root 4096 Sep 1 17:43 ..
-rw-------. 1 root root 15432 Sep 5 06:36 .bash_history
-rw-r--r--. 1 root root 18 May 20 2009 .bash_logout
-rw-r--r--. 1 root root 176 May 20 2009 .bash_profile
-rw-r--r--. 1 root root 176 Sep 23 2004 .bashrc
-rw-r--r-- 1 root root 0 Sep 5 07:16 -f
...

They tell you the -f file has to go. Can you delete it?

Common (failed) attempts:

Quoting (rm "-f"): The shell strips the quotes, so rm receives -f as an option. It reports no error, because -f tells rm to ignore missing files, but nothing gets deleted.
Escaping the minus (rm \-f): The shell processes the backslash, and rm again receives a plain -f.
Double-escaping (rm \\-f): rm receives the literal string \-f, which fails because a file named \-f doesn’t exist (only -f does).

Many candidates give up here, which does not bode well for their employment chances.

Working solutions:

Both are found in man rm:

Use -- to mark the end of options:

rm -- -f

Prefix the filename with the current directory:

rm ./-f

Follow-up question:

Which characters are forbidden in filenames?

The answer is that on Linux, only \0 (null, the string terminator) and / are disallowed. Windows has a longer list.

Question 2:

At the whiteboard, write a cron definition for a file in /etc/cron.d that appends the date/time to a file every Wednesday at 5 PM, using the format %Y-%m-%d %H:%M:%S.

An experienced candidate might quickly write:

0 17 * * 3 root date +’%Y-%m-%d %H:%M:%S’ >> /tmp/logfile

Looks right, but it isn’t. Things you might check:

Cron runs commands with the shell configured in /etc/crontab, not your favorite one (irrelevant for such a simple command).
Local PATH changes aren’t visible, though /bin is usually included.
Local environment variables aren’t visible (none are used here).

Now, none of these is the culprit. The real problem is the % character. Cron treats it as a newline. The command gets split at each % and passed to /bin/sh like this:

date +’
Y-
m-
d
H:
M:
S’ >> /tmp/logfile

This obviously produces an error. The fix is to escape every percent sign:

0 17 * * 3 root date +’\%Y-\%m-\%d \%H:\%M:\%S’ >> /tmp/logfile

Question 3

A mission-critical process is writing to a file whose contents must be preserved, but a junior admin just ran rm -f on it. The process is still running, but in a few hours log rotation will send a HUP signal to said process.

Now, it’s less scary than it sounds:

Find the process’s PID and inspect /proc/<pid>/fd to locate the deleted file’s descriptor:

ls -la /proc/966/fd/10
lrwx------ 1 root root 64 Sep 5 08:56 /proc/966/fd/10 -> /var/lib/critical/process.out (deleted)

Copy out the contents:

dd if=/proc/966/fd/10 of=/tmp/recovered.out

But what about data written after dd finishes? Possible approaches:

Send SIGSTOP to freeze the process, copy the data with dd, then kill or restart the service.
Copy the data, then immediately send SIGHUP.

Neither guarantees full preservation if the application buffers its output. (A mission-critical application should flush after every write, which is a good point to raise in the interview.)

A cleaner solution uses tail:

tail -f -n +0 /proc/966/fd/10 > /tmp/recovered.out

This captures the whole file from the beginning and keeps following new writes. You can leave it running until the log rotation event if a faster restart isn’t desirable.

Question 4

A node has hit 100% storage. An administrator investigated but all visible file sizes look normal. The node can’t be restarted because of mission-critical software.

Diagnosis: If no visible file is large enough, some process is probably holding a reference to a very large deleted file (or several deleted files that add up). lsof helps find it:

lsof | grep deleted | awk ‘{print $2,$4,$8,$9}’

2006 34w 24967 /tmp/vteAIZ4MY
2006 35u 28708 /tmp/vteXJZ4MY
2006 36w 28985 /tmp/vteYU9XMY
....

This filters lsof output for deleted files and prints four fields:

The PID of the process holding the deleted file open;
The file descriptor (with access mode);
The file size, the field to look for;
The file name.

Once identified, there are two options:

Reload or restart the process, e.g. kill -HUP <pid>
Truncate the file, e.g. truncate -s 0 /proc/<pid>/fd/<fd>

That’s it for today. Thanks for reading!

Book Review: How Google does SRE

2016-09-04T20:02:57+00:00

I’d like to share a book I’ve been working through for a while now. It’s a very dense read, packed with good practices and interesting details on how a team of exceptionally well-prepared people keeps planet-scale systems up and running.

Key lessons from this book

SRE engineers should be software engineers at heart. Infrastructure is now expressed as code, so the role demands strong engineering skills. Operational work should stay below 50% of an SRE’s time, because they have a dual mission: keeping today’s systems running while building tomorrow’s.
Going on call is the “rite of passage” for an SRE. It follows naturally from the thorough preparation during the first months on the job. There is also a sweet spot for how often an engineer gets paged each week: too few incidents and they start to lose touch with production, too many and they may begin ignoring alerts or burn out.
The environment is designed to bring out the best in people and deliver the best results for the company. Google looks for specific qualities in prospective SREs, which is a world of its own, but the core values that come through in the book are collaboration, mutual support, continuous improvement, and blameless post-incident analysis.

Technical content

Several chapters cover the technologies Google uses to run services at planetary scale (and beyond). The technical detail can get confusing at times. I personally found the chapter on consensus algorithms hard to follow. On the other hand, the chapters on load balancing are essential reading for anyone aiming to enter this field, whether or not they want to work at Google.

Where to get it

Google has made the book freely available online (link).

Publishing to Atlassian Confluence (with Python)

2016-08-19T18:32:57+00:00

Introduction

This article shows how to automate the interaction with Atlassian Confluence, a commercial product widely used in companies to publish documentation and other content for a team, a department or the whole organization. It is usually integrated with other Atlassian tools such as Bamboo and JIRA.

By “automating the interaction” I mean automated document publishing: you extract data from other systems with automated tools, then publish it in a readable format. This is possible through the REST interface that Atlassian provides for its products. REST does not cover every feature, but it is enough for this scenario. The reference for Confluence Cloud is here. The interface can differ between versions, so check the API reference for your own Confluence version.

Now that we have the tools and the method, let’s get to the implementation, and to some code.

Prerequisites

First, create the document the classic way, with the “Create” button in the web interface. A document can be created through REST calls, but you would need to supply the “page key” and the “parent page id”, which are easy to read from the interface. Once the document exists, note down its page key and page id.

The page key is visible in the page URL.
The page id becomes visible when you edit the page.

Document Retrieval

All the code below assumes a dictionary holding the credentials and the document identifiers:

confluence_data = {
    "server": "xxx",
    "user": "xxx",
    "pass": "xxx",
    "page_key": "xxx",
    "page_id": "xxx"
}

Confluence does not accept an update unless the version number (an integer) is incremented. So you must first retrieve the document (as JSON, here) to read the current version, then increment it:

import requests
import json

def confluence_retrieve(confluence_data):
    url = ('https://' + confluence_data['user'] + ':' + confluence_data['pass'] +
           '@' + confluence_data['server'] + '/confluence/rest/api/content/' +
           confluence_data['page_id'])
    headers = {'X-Atlassian-Token': 'no-check', 'content-type': 'application/json'}
    response = requests.get(url, headers=headers, verify=False)
    return int(response.json()['version']['number'])

Document Update

Updating a document takes a few steps:

Format the content in a representation Confluence accepts (details below).
Generate a new version number by incrementing the old one.
Build the payload (JSON here; XML also works).
Send the PUT request and hope for the best.

def confluence_update(confluence_data, new_title, new_content):
    old_version = confluence_retrieve(confluence_data)
    new_version = old_version + 1

    url = ('https://' + confluence_data['user'] + ':' + confluence_data['pass'] +
           '@' + confluence_data['server'] + '/confluence/rest/api/content/' +
           confluence_data['page_id'])
    data = {
        "id": confluence_data['page_id'],
        "type": "page",
        "title": new_title,
        "space": {"key": confluence_data['page_key']},
        "body": {
            "storage": {
                "value": new_content,
                "representation": "wiki"
            }
        },
        "version": {"number": str(new_version)}
    }
    headers = {'X-Atlassian-Token': 'no-check', 'content-type': 'application/json'}
    response = requests.put(url, headers=headers, data=json.dumps(data), verify=False)
    return

Confluence accepts several content representations:

plain: plain text, used as provided with no special formatting.
wiki: Confluence wiki markup. This is not the Wikipedia wiki markup but a different format. It is the easiest way to get nicely formatted documents.
storage: an HTML document, as it is meant to be displayed in the browser.

Conclusion

Automated document updating is a powerful way to distribute up-to-date information, such as service versions or usage patterns. As infrastructure grows, it may be the only way to keep support effort scaling sub-linearly with infrastructure size. That’s it for today. Thanks for reading!

Amazon EBS (Storage) Performance Limitation

2016-08-07T15:23:07+00:00

I have mentioned before that standard EBS storage in AWS runs on a credit system. Credits build up while a volume sits idle and are spent whenever the requested I/O rate rises above a predefined baseline. The details are covered in the AWS documentation. This limitation caught me off guard during the failed project I had worked on 18 months ago.

A few days ago I got the same problem while syncing data between 2 volumes, so I decided to capture what happened and share it here for anyone following this blog.

The CloudWatch read throughput graph (metric name: VolumeReadOps) looked like this during the sync:

I have marked the three areas that need explaining:

Credit consumption. The volume was in “Credit Consumption” mode here. While credits lasted, performance stayed close to the maximum Amazon allows for this instance and volume type.
Credits exhausted. Once credits ran out, this volume dropped back to its low baseline of 3 IOPS per GB, which works out to 150 for this 50GB volume.
Short idle period. A brief pause let some credits accrue, which allowed a small burst before performance fell back to the low baseline again. Here it is in more detail:

Conclusion? Plan your AWS workloads carefully. Always monitor your CloudWatch metrics, and be ready to move to Provisioned IOPS volumes if needed.

Note: This post was written by an AWS Certified Solutions Architect (Associate). Please always work with an expert when setting up production environments.

MySQL Monitoring with Amazon CloudWatch

2016-06-24T18:49:07+00:00

Amazon CloudWatch is the monitoring service for the Amazon Cloud. It supports both white box and black box monitoring of AWS-managed services, and it can also ingest custom monitoring data that you generate yourself.

This guide shows how to integrate a simple MySQL monitoring script with CloudWatch.

MySQL Monitoring

Suppose you want to track the number of active connections to your MySQL server and get a signal when that number approaches the limit set by max_connections in the configuration file. For portability, it’s also worth reporting the limit itself to the monitoring system, even though it rarely changes without someone editing the configuration.

MySQL exposes three numeric values that are useful here:

max_connections: a server variable, read with SHOW VARIABLES. It is the maximum number of connections allowed, either as set in the config file or the default.
threads_connected: a status value, read with SHOW STATUS. It is the number of connections open right now.
max_used_connections: also a status value from SHOW STATUS. It is a high-water mark of the most connections ever used since the last server restart, so it never decreases.

All three are valuable, and you may want them plotted together on a single graph.

Here is a simple shell script (mysqlmon.sh) that retrieves them:

#!/bin/sh

MYSQL_USER=xxxxx
MYSQL_PASS=xxxxx

case $1 in
	max_connections)
		mysql --user=$MYSQL_USER --password=$MYSQL_PASS -e "show variables like 'max_connections'" 2>/dev/null | grep -i "max_connections" | awk '{print $2}'
		;;
	max_used_connections)
		mysql --user=$MYSQL_USER --password=$MYSQL_PASS -e "show status like 'max_used_connections'" 2>/dev/null | grep -i "max_used_connections" | awk '{print $2}'
		;;
	threads_connected)
		mysql --user=$MYSQL_USER --password=$MYSQL_PASS -e "show status like 'threads_connected'" 2>/dev/null | grep -i "threads_connected" | awk '{print $2}'
		;;
	*)
		echo "Please provide one of max_connections, max_used_connections, threads_connected"
		exit 1
		;;
esac

With these point-in-time values available, the next step is to push them to CloudWatch.

Publishing to CloudWatch

To submit a custom metric, you need to supply a few details:

Timestamp: the time the reading applies to.
Metric name: free text that will appear in the CloudWatch console.
Namespace: also free text, but it must not collide with the reserved AWS namespaces, which only Amazon’s own services may write to.

To tie the data to a specific instance, you also need to pass the instance ID, which is easy to obtain from the instance metadata. A minimal publishing script (aws_mysql_pub.sh) looks like this:

#!/bin/sh

max_connections=$(/usr/local/bin/mysqlmon.sh max_connections)

max_used_connections=$(/usr/local/bin/mysqlmon.sh max_used_connections)

threads_connected=$(/usr/local/bin/mysqlmon.sh threads_connected)

cdate=$(date -u +%Y-%m-%dT%H:%M:00.000Z)

instanceid=$(curl http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null)

aws cloudwatch put-metric-data --metric-name MySQLMaxConnections --namespace "CustomMySQLMetrics" --dimensions="InstanceId=$instanceid" --value $max_connections --timestamp $cdate

aws cloudwatch put-metric-data --metric-name MySQLMaxUsedConnections --namespace "CustomMySQLMetrics" --dimensions="InstanceId=$instanceid" --value $max_used_connections --timestamp $cdate

aws cloudwatch put-metric-data --metric-name MySQLThreadsConnected --namespace "CustomMySQLMetrics" --dimensions="InstanceId=$instanceid" --value $threads_connected --timestamp $cdate

Note: Installing and configuring the AWS CLI is outside the scope of this guide. Amazon’s installation tutorial covers it, and you’ll need an access key pair ready for the configuration. The key should belong to either the root user (a poor choice for many reasons) or, preferably, an IAM user with full CloudWatch access.

How often should it run?

That depends on your needs. Timestamps have a one-minute resolution, but that doesn’t restrict you: if you send several readings within the same sampling period, CloudWatch can compute statistics over them, such as minimum, maximum, average, and sum.

For a one-minute interval, a basic cron entry is:

* * * * * /usr/local/bin/aws_mysql_pub.sh

Example CloudWatch Graph

The end result should look something like this:

Note: The display can be tidied up, for example by scaling the right Y axis to match the left. You can also create alarms on the graphed values, but CloudWatch currently cannot trigger an alarm by comparing one metric against another.

Looking at the graph, you may wonder about cost. CloudWatch is one of the few AWS services whose free tier doesn’t expire after the first year. Full details are on the pricing page.

That’s all for now. Thanks for reading!

Note: This text was written by an AWS Certified Solutions Architect (Associate). Always consult an expert when setting up production environments.

On SLAs and Uptime Guarantees

2016-06-15T10:55:07+00:00

In this article, I look at the practical side of the “multiple nines” reliability figures that vendors love to advertise, and explain why a true 100% uptime is not something you can achieve.

Definitions

SLA (Service Level Agreement) is a binding contract between a vendor and a customer. It is typically expressed as the percentage of a reference period (for example, a year) during which the service must work normally and deliver its intended output.

Uptime is the numeric part of that agreement, expressed either as a percentage or as a unit of time.

Note: These definitions don’t fully apply when a service is delivered from multiple infrastructure sets, such as different data centers serving different geographic regions. An outage in one region doesn’t mean every customer is affected, so a different calculation approach is needed. One option is to estimate the number of requests that went unserved during the outage, based on historical traffic data, and derive the availability figure from that.

How Many Nines?

To run the numbers yourself, you can use this calculator. As a rough guide:

99.9% allows about 9 hours of downtime per year.
99.95% allows just over 4 hours of downtime per year.
99.99% allows a little under 1 hour of downtime per year.
99.999% allows about 5 minutes of downtime per year.

Note: Amazon Web Services offers a 99.95% SLA for EC2 and EBS.

What Causes Downtime?

Downtime causes fall into two broad categories: planned and unplanned.

Planned downtime should be explicitly accounted for so that it fits within the SLA, possibly by using what’s known as an “error budget”. Unplanned downtime should also be anticipated and mitigated, though not every risk can be eliminated.

Planned outages, for example on a Linux web server, include:

Operating system updates: If a kernel upgrade requires a reboot, a single restart can cost roughly 2 minutes. Three such restarts in a year are enough to rule out 99.999% uptime.
Service updates: Depending on the service, downtime can be a few seconds, but it can also stretch to several minutes or even a significant fraction of an hour.

Unplanned outages range from hardware failures (such as a server dying) to software problems (such as a critical bug in the latest release). You should have mitigation plans for these scenarios to protect your SLA. Be aware, though, that in some situations the SLA simply can’t be preserved no matter how much effort you invest, and that possibility needs to be factored in too.

Note: Availability should also take into account the infrastructure between the end user and the service. Even if the service itself is perfectly available, a given user only experiences the uptime of their internet provider combined with that of every other component along the path. From that standpoint, offering an SLA above 99.9% may not bring much practical benefit to the average user, although it may still matter from a revenue-loss perspective.

Mitigating Unplanned Outages

When the unexpected happens, consider approaches like these:

Rollback procedures: Have a way to revert to the last known good software version, such as restoring a snapshot or re-running deployment scripts pointed at the previous version. The DevOps team should be able to do this without developer assistance, and ideally it should be automated so anyone can trigger it at any hour.
Failover architecture: Replicate the service across multiple nodes. This is the approach Amazon recommends for achieving an SLA above 99.95%.
Cold standby or rapid redeployment: Keep a standby ready, or have procedures to quickly spin up a new node with the failed node’s configuration. This can be fully automated and triggered by a health check.

Conclusion

A 100% SLA is not achievable in any real-world scenario. That said, a failover architecture, such as multiple nodes serving the same service across different availability zones or regions, can go a long way toward earning those extra nines.

Note: This article was written by an AWS Certified Solutions Architect (Associate). Always work with an expert when setting up production environments.

Instance Performance Monitoring in AWS

2016-06-10T18:50:07+00:00

AWS CloudWatch is Amazon’s built-in monitoring service. It collects metrics (including custom ones you define) and can trigger alarms when a metric crosses a threshold. It is the main tool for performance monitoring inside AWS.

This guide covers a typical scenario: you’ve deployed an application to the cloud and need to work out whether performance problems come from the application code or from limits that Amazon enforces.

Scenario

You’ve just started with AWS and are running applications on the free tier or on general-purpose T2 instances. These instances use a CPU credit system: credits let the instance burst through short load spikes, but once they run out, performance drops back to a low, fixed baseline. The details of the credit mechanics may seem obscure, but you still need to know whether your application can meet its service targets with this setup.

Prerequisites

If you’re responsible for meeting service targets, you need control over the cloud environment. Treat the following as red flags:

You don’t have full access to the AWS Console, so you can’t experiment with instance and storage volume types.
You don’t have access to the monitoring engine (CloudWatch).

If you’re only handed a bare cloud server and told to deploy the application and hit service targets, walk away. You’re being set up for failure through no fault of your own.

Internal Monitoring?

AWS doesn’t expose hypervisor metrics from inside the instance. This is standard industry practice: monitoring should be external. From within the instance, enforced resource caps show up only as general slowness or timeouts.

You can, however, retrieve static instance metadata from inside the instance:

[root@ec2instance ~]# curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
...

Monitoring with CloudWatch

Every instance and volume has a console tab with graphs of hypervisor-generated metrics: CPU utilization, network traffic and I/O statistics. By default, nothing OS-level (memory, disk usage) is included, so you must define custom metrics for those. Amazon provides helper tools, such as the CloudWatch Monitoring Scripts for Linux.

For an application on a performance-capped general-purpose instance, watch these metrics:

Instance metrics: CPU Credit Usage and CPU Credit Balance

Credit usage is a prime indicator of compute spikes.
An exhausted balance means the hypervisor throttled your instance’s CPU.
A constantly zero balance means a compute-intensive application is on the wrong instance type for the expected load. It can also point to a problem in the application itself. (The same behavior can be reproduced easily with a cryptocurrency miner or a fork bomb.)

Volume metrics: Average Queue Length

If the value doesn’t return to 0 over extended periods, read or write operations routinely exceed the IOPS limits Amazon enforces.
Read Throughput and Write Throughput give additional detail.

Ways to address I/O pressure:

Excessive reads: use instances with more memory so caching absorbs them (common for web servers).
Excessive writes: check for misconfigured logging and reduce the data written to the volume.
Otherwise: use larger volumes (for gp2, the IOPS quota scales with volume size) or switch to provisioned IOPS (io1) volumes.

Note 1: The CPU credit system applies only to “T” (general-purpose) instance types. Other types deliver their nominal performance at all times.

Note 2: IOPS limits apply only to EBS (SSD) volumes. Some instance types include local instance storage with performance closer to raw hardware limits. Newer HDD types (st1, sc1) are throughput-limited rather than operation-limited.

Advanced Monitoring

You can pull CloudWatch data with the AWS CLI, which is useful for deeper analysis such as examining specific time periods. Installing the CLI is out of scope here; you’ll need a key pair to use it.

1. Find the instance and volume IDs

You can also get these from the AWS Console, e.g. by instance name (if the Name tag is set):

aws ec2 describe-instances --filter Name=tag:Name,Values=myinstance.local

Or by instance ID:

aws ec2 describe-instances --instance-ids i-abcd1234

From the large JSON response, identify:

The instance ID
The volume ID(s), if there is more than one

2. Check the CPU credit balance

aws cloudwatch get-metric-statistics \
  --metric-name CPUCreditBalance \
  --start-time 2016-06-05T23:18:00 \
  --end-time 2016-06-06T23:18:00 \
  --period 3600 \
  --namespace AWS/EC2 \
  --statistics Maximum \
  --dimensions Name=InstanceId,Value=i-abcd1234

Example output:

{
  "Datapoints": [
    {
      "Timestamp": "2016-06-06T21:18:00Z",
      "Maximum": 287.74,
      "Unit": "Count"
    },
    {
      "Timestamp": "2016-06-06T09:18:00Z",
      "Maximum": 287.81,
      "Unit": "Count"
    }
  ]
}

This example requests the maximum, but minimum may give better insight. Experiment with parameters to find the best combination. Datapoints are returned unsorted, so import the JSON into another tool for analysis.

3. List the metrics available for each volume

aws cloudwatch list-metrics \
  --namespace AWS/EBS \
  --dimensions Name=VolumeId,Value=vol-abcd1234

Example output:

{
  "Metrics": [
    {
      "Namespace": "AWS/EBS",
      "Dimensions": [
        { "Name": "VolumeId", "Value": "vol-abcd1234" }
      ],
      "MetricName": "VolumeQueueLength"
    },
    {
      "Namespace": "AWS/EBS",
      "Dimensions": [
        { "Name": "VolumeId", "Value": "vol-abcd1234" }
      ],
      "MetricName": "VolumeReadOps"
    }
  ]
}

Note that the metric names differ from the graph titles in the AWS Console.

4. Check queue length per volume

aws cloudwatch get-metric-statistics \
  --metric-name VolumeQueueLength \
  --start-time 2016-06-05T23:18:00 \
  --end-time 2016-06-06T23:18:00 \
  --period 3600 \
  --namespace AWS/EBS \
  --statistics Sum \
  --dimensions Name=VolumeId,Value=vol-abcd1234

Example output:

{
  "Datapoints": [
    {
      "Timestamp": "2016-06-06T21:18:00Z",
      "Sum": 0.0643,
      "Unit": "Count"
    },
    {
      "Timestamp": "2016-06-06T09:18:00Z",
      "Sum": 0.0066,
      "Unit": "Count"
    }
  ]
}

Every operation passes through the queue on its way to or from storage, so the sum is never exactly 0. What matters is how long requests wait in the queue: small values indicate healthy performance. As before, datapoints are unsorted, so you may need to import the JSON elsewhere to analyze it.

Conclusion

AWS instances don’t behave like physical servers: the hypervisor enforces various resource limits. Always monitor instance behavior through CloudWatch.

Note: This article was written by an AWS Certified Solutions Architect (Associate). Always work with an expert when setting up production environments.

Cloud Distribution with AWS

2016-06-02T10:20:07+00:00

When Do You Need Cloud Distribution?

Think of cloud distribution as a form of caching. You need it once your website passes a certain threshold of users and outgoing data volume. The exact figure depends entirely on your setup:

You may serve large files, and sending them from your own server can use up your available bandwidth.
You may have many users and want to ease the load on your server by pushing part of it elsewhere.

This article shows how to do this with AWS CloudFront.

Creating the CloudFront Distribution

In the AWS Console, open CloudFront and create a Web distribution. On the creation page, first enter your domain name as the distribution origin (the primary source of your content).

Scroll down and pay attention to two more fields:

Price class: Using only the US and Europe edge locations is cheaper. Choose this if your customers are located only in those regions.
Alternate domain name: This is the address used to access content served by CloudFront. In this example, the subdomain is cdn (short for content delivery network).

Note: The form has many other configurable fields. The three above are the ones that matter for this setup. Every field has its use, so you may want to come back later to fine-tune parameters.

When you’re done, click Create distribution and wait about 10 to 15 minutes. When the status shows Deployed, the distribution is ready.

DNS Record

Next, make the alternate domain name reachable.

If you use Route 53: create an alias record pointing to the distribution.
If you manage DNS elsewhere: create a CNAME record that resolves the alternate name (e.g. cdn.brainware.ro) to the distribution domain name shown in the distribution’s properties (e.g. xxx1234.cloudfront.net).

Updating Your Website Links

With the AWS side configured, update your website so the resources you want distributed use the CDN-based URL.

There is no way around this with URL rewriting or .htaccess tricks, because those only act on requests that already arrive at your server. The goal is to avoid those requests altogether. With a cloud distribution, the first request for a resource is served by your own server, and subsequent requests (up to the TTL) are served by the cloud endpoints.

For WordPress sites, plugins can handle this for you, for example W3 Total Cache.

Practical Details

The full list of edge locations (data centers used by CloudFront) is on the AWS CloudFront Details page.
Check the CloudFront service limits, such as the number of requests served per second, even though a typical website is unlikely to exceed them.
See the detailed pricing on the AWS website. You are charged by usage only, not for simply setting up a distribution. Costs combine traffic volume and number of requests. Because the next bill can’t be fully predicted, setting up a Billing Alarm is advised.

Note: This text was written by an AWS Certified Solutions Architect (Associate). Always work with an expert when setting up production environments.

Amazon Certified Solutions Architect (Associate)

2016-05-27T19:50:07+00:00

As of today, I’m officially an AWS Certified Solutions Architect – Associate. Certification numbers are issued sequentially, so I can infer that I’m the 16,891st person worldwide to earn this title. Since certifications must be renewed every two years, though, many of those who certified earlier might have probably let theirs lapse.

Starting on the Path

I have opened a personal AWS account in the second half of 2014, around the time I was assigned to the project I wrote about in my previous post. I have barely used it, and still only use it for cloud backups. Without a professional reason to dig in, this path won’t take anyone very far.

The turning point came when I started doing DevOps work for a customer of my employer at the time. They had a fully configured AWS environment and gave me unrestricted access to that. Amazon recommends a year of hands-on experience before attempting the certification, and that is no arbitrary requirement: there’s a whole system to master before you can use it well.

Preparation

My certification journey began about a month ago, when I read this Reddit post. The timezone difference meant I couldn’t claim the free month of study that was on offer, but the idea had already taken some roots. Here’s what followed:

Confusion at first. For the first few days I didn’t know where to look or what to read. I didn’t want to pay for expensive courses and labs when I already had access to AWS environments where I could try everything myself or watch how it was done. I also wasn’t sure which certification to pursue (Architect, Developer or SysOps), especially since they overlap so much.
Course recordings. Through an “alternate channel” (don’t ask), I obtained recordings of courses from two years earlier, first for SysOps, then for Architecture. I decided to do the Architecture track first, get certified, and then move on to SysOps.
CloudAcademy. After finishing the recordings (about 14 hours), I signed up for CloudAcademy and used its free 7-day trial to the fullest. I worked through all the quizzes and the 157-question sample exam several times. Whenever I scored poorly on a topic, I opened the matching lab and followed along (for example, creating tables and indexes and working with DynamoDB). My goal was to score above 90% consistently on every quiz and on the final exam of that learning path.
Blogs, forums and apps. Once the trial ended, I searched blogs and forum posts for more material. Some recommended Android quiz apps, so I installed four. I returned the two paid ones within the two-hour refund window, but only after answering most, if not all, of their questions. By then I was already scoring above 90% consistently.
Whitepapers. I then read Amazon whitepapers on key services (EBS, VPC, Route 53). They cover details, such as EBS RAID configurations and special Route 53 record types, that aren’t found elsewhere, and the exam may include a few questions on them.
Practice exam. Feeling confident, I took the $20 practice exam and scored 90%. I then booked the real exam in the first available slot at the testing center.
The real thing. A little stressed, I sat the exam and passed with an overall score of 83%.

The Exam Itself

I can’t share any exam questions, and I don’t remember many anyway. Some people take exams specifically to memorize questions or record them with devices, then sell the “dumps” to buyers. That’s a viable business, I suppose, but I suspect Amazon has hundreds of possible questions and gives each test taker a random 60 out of way too many.

A few observations:

You may get at least one question from the sample questions on Amazon’s certification page, though those are on the easier side.
You may also get one or more questions from the practice exam.
Up to 33% of the questions fall into the “true or false: water is wet?” category, though they can still be hard if you’re not fully familiar with the topic.
Another 33% are somewhat trickier, along the lines of “what feels more wet, water or oil?”
The rest are frankly tough. They cover service internals or architecture decisions, such as which service to use for a given scenario. Often all or all but one option could legitimately work, but you must pick the one Amazon recommends.
About half the questions are multiple-answer, with no partial credit.

Logistics and Costs

The practice exam can be taken at home on your own computer. The real exam must be taken at a dedicated testing center. In my country there is only one such center for the entire country.

Total cost: $20 for the practice exam + $150 for the certification exam.

What’s Next

The future is still unwritten, but I’m considering the SysOps certification and one of the Professional-level certifications.

Until then, thanks for reading!

A Lesson on Failure: “Going Cloud”

2016-05-25T11:20:07+00:00

About a year ago, when I knew very little about Cloud Computing or Amazon Web Services, I was assigned, along with a few colleagues from the consulting company I worked for, to take an existing codebase from “alpha” to “production” and make sure it deployed smoothly to the Amazon cloud.

The customer wanted to go live in under three months and expected to handle tens of thousands of visitors clicking on banner ads and generating revenue. More likely, they were hoping for a good exit: handing the hot potato to someone else while walking away with a tidy profit. There is a term for these people, but this isn’t a meme post, so I’ll leave it there.

Starting the New Project

Things started off reasonably well. We worked through a long list of functional issues and ended up fixing more than 100 bugs and glitches. That was about all we could do, and it took us many long hours to get there.

We didn’t concern ourselves with the cloud configuration. We knew almost nothing about it, and the customer guarded the “keys to the kingdom” closely, approving instance and resource setups only case by case. They probably felt they were overpaying their Eastern European contractors (us), so I understand why they watched the Amazon bill so carefully. It was their home, their needs and their rules, and that was fine with us.

Overall, things went well from our side until performance testing began, but after that, everything fell apart. Within about a month, the project was taken away from us and then passed to different contractors, although I don’t think the product ever launched. These days the website domain is one of countless spam domains, abandoned and possibly resold a few times already.

Firefighting

What did the programmers and QA engineers see when things started going wrong?

The performance tests were returning poor results. We had a “performance environment” that closely mirrored production configuration, so we expected test results to reflect real-world behavior.

The metric falling short was Queries Per Second (QPS), a target previously agreed with the customer in our contract. Qualitatively, pages loaded slowly and timeouts were frequent. Simply navigating the site was painful because it was so sluggish and unpredictable.

We were better software engineers than lawyers (oh, the irony), so all we could do was try to improve the software stack:

Backend: The framework was Python Tornado, so we converted all “sync” operations to “async”. We were hunting for outliers that might be blocking the main server thread, but never properly identified any. The result was inconclusive: some tasks improved, others got worse.
Database: The database was MongoDB, so we spent a lot of time optimizing queries and removing unneeded indexes. Again, there was no meaningful improvement.

By the time the customer walked away, all of us were exhausted and frustrated.

What Went Wrong?

It took me about a year to understand what actually happened. It wasn’t the software stack, but the customer’s “Cloud” (AWS) setup. I only reached this conclusion after working with AWS day to day and while preparing for the AWS certification I’m currently pursuing.

In short, the customer relied heavily on the AWS free tier. They used t2.micro instances for development and testing, and most likely t2.small or t2.large for the Mongo database. I don’t remember the production web instance setup, but we probably had no proper load balancer. Load may have been distributed through DNS round robin: each instance had an elastic IP and Route 53 handled the balancing. Either way, all frontend instances used a single database node as their backend.

Here is the detailed explanation:

Burstable “T” instances: These are marketed as general purpose. Their baseline performance is limited, and extra capacity comes from a credit system: credits accrue while the instance is idle and are spent during load spikes. This works well for many non-production scenarios, but not when sustainable performance is required.
EBS storage: When creating instances with default storage settings (magnetic or general purpose), the volumes use the same credit model. During normal operation, Amazon throttles baseline IOPS (input/output operations per second) to a value well below what the underlying storage could deliver (details).
Credit exhaustion: Our performance tests burned through all accumulated EBS credits within the first few minutes, capping storage performance at a low baseline. On the database instance this appeared as high IO wait (the wa field in top). With our limited knowledge, we blamed poor application code rather than something external, i.e. the AWS-imposed limit.
MongoDB’s access pattern: MongoDB has an unpredictable storage access pattern by design, so there was likely never any burst capacity (idle-time credits) available in the first place. We often saw sluggish performance during normal daily work with no obvious cause.

What Could Have Saved the Day?

Two things might have helped:

MongoDB sharding on every frontend instance. This could have spread I/O across the whole cluster and likely mitigated the single-point-of-failure database design. On the downside, it might have caused memory swapping on t2.micro instances, cancelling out any benefit.
Provisioned IOPS for EBS storage. A dedicated DevOps engineer, which we didn’t have, could have monitored the “average queue length” metric and recreated the volume with the right provisioning to balance performance and cost.

Neither happened. Sharding was only mentioned when we were already in a downward spiral and it was too late to do anything meaningful.

Conclusion

When moving to the cloud, put a Cloud Architect on the team. It may be the single biggest difference between success and failure. Also, a cloud provider’s free tier is free for a reason, and that reason may be less obvious than what’s written in the marketing material.

Update: Two days after writing this post, I passed the AWS Certified Solutions Architect (Associate) exam. Please always work with an expert when setting up production environments.

The Second Level LPIC Certification (LPIC-2): How To

2016-04-29T10:27:07+00:00

As I mentioned in the first post of the certification series, I took the next step up in LPI’s certification ladder and earned LPIC-2 (verify). Was it easy? Was it hard? Let’s find out.

Introduction

LPIC-2 is awarded after passing two exams, but only if you already hold a LPIC-1 certification. In total, that means four exams to go from zero to LPIC-2.

Exam 201: Advanced system administration, including kernel and boot loader configuration, filesystems and troubleshooting.
Exam 202: Common services such as e-mail, HTTP, proxies and file sharing.

As with LPIC-1, the exams can be taken in any order, and the certification is granted once both are passed.

Logistics

I took both exams at a Pearson VUE test center (two different locations, in fact). Each exam normally costs around EUR 150. The second one was paid for by my current employer.

Preparation

Everything I said in the first post still holds. Books alone won’t get you through, and neither will light hands-on experience. The questions aren’t tricky: if you have actually used the software or worked through the scenarios, the answer comes quickly. If not, guessing won’t save you. I also noticed that some questions appear (seemingly at random) in both the level 1 and level 2 exams, so the boundary between “harder” level 1 topics and “easier” level 2 topics is very thin.

Two books proved useful. Keep in mind I’ve been doing hands-on Linux system administration for 13 years, so they served mainly as refreshers and helped me score higher. Without that knowledge base to connect the dots, there’s no way to pass, even if you knew every question in advance.

LPIC-2 Linux Professional Institute Certification Study Guide: Exams 201 and 202: I found it lacking in depth on the exam topics, and its practice questions were far easier than the real ones. It was printed before LPI’s last update, so it doesn’t cover everything needed. It is useful for exam 201, though, and I thought the PAM chapter was particularly well written.
Hexa Marathon Guide: LPIC-2: Linux Network Professional: Practice problems on LPI 201 and 202: It contains real exam questions (though not all of them, and I won’t ask how the author got them), but its formatting had serious shortcomings. Alongside the real questions, many others cover topics that no longer apply to current LPIC exams. On its own, it would only help people with a great photographic memory, but I found it excellent for structuring the knowledge I’d gained elsewhere and for getting higher scores.

Summary

How much did it cost to get here?

Preparation books (1, 2, 3): approx. EUR 80
Exams: four in total, EUR 480 (one was paid directly by my employer)
Other: the first two exams were taken in a conference setting, so there were additional transport and hotel costs for the trip to Brussels (excluded from this calculation).

I wouldn’t call that a lot of money to prove you know Linux. The Red Hat certifications (RHCSA, RHCE) cost at least twice as much, possibly more once you add their (likely mandatory) training. Some will say a certification doesn’t help you land a better job, but I’d argue it’s better to have something that might give you an edge over the competition.

Is LPIC-3 next? Maybe, but LPIC-3 is about specialization, and you must choose between mixed environments, security and virtualization. All of these are, more or less, out of my league for now.

That’s it. Thanks for reading!

Variable-Size Directories

2016-04-12T20:02:57+00:00

QA teams sometimes send requests like this:

For testing purposes, I need the /opt/test/xxx directory limited to 10 Megabytes. This directory is used by the abc123 application, which runs as user tester.

Can a directory’s size be limited in Linux? Yes. You need to:

Use the directory as a mount point for a size-limited storage device.
Set mount options so the specified non-root user has full access.
Disable SELinux (easy) or allow that particular user to access data on mount points (complicated).

Choosing the storage device

There are several options:

A loop device: a regular file used as a file system.
A logical volume (LVM): assuming your disk setup is already LVM-based and has enough free space for the new volume.
A newly attached storage device: for example, in a cloud environment like Amazon Web Services.

Each has pros and cons. The loop device is by far the simplest to implement, but its I/O performance is lower than the other options.

Example: loop device

# Create a directory to hold the image file
mkdir /var/spool/loop-files

# Create a 10 MB zero-filled image file
dd if=/dev/zero of=/var/spool/loop-files/xxx.img bs=1024 count=10240

# Find the first unused loop device
losetup -f
# /dev/loop2

# Associate the image file with the loop device
losetup /dev/loop2 /var/spool/loop-files/xxx.img

# Create the file system on the device
mkfs.ext4 /dev/loop2

# Update fstab to allow non-privileged user mounting
echo "/dev/loop2  /opt/test/xxx  ext4  defaults,noauto,user  0 0" >> /etc/fstab

# Mount as the non-privileged user
su - tester -c "mount /opt/test/xxx"

# Check the mounted devices
df

Note: The file system size cannot be set to an exact value because of block, superblock, journal, and other metadata overhead. The usable space after creating the file system also varies, so a “10 Megabytes” request can only be met to a close possible value, through some trial and error with the image size.

Follow-up: more than 8 loop devices

A default installation supports only 8 loop devices. To increase this:

# Tell the kernel module to support more devices
modprobe loop max_loop=64

# Create the devices, e.g.:
mknod -m 660 /dev/loop36 b 7 36

This should solve the problem. Thanks for reading!

On Companies and Career Paths – a Personal Overview

2016-03-26T18:04:40+00:00

Starting a career in computer science is hard enough, and landing with a good company is mostly down to luck. I won’t argue with that: you do need plenty of luck to get off to a good start. My aim is simply to show where the train you’ve just boarded might take you, or might not. Switching trains is always possible, but it gets harder with age.

At a high level, companies fall into two main categories:

Product Owners: Companies that build products for a real market. They either sell software or offer services to individuals or businesses through infrastructure they manage themselves.
Outsourcers: Companies that sell “work units”, directly or indirectly, to product owners or to other businesses.

There are nuances here, and government contractors and start-ups arguably deserve categories of their own.

What a fresh graduate can expect

Product Owner
- You can gain deep knowledge, though it’s limited to the company’s particular product area.
- Few corners get cut, and you may sometimes apply advanced topics in your daily work.
- Pay depends on the company’s profitability, and good bonuses are possible.
- The downside: it can be hard to move to another company, because you become “entrenched” in a narrow set of clever solutions that may not be useful elsewhere.
Outsourcer
- You learn many unrelated or loosely related topics, but none in real depth.
- Projects last from a few months to a year at most, leaving no time to learn to do things well.
- Much of the work is fixing other contractors’ code while adding technical debt of your own. You won’t care, because you’ll be gone in 3 to 6 months.
- You’re effectively paid by the billable hour (even if the company covers time between projects). Success earns you nothing extra, and failure usually just means another project assignment.
- On the plus side, moving to a similar role is easy: you know a lot of things, albeit shallowly, and you’re almost certain to use some of them at your next company.
Government contractor
- Expect a stable environment with few perks and sometimes below-average pay.
- Effectively, these are outsourcers with one large customer who, for all practical purposes, owns the product being developed.
Start-up
- Start-ups own their single product, and your experience depends largely on the initial budget and the market competition.
- They are mostly about losing time and money: time through work/life imbalance, and money through lower pay than comparable skills would earn at a “normal” company.
- This statistically evens out on the rare occasions when someone strikes a bonanza in the gold mine, meaning the product or business sells for good money.

What an experienced, mid-career person can expect

Product Owner: In some cases, it can fill the knowledge and experience gaps needed to reach a higher position, whether in architecture, management, or sometimes a better-paid role at another company.
Outsourcer: It may serve as a stepping stone to independent contracting or full entrepreneurship. Career progression in the style of a product-owner company is sometimes impossible because of how outsourcers are organized.
Government contractor: Usually offers little more than an easy path to retirement.
Start-up: Outcomes range from a premature end to your career, or being stuck in a dead end or set back for years, to the best case of getting rich and completely redefining your life.

So, where to?

Back to square one: luck. And prepare well, so that you can choose the train you want to ride rather than just taking the first one that passes through your station.

The First Linux Certification (LPIC-1): How To

2016-02-24T13:34:07+00:00

I recently earned my LPIC-1 certification (verify here). Here’s what I learned, in case you want to try it and aren’t sure where to start.

Introduction

LPIC-1 is granted after passing two exams:

Exam 101: more theory-oriented, with many general questions.
Exam 102: more practice-oriented, with an emphasis on scripting, networking and security.

You can take these exams in any order, and the certification is awarded automatically once you’ve passed both. Higher levels require the lower-level certification first. For example, any LPIC-3 certification requires passing a total of 5 exams, again in any order.

Logistics

The official route: book through a Pearson VUE test center. Buy an exam voucher (roughly EUR 150, subject to exchange-rate changes), then sit the exam on a computer in a controlled environment.

The alternative route: attend a conference that offers pen-and-paper exams. In Europe, you can register your seat on the LPI site and then attend that event. This is cheaper, and a good option if you’re going to the conference anyway.

I took both Exam 101 and Exam 102 at FOSDEM 2016 in Brussels, Belgium, on January 30 and 31.

Preparation

This isn’t a school-style exam. To have a real chance of passing, you need:

Lots of practical experience. Knowing a handful of shell commands won’t be enough. Two to three years in a DevOps or sysadmin role is a sensible point to start preparing.
Preparation books. This book helped me fill some knowledge gaps. Note that there is no official course or exam support for any LPI exam.
Targeted practice. Work on your weak areas. Many questions cover everyday topics, but some can only be answered if you’re familiar with that specific area.

About the Exams

I can’t share any exam questions (and I don’t remember many anyway), but roughly:

About half the questions cover normal day-to-day usage, which you should know if you currently work in a Linux-related role. They’re not easy, but they’re not impossible either.
The other half is split across “expertise areas,” and you’ll probably need to master a few of them to pass. For example, you may need to know which file to edit, and what to put in it, to change how name resolution behaves.
Questions carry different weights, so some contribute more to your score than others. I can’t tell which are which, and I doubt anyone outside LPI can.

Note: This is not a trivia exam. Every question is based on real-world scenarios you may have encountered at work. I suspect that lack of hands-on experience is the leading cause of failure.

Conclusion

I’ve already started preparing for level 2, which is certainly tougher. I haven’t yet decided on an exam date or how to take the two exams. I’ll post an update once I’m done. Thanks for reading!

Python: Crazy Interview Questions

2016-02-13T16:16:57+00:00

1. The Mutable Default Argument Trap

Imagine an interviewer hands you a slip of paper with this on it:

def foo(bar=[])

No context, just the question: what’s wrong with this code?

Strictly speaking, nothing. It’s valid Python (aside from the missing colon and function body), and a bug is only a bug when the code behaves differently from what the programmer intended.

Now let’s give the function a body:

def foo(bar=[]):
    bar.append(1)
    return bar

What would a typical Python programmer expect from calling foo() three times with no arguments?

foo()  # [1]
foo()  # [1]
foo()  # [1]

That’s not what happens. The actual output is:

foo()  # [1]
foo()  # [1, 1]
foo()  # [1, 1, 1]

Why? Python evaluates default argument values once, when the function is defined, not each time it’s called. The same list object is reused on every call. This differs from C/C++, and the behaviour resembles static variables there. That’s handy for caching or keeping state, but it’s rarely what you intended.

The usual fix:

def foo(bar=None):
    if bar is None:
        bar = []
    bar.append(1)
    return bar

2. Strings Are Immutable

The interviewer says: “Given a string S, replace the 3rd character with "A".”

You rightly note that the string might be empty or too short. “Ignore that,” they say. So you write:

def repl(S):
    if len(S) >= 3:
        S[2] = 'A'  # index starts at 0
    return S

The interviewer glances at it, ends the interview, and tells you HR will be in touch. (If an offer does come, you may want to think twice about working there.)

The problem: Python strings are immutable, so item assignment isn’t allowed. Instead, build a new string:

def repl(S):
    if len(S) >= 3:
        return S[:2] + 'A' + S[3:]
    else:
        return S

3. Late Binding in Closures

You’re shown this snippet:

F = [lambda x: i * x for i in range(5)]

If you haven’t fled the room yet, you’re already ahead of many. Here’s what the code is meant to do:

F is a list of functions.
It holds 5 functions.
Each takes one argument x and should return x * i, where i is 0 for the first function, 1 for the second, and 4 for the last.

So you’d expect:

F[0](2)  # 0
F[1](2)  # 2
...
F[4](2)  # 8

In reality:

F[0](2)  # 8
F[1](2)  # 8
...
F[4](2)  # 8

Why? The lambdas look up i when they’re called, not when they’re created. By then the loop has finished and i is 4, so every function uses that value.

The usual fix is to bind the current value as a default argument:

F = [lambda x, i=i: i * x for i in range(5)]

Honestly, few experienced Python developers would get this right on the spot, and asking it in an interview will likely leave candidates with a poor impression of your company.

That’s all for today. Thanks for reading!

Linux Performance Monitoring (an Introduction)

2016-02-07T20:31:07+00:00

1. Classification

“Performance” is a broad term, so it helps to split it into categories first. Two questions do most of the work:

What do we want to monitor? The whole system, or one particular application?
What kind of monitoring do we need? Basic statistics gathered by the kernel, or in-depth analysis?

Combining the answers gives four categories, each with its own set of tools:

	Stats (Counters)	Tracing / Profiling / Debugging
System-wide	mpstat, vmstat, iostat, netstat	tcpdump, dtrace, SystemTap
Per process	ps, top	strace, gdb

Notes:

netstat does more than report interface and protocol statistics. It can also monitor individual connections.
dtrace and SystemTap can also trace individual applications.

2. Stats Collection

Where do the tools in the first column get their numbers? The short answer is “the kernel”, but the actual interface is the /proc file system. For every process, the kernel exposes a stat file containing all the collected data:

# cat /proc/15588/stat
15588 (httpd) S 30837 30837 30837 0 -1 4202816 1845 0 0 0 2 1 0 0 20 0 1 0 381733777 671944704 4446 18446744073709551615 140071612739584 140071613080548 140735088901328 140735088899312 140071585579283 0 0 16781312 201344747 18446744071580587657 0 0 17 0 0 0 0 0 0

This format suits software far better than people: without the documentation, nobody can tell which number is which. Fortunately, a human-readable version exists:

# cat /proc/15588/status
Name:   httpd
State:  S (sleeping)
Tgid:   15588
Pid:    15588
PPid:   30837
TracerPid:      0
Uid:    48      48      48      48
Gid:    48      48      48      48
Utrace: 0
FDSize: 64
Groups: 48
VmPeak:   656708 kB
VmSize:   655684 kB
VmLck:         0 kB
VmHWM:     17920 kB
VmRSS:     17752 kB
VmData:    87788 kB
VmStk:       100 kB
VmExe:       336 kB
VmLib:     62692 kB
VmPTE:       872 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/63581
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 000000018c0046eb
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   ff
Cpus_allowed_list:      0-7
Mems_allowed:   00000000,00000000,...,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        166
nonvoluntary_ctxt_switches:     0

The kernel also provides system-wide statistics:

# cat /proc/stat
cpu  20887396 0 10349385 3015436747 407563 3166 1467937 0 0
cpu0 10678475 0 3883808 362454876 232413 3166 1390223 0 0
cpu1 1661441 0 1011415 378955537 31275 0 11441 0 0
cpu2 746371 0 330726 380344321 23829 0 11456 0 0
cpu3 519138 0 214791 380738302 16767 0 10813 0 0
cpu4 5386158 0 3986695 371475949 54473 0 12596 0 0
cpu5 682554 0 324926 380359994 18308 0 11663 0 0
cpu6 705281 0 343150 380407382 15057 0 10218 0 0
cpu7 507974 0 253871 380700384 15438 0 9525 0 0
intr 5529843084 126 2 0 0 0 0 0 0 1 0 ... (long run of mostly zeros omitted)
ctxt 5379451608
btime 1451035714
processes 6474048
procs_running 1
procs_blocked 0
softirq 11537062915 0 4052674276 583728909 992215181 16867459 0 13667315 1762309239 2086018 4113514518

There is no human-readable version of the system-wide /proc/stat, which is why you need the tools listed above to interpret it.

3. Tracing

Tracing is where things get more complicated.

Userspace tracing. You can monitor individual processes with signals such as SIGTRAP. This makes tracing possible with userspace-only tools, but the overhead is significant, so it is not advisable for performance-sensitive work. Still, strace and gdb remain the best tools for debugging and spotting erratic behaviour in individual programs.

Kernel-assisted tracing. More advanced tools such as dtrace rely on the kernel to report events for one process or for all of them. This is sometimes called “zero-cost” tracing, and it can be used in production environments.

dtrace and SystemTap both include scripting languages for handling the results when a “probe” fires, for example when a particular system call runs or execution reaches a certain line in the monitored program. This makes it possible to:

Process data in advanced ways (e.g. count specific behaviours)
Aggregate statistics
Perform “open-heart surgery” on programs, intercepting data flows or replacing data that would otherwise trigger undesired behaviour

Static vs. dynamic tracing. Tracing falls into two categories:

Static tracing looks at invariants such as system calls or well-known points within a feature. These are well documented and don’t change.
Dynamic tracing is like setting breakpoints on individual lines of code. Such breakpoints work for specific versions and may not apply to later versions of the program.

Example: a dtrace one-liner that prints a timestamp for every system call made by a given application:

$ dtrace -n 'syscall::: /execname=="myapp"/ { printf("%d", timestamp); }'

More one-liners are available here: DTrace One Liners.

That wraps up my introduction to performance monitoring. I hope you enjoyed it!

Python Cheat Sheet: Lambdas

2016-01-15T21:20:57+00:00

Do you think list comprehensions are a tricky topic? After reading this post, you may well change your mind about what counts as “complicated” in Python.

Lambdas

A lambda is a small function defined right where you need it, typically for light data-processing jobs. There are two main reasons to use them:

Speed: a compiler can optimize them, first by eliminating an actual function call, and then by opening the door to further internal code re-arrangements.
Brevity: you write less code.

Here is a lambda that squares a number:

g = lambda x: x ** 2    # e.g. g(3) yields 9

Lambdas are most often paired with the data-processing functions filter, map and reduce.

Filtering

To keep only part of an input, filter combined with a lambda works nicely:

print filter(lambda x: x % 2 == 0, xrange(0, 11))
# [0, 2, 4, 6, 8, 10]

Transformation

To apply a transformation to every element of the input, use map:

print map(lambda x: x * 2, xrange(0, 11))
# [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

Summarizing

To boil the whole input down to a single result, use reduce. Note that the lambda must accept two arguments: at first it receives the first two elements, and afterwards it receives the previous result plus the next element, until the input runs out.

print reduce(lambda x, y: x + y, xrange(0, 11))
# 55    # 1 + 2 + ... + 10

Now for the more serious material.

MapReduce in Python

print reduce(lambda x, y: x + y, map(lambda x: x * 2, xrange(0, 11)))
# 110    # 1*2 + 2*2 + ... + 10*2

The Intersection of Two Lists

def intersection(A, B):
    S = set(A)
    return filter(lambda x: x in S, B)

An answer like this probably isn’t what an interviewer expects. If you’re in an interview, make sure you can explain what’s happening before you write it down (for example, how a set works and what the complexity is in big O notation).

Removing Duplicates from a List

If the order of the remaining unique items doesn’t matter, calling set(A) is enough. Otherwise, we need to get creative:

def dedup(A):
    S = set()
    return filter(lambda x: x not in S and None == S.add(x), A)

A set’s add method always returns None, which evaluates to boolean False. To make the logic work, a small hack is needed (S.add() == None).

Also note that this solution keeps the first occurrence of each element and rejects the later duplicates. Solutions that keep the last or the n-th occurrence exist, but they’re much easier to write when the order of elements in the result doesn’t matter:

def dedup_last_unordered(A):
    S = set()
    map(lambda x: ((x in S and None == S.remove(x)) or True) and None == S.add(x), A)
    return S

As you can see, that version is pushing past what lambdas are meant for. Writing it the “normal” way is much clearer:

def dedup_last_unordered(A):
    S = set()
    for i in A:
        if i in S:
            S.remove(i)
        S.add(i)
    return S

A solution for the n-th duplicate can’t be written purely with lambdas, because it needs a counter that is assigned and incremented, and lambdas don’t allow that. A “normal” approach would use a dictionary that stores the counter as the value, but I think it’s best to stop here.

That’s all for today. Hope you enjoyed it!

Python Cheat Sheet: Dictionaries

2016-01-08T18:23:57+00:00

Python dictionaries are built on hash tables. They can hold arbitrary key-value pairs, and the keys and values don’t have to share a single data type. Dictionaries are unordered, meaning no ordering between elements is defined, so you can’t rely on any particular sequence when iterating over them.

Note: Unless stated otherwise, all examples use Python 2 syntax.

Creating an empty dictionary

D = {}

Initializing a dictionary

D = dict((i, False) for i in xrange(1, 11) if i % 2 == 0)   # Python 2
D = {i: False for i in range(1, 11) if i % 2 == 0}          # Python 3

Note: the second form is known as a dictionary comprehension.

Iterating over all items

for i in D:
    print i, D[i]   # key, value

Or alternatively:

for k, v in D.iteritems():
    print k, v      # key, value

Checking whether a key exists

if k in D:
    print k

Adding elements

D[key] = value

Note: if the key already exists, no error is raised; its value is simply overwritten.

Removing elements

del D[key]

Removing duplicates from a list with a dictionary as a lookup table

D = {}
unique = []
for i in L:
    if i not in D:
        D[i] = True
        unique.append(i)
L = unique

Note: this keeps the first occurrence of each element and preserves the original order. To modify the list in place instead of rebinding the name, use L[:] = unique as the last line.

A set is a better fit for this job. If the original order of the list doesn’t matter, you can use the shorter form list(set(L)).

That’s all for today. Thanks for reading!

Note to Self: HTTP Server KeepAlive Timeout

2015-12-28T11:41:07+00:00

Note: I have ran into this problem on Apache httpd, but it could just as easily affect any other HTTP server.

HTTP KeepAlive

The idea behind “KeepAlive” is straightforward: instead of opening a new connection for every request, the browser opens one connection to the server and reuses it to fetch many resources, such as the main page, stylesheets, JavaScript files, and images. In theory, this cuts page load times and gives visitors a better experience. So far, so good.

Server Configuration

Apache httpd manages this feature with three directives, all part of the core module:

KeepAlive: on or off (default: on)
KeepAliveTimeout: a number of seconds (default: 5)
MaxKeepAliveRequests: a positive integer (default: 100)

Apache httpd has no way to serve KeepAlive and non-KeepAlive requests side by side (for instance, allowing up to 100 KeepAlive requests per connection and handling anything beyond that differently). You have to decide on the behavior up front.

The Traffic

Time for some math. Starting from the MaxClients value (default: 256), how many new clients per second can the server handle without hurting the user experience? You might answer “MaxClients,” but let’s not jump to conclusions. A few factors come into play:

The time between opening a connection and the KeepAliveTimeout expiring. With default settings that’s at least 5 seconds, and in typical real-world conditions probably 7 to 10 seconds.
Fluctuations in traffic (spikes).
Apache httpd’s own overhead, such as spawning new processes to handle incoming connections.

Things get complicated quickly. But if we assume a typical 7 seconds per client, during which a server process is either busy or waiting for data, and that all clients behave roughly the same way, then the default MaxClients of 256 supports a steady rate of 256 / 7 ≈ 36 new requests per second. Any spike above that will cause slow page loads and a poor user experience.

Better Planning

If turning KeepAlive off isn’t an option, plan around the worst spike you expect to handle (for example, 2x or 3x the average during your busiest period). If your busy-period average is 50 requests/second, the server should be able to cope with 100 or even 150 requests/second without degrading the user experience. If the hardware is already pushed to its limit, you’ll need other approaches, such as multiple servers or load balancers (which I won’t go into here).

For a peak of 150 requests/second with the default 5-second KeepAliveTimeout, you’d likely need to raise MaxClients to around 1,000. If you’re using the prefork MPM, ServerLimit needs to be raised as well.

Conclusion

Don’t rely on the defaults. Begin your parameter calculations with the outcome you want to achieve. On hardware that is short on memory, disabling KeepAlive is the best way to squeeze out extra performance. And don’t assume your hardware can deliver what your calculations demand; when something fails or was planned badly, treat it as a chance to learn.

That’s all for today. Have fun!

Python Cheat Sheet: Lists

2015-12-26T13:55:57+00:00

Lists are ordered, linear collections that offer constant-time element lookup by index. They can grow or shrink, be searched, and be sorted (including with a custom comparison function). They also aren’t limited to one data type, so a single list can hold a mix of values. Python lists are zero-indexed.

Note: Unless stated otherwise, all examples use Python 2 syntax.

Creating a pre-filled list of 10 numbers (all zeros):

A = [0] * 10

Creating a pre-filled list of 10 numbers (powers of 2):

A = [2**i for i in xrange(0, 10)]

Note 1: xrange in the example above produces the values 0 through 9, inclusive.

Note 2: This syntax is called a list comprehension.

Looping over every element (read-only):

A = [0, 'a', {'b': 'c'}]
for e in A:
    print e

Looping over every element (read/write):

A = [0, 'a', {'b': 'c'}]
for i in xrange(len(A)):
    print i, A[i]

Appending elements to the end of a list:

A = [0, 1, 2, 3]
A += [4]

Note: The append method can be used for the same purpose.

Inserting elements:

A = [0, 1, 2, 3]
A.insert(0, -1)  # position, value

Note: The insert call above places the new element at the front of the list.

Finding elements in a list:

A = [0, 1, 2, 3]
A.index(2)

Note: index performs a linear search for the given value. If the value isn’t in the list, a ValueError is raised.

Removing elements from a list:

A = [0, 1, 2, 3]
A.remove(2)  # by value
del A[1]     # by index

Using a Python list as a stack:

A = []
A.append(1)
A.append(2)          # always add to the end
stacktop = A.pop()   # returns 2, the most recently added element

Note: pop raises an IndexError if the list is empty.

Using a Python list as a queue:

A = []
A.insert(0, 1)
A.insert(0, 2)    # always insert at the front of the list
elem = A.pop()    # returns 1, the earliest element added

Note: For an optimized implementation of both stacks and queues, take a look at collections.deque.

That’s all for today. Have fun!

Setting up a Secure Linux Server

2015-12-10T10:18:07+00:00

No server is ever completely secure. Whenever you expose services to the public, someone will eventually try to attack them, whether through privilege escalation or denial of service. What you can do is reduce the odds that an attack succeeds, or at least limit the damage it causes.

This article doesn’t cover advanced security techniques. It covers practical, common-sense measures that will stop most opportunistic attackers and automated bots. Let’s begin.

1. Set Up iptables

You might wonder why you need a configured firewall if your server runs only three services and those are the only ones listening on sockets.

The answer: a firewall is always necessary. A “deny everything, allow exceptions” policy makes any rogue service an attacker manages to plant through privilege escalation useless, since it can’t be reached from outside.

On RedHat/CentOS 6 systems, the firewall rules live in /etc/sysconfig/iptables. A typical restrictive configuration looks like this:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22,80,443 -j ACCEPT

This configuration accepts new TCP connections to ports 22 (SSH), 80 (HTTP) and 443 (HTTPS). It permits all outgoing traffic along with the replies to it, and allows ICMP (ping). Everything else is blocked.

2. Secure /tmp

/tmp is the best-known place on a Linux server where anyone can create files. In a default installation it’s just another directory on the root partition, which leads to problems such as:

An attacker can create huge numbers of files in /tmp, filling the disk or, worse, exhausting the file system’s inodes.
Attack scripts can be dropped into /tmp and executed, hoping to exploit a kernel bug or a flaw in a daemon running as root.
If a scripting language isn’t powerful enough for the attack, the attacker may even bring a compiler and build a binary.

The fix is to give /tmp its own mount point and apply special mount options: noexec or, if that isn’t possible, nosuid. The first prevents anything from being executed in /tmp, even by the superuser. The second disables the “suid” bit on executables, blocking a class of privilege escalation techniques that rely on such scripts or binaries.

An example /etc/fstab entry:

/dev/xvdf2	/tmp	xfs		defaults,nosuid,noexec	1 2

3. Isolate User Files

User files may live in /home (a classic Linux setup) or somewhere under /var (typical for a web server). Either way, they should sit on a dedicated mount point, for several important reasons:

Disk quotas can only be enforced per mount point, so a dedicated one makes setup simpler.
Even without quotas, no user can cripple the server by filling the disk or using up all the inodes.
Hard links to system files become impossible. Otherwise, a rogue user could keep a copy of a vulnerable suid binary for their own use, and the sysadmin would never know.
Mount options such as noexec or nosuid can be applied to user files alone without affecting the rest of the system.

Password login is insecure: passwords are easier to steal and guess than private keys. On large installations with many nodes, remembering and typing passwords for every login is also impractical and tedious.

Disabling password authentication completely shuts out brute-force bots, which then have no theoretical chance of getting in.

This one arguably belongs at the top of the list. In production, there should be no direct root access, by password or by SSH key, without exception. All administrative work should be done through sudo from a dedicated, ordinary user account that is reachable only by SSH key.

In lower-level environments or fully private servers, this rule can safely be relaxed.

6. Other Measures

Enable SYN cookies if they aren’t already on by default.
Use a fail2ban-style tool.
Consider using SELinux.

That’s all. I hope you found it useful!

Crazy DevOps Interview Questions (2)

2015-11-24T17:41:57+00:00

Note: This is the second article of the series. The others are: part 1, part 3 and part 4.

Question 1

Imagine you run these commands:

# cd /tmp
# mkdir a
# cd a
# mkdir b
# cd b
# ln /tmp/a a

What happens?

A candidate might note that this hardlink would create a circular reference. That’s right, but it’s only part of the story. How would the operating system or file system actually deal with the command?

A command-line expert might brush the question off by saying hardlinks to directories aren’t allowed. Another might notice that ln is missing the -d option, so the command fails before anything else. Both are correct, but neither is the full answer an interviewer is looking for.

A complete answer should cover the following:

Most file systems forbid directory hardlinks, but not all. The best-known exception is HFS+ (Apple OS X).
By definition, hard links are directory entries that point to the same inode. The inode holds a link counter, and deleting a hard link only removes the file itself once that counter reaches 1.
Directory hard links aren’t inherently dangerous, so that’s not why they’re disallowed. The real problem is the circular reference described above. Graph theory can solve it, but doing so takes a lot of CPU and memory.
The decision to forbid directory hard links was made because of that cost, which grows as the file system gets larger.

A thorough answer like this is what you’d typically expect in an interview at one of the “Big Four” technology companies.

Question 2

Suppose you run this command on Linux:

# mv /home/testuser/magicalapp /usr/local/

What does it do, and how does it work? If it doesn’t work, why not?

The answer comes down to how mv basically works. It doesn’t necessarily move any data. Most of the time it just creates a new directory entry at the destination that points to the same inode as the source, and the source entry is removed in the process.

When the source and destination are on different file systems, mv generally behaves like this:

# cp -r /home/testuser/magicalapp /usr/local/
# rm -rf /home/testuser/magicalapp

Note 1: Very old versions of mv couldn’t move directories between file systems.

Note 2: The PowerShell equivalent (move item) works differently. It can move files across file systems but has no recursion option.

It’s a good interview question, don’t you think? I’m not sure how many candidates would get it right.

Question 3

What does each field in the CPU usage line of top mean (for example us, sy, wa)?

The general answer is that these fields are percentages that add up to 100. Each one shows the share of CPU time spent on a particular kind of work:

us: user-space programs, meaning ordinary processing that isn’t otherwise categorized.
sy: kernel-mode processing, i.e. everything the kernel does.
ni: user-space programs running with a modified priority via nice.
id: idle time, when the CPU effectively runs NOP instructions.
wa: time spent waiting on a resource such as a disk or the network. It doesn’t mean busy-waiting. It’s CPU time spent sending or receiving raw data to or from external devices.
hi: time spent servicing hardware interrupts raised by external devices, such as a physical network card. A high value here can point to a DDoS against the network interface.
si: time spent servicing software interrupts. User programs ask the kernel for services through software interrupts, though there can be other uses as well.
st: stolen time (not applicable on physical machines). This is CPU time taken by the hypervisor in a virtualized environment.

That wraps up the second installment of the interview questions. I hope you enjoyed it!

A Bit of PowerShell Scripting

2015-11-15T22:10:24+00:00

Almost every large deployment ends up with a stray Windows machine somewhere. Actually, that’s not quite right: plenty of organizations run on Windows exclusively. The reason is simple: most closed-source SDKs only work on Windows, so to use their features you have to build a Windows service. Luckily for anyone who loves the command line, Microsoft provides an outstanding toolset in PowerShell.

When writing PowerShell code, keep two things in mind:

Script files should use the .ps1 extension instead of .bat. You will usually also need to change a security setting (the execution policy) before the system lets you run them.
Most tasks can be done in a single line.

If you come from Linux scripting, some behavior may seem strange. For example, you can’t directly move a directory; you have to copy it recursively and then delete the original. Still, the work gets done. Below are a few basic tasks you may need at some point.

1. Download a zip file from an HTTP server and extract it

$source_file = "http://internal.repo/package/module.zip"
$tmp_folder = "C:\Temp"
$dest_folder = "C:\Program Files\Internal\Application"

$source_file_name = [io.path]::GetFileNameWithoutExtension($source_file)
$dest_zip = "$tmp_folder\$source_file_name"

New-Item -ItemType Directory -Force -Path $tmp_folder
Invoke-WebRequest $source_file -OutFile $dest_zip

New-Item -ItemType Directory -Force -Path $dest_folder
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory($dest_zip, $dest_folder)

Remove-Item -Path "$dest_zip" -Force

### No -Force for the temp folder
Remove-Item -Path "$tmp_folder"

2. Register and start a new service

New-Service -BinaryPathName $exePath -Name $serviceName -DisplayName $serviceName -description $serviceDescription -StartupType Automatic
Start-Service $serviceName

3. Stop and delete a service

$existingService = Get-WmiObject -Class Win32_Service -Filter "Name='$serviceName'"
if ($existingService)
{
	Stop-Service $serviceName
	### We should sleep for a few seconds to give time for the service to stop
	Start-Sleep -s 5
	$existingService.Delete()
	### When upgrading an existing service, if we are to put a new binary for the same name, we should also sleep a few seconds here
}

4. Open a firewall port (TCP)

$port = "30000"
$existing_fw_rule = netsh advfirewall firewall show rule name="access to port $port"
if ($existing_fw_rule -match "No rules match")
{
	netsh advfirewall firewall add rule name="access to port $port" dir=in action=allow protocol=TCP localport=$port
}

That wraps up today’s PowerShell tips. Enjoy!

AWSome Day Bucharest 2015

2015-11-02T23:12:57+00:00

Less than a month ago, I attended an Amazon event called “AWSome Day”. It was far from boring, and it clearly reflected the company’s image: polished, sophisticated and focused on the customer.

The day had two parts. The first was lighter and business-oriented, while the second, after lunch, went deeper into the technical side. Still, it’s worth remembering one thing: when a product costs you nothing (or appears to), you are the product.

Amazon covered the venue, the talks, lunch and refreshments, and in return delivered what amounted to a very classy sales pitch.

The services covered in depth, or shown in the hands-on demo, were:

Elastic Compute Cloud (EC2)
Simple Storage Service (S3)
Content Delivery Web Service (CloudFront)
The Elastic Load Balancer
Relational Database Service (Amazon RDS)

Other services were mentioned only briefly:

All in all, it was a packed day. The complete presentation is on SlideShare (link) if you’d like to go through it again. Many slides were skipped because of time constraints, and the deck seems to describe even more AWS products.

When thinking about the Cloud, and Amazon Web Services in particular, there are a few important points to keep in mind. As a product suite, AWS is a couple of years ahead of its competitors in terms of features and interface. Note, however, that there is competition: several companies offer similar services, sometimes at a lower price or with better cost efficiency than Amazon.

In my opinion, AWS is the right choice for:

Experiments, test deployments and temporary solutions;
Non-production environments whose lifespan and makeup keep changing (e.g. development and/or QA);
Unusually heavy loads lasting only a few days a year (e.g. traffic peaks on Black Friday or during Christmas sales);
Medium- and long-term data storage (backups).

If your environment is fixed and won’t change or scale significantly for a year or more (e.g. a production website and/or a SaaS product), AWS may not be the only cost-effective option to consider.

Conclusion: if a similar event is held near you, go and reserve your seat! Just remember that you are attending a sales pitch.

Crazy DevOps Interview Questions (1)

2015-10-28T22:09:57+00:00

Note: There are three more episodes in this interview series: part 2, part 3 and part 4.

Nobody enjoys interviews. Well, maybe it depends on the day or on what side of the table you happen to be in.

If an interviewer asks you any of the questions below, then they either have read this post or had drawn on the same sources I did. Either way, you’re now a step ahead.

Question 1

You log into a node and find this:

# ls -la /bin/chmod
rw-r--r-- 1 root root 56032 ian 14  2015 /bin/chmod

The chmod binary has lost its execute bit. Can this be fixed?

Yes, most of the time. Remember how Linux starts executables: through a system loader, something like ld-linux. Let’s find out which one chmod uses:

# ldd /bin/chmod
	linux-vdso.so.1 =>  (0x00007ffdf27fc000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb11a650000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb11aa15000)

Now call the loader directly and let it run chmod for you:

# /lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod

Neat, right? The obvious follow-up is: “What if the loader’s own permissions are wrong?” Not every filesystem problem has a simple command-line fix. You may have to mount the filesystem from a different installation (for example from a Live CD, or by attaching the virtual disk to another healthy node) and repair it from there.

Question 2

A running process hangs while reading from a socket. Variations: it may be deadlocked on a read or on a write, or it may keep receiving EAGAIN and sit in a livelock. How do you close the bad socket / file descriptor without killing the process?

First, where can we see the files and sockets a process has open, from the outside? In /proc:

# ps -e | grep myprocess
  979 pts/0    00:00:00 /usr/bin/myprocess

# ls -la /proc/979/fd
total 0
dr-x------ 2 root   root  0 oct 22 21:37 .
dr-xr-xr-x 9 nobody dip   0 oct 22 21:37 ..
lrwx------ 1 root   root 64 oct 22 21:37 0 -> /dev/null
lrwx------ 1 root   root 64 oct 22 21:37 1 -> /dev/null
lrwx------ 1 root   root 64 oct 22 21:37 2 -> /dev/null
lrwx------ 1 root   root 64 oct 22 21:37 3 -> socket:[9814]
lrwx------ 1 root   root 64 oct 22 21:37 4 -> socket:[9816]
lrwx------ 1 root   root 64 oct 22 21:37 5 -> socket:[9817]
lrwx------ 1 root   root 64 oct 22 21:37 6 -> socket:[9818]
lr-x------ 1 root   root 64 oct 22 21:37 7 -> pipe:[9825]
l-wx------ 1 root   root 64 oct 22 21:37 8 -> pipe:[9825]
lrwx------ 1 root   root 64 oct 22 21:37 9 -> socket:[9827]

We now have the pid and the fd (file descriptor). You can’t remove or manipulate an open socket from outside the process, so we have to get inside, by using gdb. Assuming pid == 979 and fd == 9:

# gdb
  ... (lots of output) ...
(gdb) attach 979
Attaching to process 979
  ... (lots of output) ...
(gdb) p close(9)
$1 = 0
(gdb) detach
Detaching from program: /usr/bin/myprocess, process 979
(gdb) quit

Tricky, wasn’t it? An interviewer who asks this is probably looking for a real expert.

Question 3

How would you deal with this situation?

# ls
bash: fork: retry: Resource temporarily unavailable

In short, the system has run out of pids, and this shell is the only access you have left.

Note: This is the classic fork bomb question that companies such as Google sometimes ask. They aren’t after one definitive answer (none exists); they want to see how you approach the problem.

Realistically, the system may be so overloaded (many processes compete for CPU time) that a hard reboot is the only real fix. In an interview, though, you’re usually expected to give a “clever” answer with shell scripting tricks.

Before writing that answer, three sub-problems need solving beforehand:

How do you find the process name? In a fork bomb, the same name often occupies all the available pids.
Once you have the name, how do you identify every matching pid?
Once you have a (very long) pid list, how do you run the kill command?

Interviewers rarely ask about the first; the name is usually “known,” or can be found by trial and error in /proc/. Even that isn’t trivial, since ls and cat are external commands and can’t be launched. Shell builtins could do it, though:

# read t < /proc/1091/cmdline ; echo $t
/usr/sbin/bash

You can also iterate over the pids with builtins only. (Note: while spawns a process, so it can’t be used here.)

# for p in /proc/[0-9]* ; do echo ${p#/proc/} ; done
 ... long pid list, one per line ...

Now to the process killing part. A plain kill -9 is easy but doesn’t solve anything: each freed pid just lets another process spawn. So you need two passes:

Freeze all the offending processes (kill -SIGSTOP).
Then remove them (kill -SIGKILL).

The code looks like this. (Note: the double-bracket [[ ... ]] form is a builtin, while the single-bracket [ starts a new process.)

# for p in /proc/[0-9]*/cmdline ; do read cmd < $p ; if [[ $cmd = 'bomb_process_name' ]]; then t=${p%/cmdline} ; kill -SIGSTOP ${t#/proc/} ; fi ; done
# for p in /proc/[0-9]*/cmdline ; do read cmd < $p ; if [[ $cmd = 'bomb_process_name' ]]; then t=${p%/cmdline} ; kill -SIGKILL ${t#/proc/} ; fi ; done

Notes:

The string comparison relies on whitespace around the variable, the equals sign and the string. a = "xxx" is a comparison, while a="xxx" is an assignment in Bash.
= and == behave identically in this comparison.
The # and % operators remove the shortest match. Since we match a fixed string rather than a pattern, ## and %% (longest match) would give the same result.
cmdline in /proc/<pid> is not a reliable source for the process name. It works against simple bash fork bombs or sloppy code, but a determined attacker can randomize it, for example with snprintf(argv[0], size, "randomstuff").
Other sources (/proc/<pid>/comm, /proc/<pid>/status) can also be overridden, for example with prctl(PR_SET_NAME, (char*) "randomstuff"). Yes, yes, I know about const_cast.

Follow-up questions:

What if the bomb is a bash fork bomb and you’re in a bash shell? Wouldn’t the first loop kill your own shell too?
What if kill isn’t a builtin and only the external version exists?

Not everything can be solved cleanly. /proc/self is a symlink to your own pid, and readlink can read it, but that’s an external command. Luckily, the builtin echo $$ gives you your own pid, which you can exclude in the if condition above.

With kill only available as an external command, I’m not sure the problem can always be solved.

If a normal user triggered the bomb, you can immediately append a limit to /etc/security/limits.conf:

echo "forkuser hard nproc 20" >> /etc/security/limits.conf

This stops the active part of the bomb: even if you run exec ... in the terminal, the fork bomb can’t claim the pid. Running the second kill command will then probably free enough pids for normal operation to resume.

If the user is root, or has certain capabilities such as CAP_SYS_ADMIN, this option doesn’t work. Collecting all the pids in a shell variable and running exec /usr/bin/kill then locks you out when it’s time for the second command of the pair. Cleaning up the now stale pids becomes the hard part.

Note: There’s no guarantee that the pid released by the vanished shell stays free for another process under your control. Other daemons on the machine may compete for the same resource.

Still, you can try:

cron: a best-effort approach, since there’s no guarantee cron can run.
inittab, if the system has one (newer distributions don’t). Put the kill commands into a file executed on switching to a runlevel (e.g. 4), then force the change with telinit. Again, no guarantees.

On machines you can physically access, you usually have several login screens available via key combinations like ALT + digit, which lets you get back in. On machines reached through virtual terminals (ssh, xterm, GNOME Terminal), that isn’t possible, so the question has no real answer there.

That’s it for today’s “crazy” interview questions. Hope you enjoyed them!

What is DevOps?

2015-10-25T21:37:57+00:00

DevOps: Some people dismiss it as hype or as the sysadmin role under a new name, while others see it as just another skill you pick up from a course or a degree. It is none of these. In my view, it is a blend of skills: some come from school, some from formal training, and the rest from hands-on work day to day.

Why isn’t it simply a renamed SysAdmin role?

A “classical” SysAdmin generally doesn’t rely on sophisticated automation tools. The typical routine means moving from one node to the next, applying configurations, installing software, and repairing whatever is broken.
A standard SysAdmin role usually involves no real programming beyond small scripts that run on individual nodes.
A “classical” SysAdmin typically works with physical machines, and much of the day goes to storage problems (rebuilding file systems, restoring from backups), tracking down hardware incompatibilities, and applying patches.

Why do serious, formal coding skills matter for a DevOps engineer?

The current industry trend is to describe infrastructure as Code. Today you can create whole environments (for example on AWS), complete with correctly sized hardware and pre-installed services, using nothing but Code. Automation tools like Ansible or Chef make all this possible. It takes more than basic scripting, though it doesn’t call for complex algorithms either.

As noted above, DevOps is defined by Automation, since working with nodes is no longer almost entirely manual. Commands now run against many nodes simultaneously, and automated monitoring tools raise alerts and sometimes trigger corrective actions, such as scaling resources or adding new nodes to a cluster.

Who needs DevOps skills?

In my opinion, any organization running a computing environment so large that scaling it through classical SysAdmin work is no longer practical or cost-effective. The same goes for any organization planning to move its infrastructure to the Cloud. Because this is a relatively new position that doesn’t fit neatly alongside Devs, SysAdmins or QAs, it can be hard to define job requirements, map out a career path, or even settle on compensation, but that is beginning to change.

That was my introduction to the DevOps concept. Thanks for reading!

Simple E-Mail Config with Postfix and Cyrus IMAP

2015-10-23T08:42:57+00:00

This is a quick, no-frills guide to getting email running on a minimal Linux installation (Red Hat, Fedora or CentOS), such as the one you get when you launch a node in AWS from a predefined AMI (Amazon Machine Image).

Red Hat-family distributions ship with Postfix as the MTA (Mail Transport Agent) and Cyrus IMAP as the MDA (Mail Delivery Agent), so this guide focuses on those two. Once both packages are installed, the work is mostly in the configuration files.

Key Configuration Files

/etc/postfix/main.cf: the Postfix configuration. It controls which emails are accepted and how they are handled.
/etc/imapd.conf: Cyrus IMAP settings, such as where mailbox passwords are stored and which authentication mechanisms POP/IMAP support.
/etc/sasl2/smtpd.conf: SMTP authentication settings, needed if you want to run a mail relay. (This file may instead be at /usr/lib/sasl2/smtpd.conf.)

There is also /etc/cyrus.conf, which normally has sensible defaults after installation, so you probably won’t need to touch it. It defines the supported protocols (POP3, POP3S, IMAP, IMAPS) and the location of the LMTP socket, the interface between Postfix and Cyrus IMAP.

Step 1: Configure Postfix

Add these essential settings to /etc/postfix/main.cf:

mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp
virtual_transport = $mailbox_transport
virtual_mailbox_domains = /etc/postfix/virtual_domains
virtual_mailbox_maps = hash:/etc/postfix/virtual_maps

What this does: Mail addressed to the domains listed in virtual_domains is accepted for further processing. Postfix then looks up the destination mailbox in virtual_mailbox_maps; if no mailbox is found, an error goes back to the sender. Actual delivery happens through the socket defined in mailbox_transport.

Example file contents:

# cat /etc/postfix/virtual_domains
brainware.ro

# cat /etc/postfix/virtual_maps
john.doe@brainware.ro	brainware.ro/john.doe

The virtual_maps file must be converted into a hash database with postmap:

# postmap hash:/etc/postfix/virtual_maps

Step 2: Check the Cyrus IMAP LMTP Socket

Confirm the Cyrus daemon is listening on the right LMTP socket (it should be by default):

# cat /etc/cyrus.conf | grep lmtp
lmtpunix	cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1

Note: At this point you may want to either disable SELinux (easy), or add the required SELinux rules (harder) so that Postfix and Cyrus IMAP can communicate over the socket.

Step 3: Configure Cyrus IMAP

For corporate-style addresses (name.surname@domain), the mailbox namespace separator has to be something other than the dot. Change it in /etc/imapd.conf:

unixhierarchysep: 1		# changes the separator to "/"

Other settings to review in imapd.conf:

admins: cyrus					# user allowed to run Cyrus admin commands
sasl_mech_list: plain login cram-md5 digest-md5 ntlm gssapi		# enable all mechanisms
sasl_pwcheck_method: auxprop	# look up passwords in /etc/sasldb2
virtdomains: userid				# derive the domain from the mailbox name (user@domain)
allowanonymouslogin: no			# security
allowplaintext: yes				# allow older login methods (e.g. testing with telnet)

Step 4: Restart the Services

Restart both services to apply the new configuration:

# service postfix restart
Shutting down postfix:				[  OK  ]
Starting postfix:					[  OK  ]

# service cyrus-imapd restart
Shutting down cyrus-imapd:			[  OK  ]
Exporting cyrus-imapd databases:	[  OK  ]
Importing cyrus-imapd databases:	[  OK  ]
Starting cyrus-imapd:				[  OK  ]

Step 5: Create the Admin User and a First Mailbox

We’re close to done. First, check the permissions on /etc/sasldb2. (Be aware that passwords are stored there without any encryption.)

# ls -la /etc/sasldb2
-rw-r----- 1 cyrus mail 12288 Oct 15 12:34 sasldb2

Before running cyradm (the Cyrus IMAP admin tool) for the first time, set a password for the cyrus admin user defined in /etc/imapd.conf:

# saslpasswd2 -c "cyrus"
Password:
...

Now run cyradm, create your first mailbox, then set its password:

# cyradm -u cyrus localhost
IMAP Password:
localhost> cm user/john.doe@brainware.ro
localhost> quit
# saslpasswd2 -c "john.doe@brainware.ro"
Password:
...

Test a plaintext POP3 login using telnet:

# telnet localhost 110
Trying ::1...
Connected to localhost.
Escape character is '^]'.
+OK ...
user john.doe@brainware.ro
+OK Name is a valid mailbox
pass ...
+OK Mailbox locked and ready
list
+OK scan listing follows
...
quit
+OK
Connection closed by foreign host.

Step 7 (Optional): Relaying with SMTP Authentication

If you also want to relay email using SMTP authentication, first enable it in Postfix by adding to /etc/postfix/main.cf:

smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, permit
smtpd_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous

Then set up the authentication mechanism in /etc/sasl2/smtpd.conf (remember the alternate location, /usr/lib/sasl2/smtpd.conf, mentioned above):

pwcheck_method: auxprop 		# use /etc/sasldb2
auxprop_plugin: sasldb
mech_list: plain login cram-md5 digest-md5 ntlm gssapi		# enable all mechanisms

Wrapping Up

After one more Postfix restart, you should have a simple, working email service built on Postfix and Cyrus IMAP. Thanks for reading!

Gmail-style Aliasing with Postfix

2015-10-20T10:43:57+00:00

One of Gmail’s lesser-known features is plus addressing: e-mail sent to an alias of your main address, such as john.doe+22@gmail.com, is delivered to the main mailbox, john.doe@gmail.com, where it can be labeled and placed in different folders. (Note: this address is made up, and hopefully it doesn’t belong to a real person.)

How could you reproduce this on a mail server you run yourself? I won’t go into every detail, just scratch the surface. With Postfix (the default MTA on Red Hat, Fedora and CentOS), the place to start is the virtual_alias_maps option in the configuration file, main.cf.

Maps are a core concept throughout Postfix’s configuration. A map translates an input (for example, an e-mail address) into a desired output (for example, a mailbox location on disk or the real destination address). Postfix treats “virtual” as anything not tied to a real Unix account. To get Gmail-style aliasing, you therefore need a virtual map that converts addresses containing a + into the main mailbox address.

Since the aliased addresses can’t be predicted in advance, the only way to match them to the real destination is with some kind of regular expression matching. Simple Postfix map types, such as hash tables, only do exact matching, so you could alias john.doe+1 to john.doe, but that wouldn’t cover john.doe+2 or john.doe+99, unless you added a separate line for each possible alias.

The better news is that Postfix can match regular expressions through the corresponding built-in module, so the configuration could look like this:

# cat /etc/postfix/main.cf | grep virtual_alias_maps

virtual_alias_maps = regexp:/etc/postfix/virtual_alias

The virtual_alias file could then contain:

# cat /etc/postfix/virtual_alias

/^sample.name+(.*)@brainware.ro$/	sample.name@brainware.ro

This works, but it has some drawbacks:

Manual maintenance. You have to edit the virtual_alias file by hand to add new aliases. That isn’t practical for large installations where accounts are created and removed through web interfaces.
Performance. Postfix itself performs the regex matching, which can in theory reduce mail throughput. On a busy server with thousands of such aliases this might become noticeable, though it’s probably not a serious problem on modern hardware.

The fix for both issues is to hand the address matching over to a database such as MySQL or PostgreSQL and do the regex matching in a stored procedure. I may return to this in a later post, but for now, thanks for reading!

Adding Swap Space to a Node

2015-10-16T09:24:57+00:00

You might wonder why swap is even needed, because this isn’t enabled by default on any node in AWS or on any other cloud provider. The most likely reasons are the added complexity and the risk of wasting storage. And with instances now available with 200+ GB of RAM, why would anyone still need swap?

Swap space can be a sensible option for small nodes with 1-2 GB of memory, such as t2.micro, but because of the performance penalty, this is not suitable for production workloads. For experiments, dev testing, or QA, though, small instances are a cost-effective choice for most use cases (t2.micro is even free within certain limits during the first year).

Note: If you add swap to an EBS-backed node in AWS, keep the EBS storage limitations in mind.

Creating the swap file

A few commands are enough to add swap (1 GB in this example):

# dd if=/dev/zero of=/swapfile bs=1024 count=1048576
# mkswap /swapfile
# swapon /swapfile
# echo -e "/swapfile\t\tswap\t\t\tswap\tdefaults\t0 0" >> /etc/fstab

These commands create a 1 GB file in the root directory, format it as swap, tell the kernel to use it, and register it in /etc/fstab so it is enabled on every boot.

Tuning swappiness

Creating the swap space may not be the whole job. You may also want to adjust the “swappiness” behavior, which controls when memory pages get moved to swap to free up RAM for running applications. This is governed by the kernel parameter /proc/sys/vm/swappiness.

This parameter holds a number that represents the minimum amount of free memory the kernel tries to preserve before it starts swapping out pages. For example, a value of 10 means no pages are swapped until free memory falls below 10%. From the console you can set it with sysctl:

# sysctl vm.swappiness=10

To make this setting survive reboots, add it to the system configuration:

# echo "vm.swappiness=10" >> /etc/sysctl.conf

Hope you found this useful. Have a nice day!

Worst Mistake when Creating a RPM...

2015-10-16T09:24:57+00:00

…is overlooking the order in which RPM scriptlets are executed. Ideally, you want the package to behave like this:

First install: register the service with chkconfig and start it.
Complete removal: stop the service and unregister it from chkconfig.
Upgrade: stop the running service, replace the binary, and start the new version, leaving the chkconfig registration untouched.

The tempting shortcut is to skip the upgrade case entirely and assume that combining the removal and installation steps will cover it. In the .spec file, that might look like this:

%post
chkconfig --add my_wow_service
service my_wow_service start

%preun
service my_wow_service stop
chkconfig --del my_wow_service

This works fine as long as the package is only ever installed or removed. During an upgrade, however, this fails badly, because the scriptlets run in a different order than this example assumes:

%pre of the new package
%post of the new package
%preun of the old package
%postun of the old package

As a result, an upgrade with these scriptlets leaves the service stopped and removed from chkconfig, which is a real problem.

The fix is to check the first argument ($1) that RPM passes to each scriptlet. RPM supplies a value that tells the script whether it is running during an install, a removal, or an upgrade. Using that value, we can write better scriptlets:

%post
if [ $1 -gt 1 ] ; then
  service my_wow_service restart
else
  chkconfig --add my_wow_service
  service my_wow_service start
fi

%preun
if [ $1 -eq 0 ] ; then
  service my_wow_service stop
  chkconfig --del my_wow_service
fi

That solves the problem above.

Hope you found this useful!

mail() Function in PHP and Bounced E-Mails

2015-10-08T19:15:57+00:00

Everyone’s favorite way of sending spam is the mail() function in PHP. The question of the day is what happens to the bounced e-mails that may result from this operation.

Where do bounced e-mails go?

No, they’re not necessarily sent back to the “From” address of the original message. Mail servers return undeliverable messages to the envelope sender, which may or may not match the “From” field. There used to be a special header, Errors-To, but it was deprecated and as of today that is often ignored.

The envelope sender is set by the SMTP client when it connects to the server, so you might expect PHP’s mail() to handle it intelligently. First, let’s see how the function really works, starting with php.ini:

[mail function]
; For Win32 only.
SMTP = localhost
smtp_port = 25

; For Win32 only.
sendmail_from = me@example.com

; For Unix only. You may supply arguments as well (default: "sendmail -t -i").
sendmail_path = /usr/sbin/sendmail -t -i

On Windows, mail() appears to handle everything by itself. On Linux, the work is delegated to an external program: sendmail. The sendmail man page (the Postfix version) explains the two flags that are used here:

-i When reading a message from standard input, don’t treat a line containing only a . character as the end of input.

-t Extract recipients from the message headers. These are added to any recipients given on the command line.

So the mail() function actually prepares the message and pipes it to sendmail. How about the envelope sender, this elusive piece of data we were actually looking for? The man page above has the answer as well:

-f sender Set the envelope sender address. This is the address where delivery problems are sent. With Postfix versions before 2.1, the Errors-To: message header overrides the error return address.

Since no -f parameter is being passed here, bounced e-mails end up being sent to one of the following:

For older mail systems, the address in the Errors-To: header (if you supplied one in the 4th parameter of mail()).
The address in the Return-Path: header (if you supplied one in the 4th parameter of mail()).
The address automatically derived from the local user who ran the sendmail command (e.g. apache, nobody or a local user name). Postfix records all this in the automatically added X-Postfix-Sender: header.

Avoiding the third scenario

To avoid the third case, we need to to work our way around this. First, note that php.ini lets you replace the mail-sending program with your own script, which can do some additional processing before calling into sendmail:

sendmail_path = /usr/local/sbin/sendmail_wrapper

The script needs the right permissions to run:

rwxr-xr-x 1 root root 814 Oct 4 15:45 sendmail_wrapper

You could write the wrapper in Python or Ruby, but PHP works nicely too, as long as the script itself doesn’t call mail():

#!/usr/bin/php -q
<?php
set_time_limit(0);

$mtext = "";
$from = "";
$has_return_path = false;
// Feel free to replace the regex with a better e-mail match
$regex = "/[a-z0-9]+([_\\.-][a-z0-9]+)*@([a-z0-9]+([\.-][a-z0-9]+)*)+\\.[a-z]{2,}/i";
$stdin = fopen('php://stdin', 'r');
while (!feof($stdin))
{
  $cline = fgets($stdin);

  if (!((strpos($cline, "Return-Path:") === false))) {
    if (preg_match_all($regex, $cline, $matches, PREG_PATTERN_ORDER)) {
      $has_return_path = true;
    }
  }

  if (!((strpos($cline, "From:") === false))) {
    if (preg_match_all($regex, $cline, $matches, PREG_PATTERN_ORDER)) {
      $from = $matches[0][0];
    }
  }

  $mtext .= $cline;
}
fclose($stdin);
if (!empty($from) && !$has_return_path) {
    $from_opt = ' -f' . $from;
}
$fd = popen('/usr/sbin/sendmail -t -i' . $from_opt, 'w');
fwrite($fd, $mtext);
pclose($fd);

This is all that it is. Hope you found it useful!