Monthly Archives: October 2015
Gmail-style aliasing with postfix

One of the less known features of Gmail is the ability to receive e-mails sent to particular aliases of the main e-mail address, e.g. mails sent to john.doe+22@gmail.com will get to the main mailbox, john.doe@gmail.com. (NB: the address is made up, hopefully this is not someone’s real address).

How would you replicate such feature on a local node you manage? I won’t cover all the details, at most I will just scratch the surface a bit. When using Postfix (the default MTA installed with Redhat/Fedora/CentOS), one must first look to the place where such functionality can be put in, the virtual_alias_maps option in the configuration file (main.cf).

The mapping concept in Postfix is common throughout its options. A “map” must be able to provide a translation between an input (e.g. an e-mail address) and some desired output (e.g. the mailbox location on the disk, the real destination e-mail). The “virtual” part is understood by Postfix as anything that is not tied to a real Unix account. So, in order to be able to provide the Gmail-style aliasing, one must essentialy create a (virtual) conversion map that would relate an input (addresses with “+“) to the desired output (the main mailbox).

Looking back to the wildcard e-mail addresses that we may get e-mails sent to, the only solution to get them matched to the real destination is by using some sort of regular expression. Simple map types accepted by Postfix (e.g. hash tables) perform exact matching, e.g. we may be able to alias john.doe+1 to john.doe but not john.doe+2, neither +99 (well, we are able, but we must put in a line for every such alias). Postfix does support regular expression matching through a built-in module, so we may have in the configuration something like:

# cat /etc/postfix/main.cf | grep virtual_alias_maps
  
virtual_alias_maps = regexp:/etc/postfix/virtual_alias

And the “virtual_alias” file may contain something like:

# cat /etc/postfix/virtual_alias

/^sample.name+(.*)@brainware.ro$/	sample.name@brainware.ro

This solution just works, but there are some drawbacks to it:

  • One must manually edit the “virtual_alias_maps” file to add new aliases. This is not practical for large installations with e-mail accounts created and deleted from web interfaces.

  • The regular expression matching is done within Postfix, theoretically slowing down the mail system throughput. On a busy server with thousands of such aliases this may become a noticeable issue (maybe not much of a serious problem with modern hardware, though).

For both problems, the solution is to offload the address matching to a database installation (e.g. MySQL, PostgreSQL) and do the regex matching from a stored procedure. I may come back to this on a later date, though. But for now, thank you for your time!

A small introduction to Chef

Chef Logo

If you know what Chef is, you may skip this one or actually stay for a (hopefully) interesting read. Either way, Chef is a tool that one may use for automating the node configurations – e.g. installed software, their configuration files, system configuration files, NFS mounts and so on.

When interacting with chef, one may realize that there are 3 main components involved:

  • The Chef Server, which is actually just a big data repository.
  • The Chef Client, which is installed and runs on endpoints and does all the “dirty work” such as changing files and installing packages. The client authenticates itself to the server by a public/private key mechanism.
  • The Knife: this is the tool used by the sysadmins to actually do work with Chef.

What is stored in the repository known as Chef Server?

  • Cookbooks;
  • Data Bags;
  • Environment and Node data.

A Cookbook is a small project written in Ruby. This project contains files for:

  • Recipes – these are individual files that contain rules to be aplied on clients (e.g. install a rpm package, deploy a config file from a template). The file contents is plain Ruby code using constructs (calls to libraries) provided by Chef.
  • Attributes – or, better phrased, default attributes (e.g. port numbers, sizes, paths etc).
  • Templates – templatized configuration files with attributes replaced with template variables that are to be initialized in the recipe from default or user provided values.

These small projects are versioned and usually are also maintained – for development purposes – in an external repository (most likely git based). There is a system of includes and dependencies; starting some recipe installation will maybe trigger the installation of a full environment.

The “data bag” is an interesting concept: these are json data pieces stored inside the Chef Server. They can be encrypted, making them suitable for sensitive data such as passwords or private keys.

What is a (Chef) Environment? One may see it as a form of grouping nodes (servers). This allows for having shared, common data for all the nodes within a certain environment and also for running commands to groups of nodes identified by the environment they belong to. The (Chef) Node also has a data record associated with it that may override the inherited settings from the environment. Such data record (also json encoded) contains overrides for the default attributes described before and, on individual nodes, the recipe list to be applied (also called the “run list”).


As I have mentioned before, the effective work is performed with knife. The entire documentation is available on the Chef website:

From a DevOps perspective, the most frequent operations are:

  • Modifying environment and node data;
  • Uploading and downloading cookbooks without dependency checking; for dependencies there is another tool, berks;
  • Running the same shell command on a group of nodes (query based).

Is it a fun tool? It sure is. But you know the saying, “don’t drink and drive”; in this context, due to the impact of individual commands, one must be extra careful (and most likely sober).

That’s it for a small introduction on Chef concepts. Thank you for the read and have a nice day!

Adding swap space to a node

Why would you need to add swap, you may ask (or not); nevertheless, the answer is that swap space is not added by default to any node in AWS or on any Cloud Provider. The reasons for this are many, the main one being, most likely, the complexity implied, along with the possibility of wasting storage resources. But anyway, one may get these days instances with 200+ gigabytes of internal memory, why would anybody still need swap?

Enabling swap can indeed be an option for nodes with 1-2 gigabytes of memory such as t2.micro, but not for production workloads due to the performance issues. If you want to put up a node for experiments, dev testing or qa, such instances are cost effective approaches for most use cases (t2.micro is also free to use, within limits, during the first year).

Note: When adding swap space to an EBS-backed node in AWS, please keep the EBS storage limitations in mind when going on with this approach.

Adding swap space (e.g. 1Gb) is possible with a few commands:

# dd if=/dev/zero of=/swapfile bs=1024 count=1048576
# mkswap /swapfile
# swapon /swapfile
# echo -e "/swapfile\t\tswap\t\t\tswap\tdefaults\t0 0" >> /etc/fstab

This creates a special 1Gb file in the root directory, tells the kernel to use it as swap space and adds this setup as default in /etc/fstab.

Just defining the swap space may not be enough for the desired setup. One may also want to alter the “swappiness” behavior – when memory pages are sent to swap to free up space for running applications. For this a certain kernel parameter, /proc/sys/vm/swappiness, must be altered to suit one’s needs.

This parameter contains a numeric value, corresponding to the minimum allowed quantity of free memory available before the kernel starts to swap pages. A value of “10” means that no memory pages will be sent to swap before the free memory drops under 10%. Using the console, the parameter can be set with sysctl:

# sysctl vm.swappiness=10

One may also add it to the system configuration so that it survives reboots:

# echo "vm.swappiness=10" >> /etc/sysctl.conf

Hope you enjoyed this one. Have a nice day!


Previous Page · Next Page