Note: While I found this issue on Apache httpd, it may apply to any http server out there.
HTTP KeepAlive
The “KeepAlive” concept is simple: the browser opens a connection to the server and sends out multiple requests (e.g. for the main page, for stylesheets, javascript includes and images) through a single connection. This effectively reduces the page load time, providing – or at least that’s what the theory says – a better customer experience. All good from this perspective.
Server Configuration
The Apache httpd server controls this feature through 3 configuration directives, all of them in the core module:
KeepAlive: on/off, default on;
KeepAliveTimeout: seconds, default 5;
MaxKeepAliveRequests: positive number, default 100.
There is no setting in Apache httpd to somehow allow KeepAlive and non-KeepAlive requests at the same time (e.g. to allow up to 100 KeepAlive requests, what comes above to be treated differently). One must choose the server behavior from the very beginning.
The Traffic
Now it’s math time. Starting from the MaxClients value (default 256), what is the request rate (new clients / second) that can be served without compromising the user experience? MaxClients you may say, but let’s not draw conclusions too fast. There are some issues to be considered:
The time between opening the connection and the KeepAliveTimeout expiration. On a default configuration, at the very least 5 seconds, but on a more typical side, maybe 7-10 seconds;
Traffic fluctuation (spikes);
Internal Apache httpd time (spawning new processes to handle new connections, etc).
It’s getting complicated. But assuming a typical 7 seconds time for every process that serves a client and an uniform behavior (all or almost all clients keep a server process busy or waiting for data for 7 seconds), on a default setting of 256 MaxClients, the uniform traffic rate that can be served is 256/7 = 36 (new) requests / second. Any spike larger than that will cause page loading delays and a poor user experience.
Better Planning
If disabling KeepAlive is not an option, all the planning should start from the worst expected spike that is to be handled (e.g. 2x or 3x the average during the busiest period). For a 50 req/s busy period average, the server should be able to allow 100 or even 150 req/s without compromising the user experience. If the configuration is already maxed out from the hardware point of view, then other solutions should be looked into (multiple servers, load balancers, I’m not dwelling into that).
Assuming a maximum rate of 150 req/s, with a KeepAliveTimeout at the 5 seconds default, one may need to adjust MaxClients (and ServerLimit if prefork mpm is used) to somewhere in the area of 1,000.
Conclusion
Don’t leave the defaults on; always start the parameter calculation from the desired outcome first. On suboptimal hardware (memory wise) disabling KeepAlive is the way to go to squeeze a bit more performance. Oh, and don’t assume the hardware is capable of keeping up with your calculations; turn any failure or mis-planning in a learning subject.
That’s it for today, have fun!
Who likes interviews? Me neither. Well, depends…
If you get any of the following questions during an interview then either the interviewer did read this one or he’s getting info from the same sources as I am. Either way, let’s get one step forward.
Question 1:
Suppose you log into a node and find the following situation:
# ls -la /bin/chmod rw-r--r-- 1 root root 56032 ian 14 2015 /bin/chmod
Is it fixable?
… yes. Most of the time. Let’s remember how the executables are being started on Linux: with a system loader, ld-linux-something. Let’s check:
# ldd /bin/chmod linux-vdso.so.1 => (0x00007ffdf27fc000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb11a650000) /lib64/ld-linux-x86-64.so.2 (0x00007fb11aa15000)
OK, got it:
# /lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod
Fun, isn’t it? The follow up question obviously is “what if the loader’s rights are unproperly set” or something similar. The answer to that is that not all the filesystem issues are fixable with easy commands. One may even have to mount the file system on a different installation (e.g. from a Live CD or attach the virtual storage to another, “good” node) and fix things up from there.
Continue Reading →
Why would you need to add swap, you may ask (or not); nevertheless, the answer is that swap space is not added by default to any node in AWS or on any Cloud Provider. The reasons for this are many, the main one being, most likely, the complexity implied, along with the possibility of wasting storage resources. But anyway, one may get these days instances with 200+ gigabytes of internal memory, why would anybody still need swap?
Enabling swap can indeed be an option for nodes with 1-2 gigabytes of memory such as t2.micro, but not for production workloads due to the performance issues. If you want to put up a node for experiments, dev testing or qa, such instances are cost effective approaches for most use cases (t2.micro is also free to use, within limits, during the first year).
Note: When adding swap space to an EBS-backed node in AWS, please keep the EBS storage limitations in mind when going on with this approach.
Adding swap space (e.g. 1Gb) is possible with a few commands:
# dd if=/dev/zero of=/swapfile bs=1024 count=1048576 # mkswap /swapfile # swapon /swapfile # echo -e "/swapfile\t\tswap\t\t\tswap\tdefaults\t0 0" >> /etc/fstab
This creates a special 1Gb file in the root directory, tells the kernel to use it as swap space and adds this setup as default in /etc/fstab.
Just defining the swap space may not be enough for the desired setup. One may also want to alter the “swappiness” behavior – when memory pages are sent to swap to free up space for running applications. For this a certain kernel parameter, /proc/sys/vm/swappiness, must be altered to suit one’s needs.
This parameter contains a numeric value, corresponding to the minimum allowed quantity of free memory available before the kernel starts to swap pages. A value of “10” means that no memory pages will be sent to swap before the free memory drops under 10%. Using the console, the parameter can be set with sysctl:
# sysctl vm.swappiness=10
One may also add it to the system configuration so that it survives reboots:
# echo "vm.swappiness=10" >> /etc/sysctl.conf
Hope you enjoyed this one. Have a nice day!