Crazy DevOps Interview Questions (1)

Note: There are three more episodes in this interview series: part 2, part 3 and part 4.

Nobody enjoys interviews. Well, maybe it depends on the day or on what side of the table you happen to be in.

If an interviewer asks you any of the questions below, then they either have read this post or had drawn on the same sources I did. Either way, you’re now a step ahead.

You log into a node and find this:

# ls -la /bin/chmod
rw-r--r-- 1 root root 56032 ian 14  2015 /bin/chmod

The chmod binary has lost its execute bit. Can this be fixed?

Yes, most of the time. Remember how Linux starts executables: through a system loader, something like ld-linux. Let’s find out which one chmod uses:

# ldd /bin/chmod
	linux-vdso.so.1 =>  (0x00007ffdf27fc000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb11a650000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb11aa15000)

Now call the loader directly and let it run chmod for you:

# /lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod

Neat, right? The obvious follow-up is: “What if the loader’s own permissions are wrong?” Not every filesystem problem has a simple command-line fix. You may have to mount the filesystem from a different installation (for example from a Live CD, or by attaching the virtual disk to another healthy node) and repair it from there.

A running process hangs while reading from a socket. Variations: it may be deadlocked on a read or on a write, or it may keep receiving EAGAIN and sit in a livelock. How do you close the bad socket / file descriptor without killing the process?

First, where can we see the files and sockets a process has open, from the outside? In /proc:

# ps -e | grep myprocess
  979 pts/0    00:00:00 /usr/bin/myprocess

# ls -la /proc/979/fd
total 0
dr-x------ 2 root   root  0 oct 22 21:37 .
dr-xr-xr-x 9 nobody dip   0 oct 22 21:37 ..
lrwx------ 1 root   root 64 oct 22 21:37 0 -> /dev/null
lrwx------ 1 root   root 64 oct 22 21:37 1 -> /dev/null
lrwx------ 1 root   root 64 oct 22 21:37 2 -> /dev/null
lrwx------ 1 root   root 64 oct 22 21:37 3 -> socket:[9814]
lrwx------ 1 root   root 64 oct 22 21:37 4 -> socket:[9816]
lrwx------ 1 root   root 64 oct 22 21:37 5 -> socket:[9817]
lrwx------ 1 root   root 64 oct 22 21:37 6 -> socket:[9818]
lr-x------ 1 root   root 64 oct 22 21:37 7 -> pipe:[9825]
l-wx------ 1 root   root 64 oct 22 21:37 8 -> pipe:[9825]
lrwx------ 1 root   root 64 oct 22 21:37 9 -> socket:[9827]

We now have the pid and the fd (file descriptor). You can’t remove or manipulate an open socket from outside the process, so we have to get inside, by using gdb. Assuming pid == 979 and fd == 9:

# gdb
  ... (lots of output) ...
(gdb) attach 979
Attaching to process 979
  ... (lots of output) ...
(gdb) p close(9)
$1 = 0
(gdb) detach
Detaching from program: /usr/bin/myprocess, process 979
(gdb) quit

Tricky, wasn’t it? An interviewer who asks this is probably looking for a real expert.

How would you deal with this situation?

# ls
bash: fork: retry: Resource temporarily unavailable

In short, the system has run out of pids, and this shell is the only access you have left.

Note: This is the classic fork bomb question that companies such as Google sometimes ask. They aren’t after one definitive answer (none exists); they want to see how you approach the problem.

Realistically, the system may be so overloaded (many processes compete for CPU time) that a hard reboot is the only real fix. In an interview, though, you’re usually expected to give a “clever” answer with shell scripting tricks.

Before writing that answer, three sub-problems need solving beforehand:

• How do you find the process name? In a fork bomb, the same name often occupies all the available pids.
• Once you have the name, how do you identify every matching pid?
• Once you have a (very long) pid list, how do you run the kill command?

Interviewers rarely ask about the first; the name is usually “known,” or can be found by trial and error in /proc/. Even that isn’t trivial, since ls and cat are external commands and can’t be launched. Shell builtins could do it, though:

# read t < /proc/1091/cmdline ; echo $t
/usr/sbin/bash

You can also iterate over the pids with builtins only. (Note: while spawns a process, so it can’t be used here.)

# for p in /proc/[0-9]* ; do echo ${p#/proc/} ; done
 ... long pid list, one per line ...

Now to the process killing part. A plain kill -9 is easy but doesn’t solve anything: each freed pid just lets another process spawn. So you need two passes:

1. Freeze all the offending processes (kill -SIGSTOP).
2. Then remove them (kill -SIGKILL).

The code looks like this. (Note: the double-bracket [[ ... ]] form is a builtin, while the single-bracket [ starts a new process.)

# for p in /proc/[0-9]*/cmdline ; do read cmd < $p ; if [[ $cmd = 'bomb_process_name' ]]; then t=${p%/cmdline} ; kill -SIGSTOP ${t#/proc/} ; fi ; done
# for p in /proc/[0-9]*/cmdline ; do read cmd < $p ; if [[ $cmd = 'bomb_process_name' ]]; then t=${p%/cmdline} ; kill -SIGKILL ${t#/proc/} ; fi ; done

Notes:

• The string comparison relies on whitespace around the variable, the equals sign and the string. a = "xxx" is a comparison, while a="xxx" is an assignment in Bash.
• = and == behave identically in this comparison.
• The # and % operators remove the shortest match. Since we match a fixed string rather than a pattern, ## and %% (longest match) would give the same result.
• cmdline in /proc/<pid> is not a reliable source for the process name. It works against simple bash fork bombs or sloppy code, but a determined attacker can randomize it, for example with snprintf(argv[0], size, "randomstuff").
• Other sources (/proc/<pid>/comm, /proc/<pid>/status) can also be overridden, for example with prctl(PR_SET_NAME, (char*) "randomstuff"). Yes, yes, I know about const_cast.

Follow-up questions:

• What if the bomb is a bash fork bomb and you’re in a bash shell? Wouldn’t the first loop kill your own shell too?
• What if kill isn’t a builtin and only the external version exists?

Not everything can be solved cleanly. /proc/self is a symlink to your own pid, and readlink can read it, but that’s an external command. Luckily, the builtin echo $$ gives you your own pid, which you can exclude in the if condition above.

With kill only available as an external command, I’m not sure the problem can always be solved.

If a normal user triggered the bomb, you can immediately append a limit to /etc/security/limits.conf:

echo "forkuser hard nproc 20" >> /etc/security/limits.conf

This stops the active part of the bomb: even if you run exec ... in the terminal, the fork bomb can’t claim the pid. Running the second kill command will then probably free enough pids for normal operation to resume.

If the user is root, or has certain capabilities such as CAP_SYS_ADMIN, this option doesn’t work. Collecting all the pids in a shell variable and running exec /usr/bin/kill then locks you out when it’s time for the second command of the pair. Cleaning up the now stale pids becomes the hard part.

Note: There’s no guarantee that the pid released by the vanished shell stays free for another process under your control. Other daemons on the machine may compete for the same resource.

Still, you can try:

• cron: a best-effort approach, since there’s no guarantee cron can run.
• inittab, if the system has one (newer distributions don’t). Put the kill commands into a file executed on switching to a runlevel (e.g. 4), then force the change with telinit. Again, no guarantees.

On machines you can physically access, you usually have several login screens available via key combinations like ALT + digit, which lets you get back in. On machines reached through virtual terminals (ssh, xterm, GNOME Terminal), that isn’t possible, so the question has no real answer there.

That’s it for today’s “crazy” interview questions. Hope you enjoyed them!