Crazy DevOps Interview Questions (3)

Note: This is the third article of the series. The others are: part 1, part 2 and part 4.

The interviewer hands you this ls -la listing:

ls -la

total 108
dr-xr-x---. 7 root root 4096 Sep 5 07:16 .
dr-xr-xr-x. 22 root root 4096 Sep 1 17:43 ..
-rw-------. 1 root root 15432 Sep 5 06:36 .bash_history
-rw-r--r--. 1 root root 18 May 20 2009 .bash_logout
-rw-r--r--. 1 root root 176 May 20 2009 .bash_profile
-rw-r--r--. 1 root root 176 Sep 23 2004 .bashrc
-rw-r--r-- 1 root root 0 Sep 5 07:16 -f
...

They tell you the -f file has to go. Can you delete it?

• Quoting (rm "-f"): The shell strips the quotes, so rm receives -f as an option. It reports no error, because -f tells rm to ignore missing files, but nothing gets deleted.
• Escaping the minus (rm \-f): The shell processes the backslash, and rm again receives a plain -f.
• Double-escaping (rm \\-f): rm receives the literal string \-f, which fails because a file named \-f doesn’t exist (only -f does).

Many candidates give up here, which does not bode well for their employment chances.

Both are found in man rm:

1. Use -- to mark the end of options:

rm -- -f

2. Prefix the filename with the current directory:

rm ./-f

Which characters are forbidden in filenames?

The answer is that on Linux, only \0 (null, the string terminator) and / are disallowed. Windows has a longer list.

At the whiteboard, write a cron definition for a file in /etc/cron.d that appends the date/time to a file every Wednesday at 5 PM, using the format %Y-%m-%d %H:%M:%S.

An experienced candidate might quickly write:

0 17 * * 3 root date +’%Y-%m-%d %H:%M:%S’ >> /tmp/logfile

Looks right, but it isn’t. Things you might check:

• Cron runs commands with the shell configured in /etc/crontab, not your favorite one (irrelevant for such a simple command).
• Local PATH changes aren’t visible, though /bin is usually included.
• Local environment variables aren’t visible (none are used here).

Now, none of these is the culprit. The real problem is the % character. Cron treats it as a newline. The command gets split at each % and passed to /bin/sh like this:

date +’
Y-
m-
d
H:
M:
S’ >> /tmp/logfile

This obviously produces an error. The fix is to escape every percent sign:

0 17 * * 3 root date +’\%Y-\%m-\%d \%H:\%M:\%S’ >> /tmp/logfile

A mission-critical process is writing to a file whose contents must be preserved, but a junior admin just ran rm -f on it. The process is still running, but in a few hours log rotation will send a HUP signal to said process.

Now, it’s less scary than it sounds:

1. Find the process’s PID and inspect /proc/<pid>/fd to locate the deleted file’s descriptor:

ls -la /proc/966/fd/10
lrwx------ 1 root root 64 Sep 5 08:56 /proc/966/fd/10 -> /var/lib/critical/process.out (deleted)

2. Copy out the contents:

dd if=/proc/966/fd/10 of=/tmp/recovered.out

But what about data written after dd finishes? Possible approaches:

• Send SIGSTOP to freeze the process, copy the data with dd, then kill or restart the service.
• Copy the data, then immediately send SIGHUP.

Neither guarantees full preservation if the application buffers its output. (A mission-critical application should flush after every write, which is a good point to raise in the interview.)

A cleaner solution uses tail:

tail -f -n +0 /proc/966/fd/10 > /tmp/recovered.out

This captures the whole file from the beginning and keeps following new writes. You can leave it running until the log rotation event if a faster restart isn’t desirable.

A node has hit 100% storage. An administrator investigated but all visible file sizes look normal. The node can’t be restarted because of mission-critical software.

Diagnosis: If no visible file is large enough, some process is probably holding a reference to a very large deleted file (or several deleted files that add up). lsof helps find it:

lsof | grep deleted | awk ‘{print $2,$4,$8,$9}’

2006 34w 24967 /tmp/vteAIZ4MY
2006 35u 28708 /tmp/vteXJZ4MY
2006 36w 28985 /tmp/vteYU9XMY
....

This filters lsof output for deleted files and prints four fields:

1. The PID of the process holding the deleted file open;
2. The file descriptor (with access mode);
3. The file size, the field to look for;
4. The file name.

Once identified, there are two options:

• Reload or restart the process, e.g. kill -HUP <pid>
• Truncate the file, e.g. truncate -s 0 /proc/<pid>/fd/<fd>

That’s it for today. Thanks for reading!