The "Debugging" Interview – a Few Pointers

The long interview day is almost over. Googamazbook has drawn out the best and the worst of you (well, neither, but I’m trying to sneak in some literature). The last interviewer walks in, smirks, and says:

Time for the easy interview, heh?

You have every reason to worry that you’re one step from failure. Why didn’t you spend the day by the pool in the basement of that many-starred hotel they booked for you? Without further ado, the questions start coming.

How do you figure out whether a process is CPU bound or I/O bound?

This is trickier than it looks. Two variables give four possibilities:

1. The process is CPU bound.
2. The process is I/O bound.
3. The process is both CPU and I/O bound.
4. The process is neither.

In the real world, most processes fall into the last category. They spend nearly their whole life in the S state (sleeping).

Processes also rarely run one operation, or one fixed set of operations, and then exit. Over its lifetime, a process can be CPU bound at some moments and I/O bound at others, even if those moments are a tiny fraction of its run time. Watching top or ps for long stretches in state R (running) or D (uninterruptible sleep / I/O wait) may therefore tell you nothing useful.

The conclusion: if you treat the process as a black box, you cannot answer the question. With access to the source code, you can find the relevant code blocks or add tracing. Otherwise, the best you can say is one of the following:

• The process uses CPU in a way that would probably benefit from more computing power, at least for some operations.
• The process reads or writes large amounts of data and would probably benefit from faster storage.

The real trick is verifying either statement for an arbitrary black-box process. It isn’t hard, but you have to look under the operating system’s hood:

• High CPU usage shows up in /proc/pid/sched. Check nr_involuntary_switches and nr_voluntary_switches. If involuntary switches outnumber voluntary ones, the kernel is forcibly taking the process off the CPU, which suggests a CPU-intensive process.
• High I/O also shows up in /proc/pid/sched, in the se.statistics.iowait_sum attribute. This is an absolute figure, so correlate it with the process’s running time and compare it with other processes. Monitoring with ps or top is still the best approach for I/O.

Done? Yes. On to the next one.

How do you debug an unresponsive service?

This is too general. First ask which service it is, and whether the node is still reachable some other way (for example, over ssh). The scenario usually involves a single unusable service on a node that is up and accessible. Otherwise, “reboot the node” would be the answer, and the bar can’t be that low for an interview like this.

A typical exchange goes like this:

1. What service is it? HTTP.
2. Is it accessed by IP or by name? By name, via DNS.
3. Does DNS resolve the name to an IP? Yes.
4. Is that the IP of the node with the unresponsive service? Yes.
5. On the node itself, is the service listening on the expected port? Yes.
6. Is any firewall filtering that port? No.

All the trivial explanations are now ruled out, so it’s time for the big guns. The scenario continues:

7. Is the service up and running? Yes.
8. How many processes does the service have? About 50.
9. (Getting smart) What is the service’s process limit? Around that number.
10. (Eureka!) The service is overloaded, so raise the limits and do a graceful restart. After doubling the limit, the service is still unresponsive.

You can smell failure, as many candidates have before you. That’s a pity, because you could simply inspect individual processes and see where they’re blocked.

11. Run strace -p on individual processes to see what they’re doing. They’re blocked in a read() call on some file descriptor.
12. (Past the mountain’s peak) Check ls -la /proc/pid/fd/xx. It’s a file under /www/service.
13. How is that path mounted? It’s an NFS directory.
14. Is the NFS server reachable, e.g. via ping? Yes.
15. Go to that server and run top, ps, and so on. top reports 99% wa.
16. (The cause is identified, now for the fix) Run ps and look for processes in state D. /usr/local/bin/maintenance.sh is in state D.
17. What does it do? It should prune some logs, but because of a bug it keeps repeating the job.
18. Try kill -9, remove it from cron, and chmod a-x the script. That fixed it.

Crazy, wasn’t it? I made this scenario up myself, but it is a plausible interview exercise. A few pointers to remember:

• The solution is never trivial (no client error, no external service error).
• The problem isn’t immediately visible with the usual tools (ps, top, df).
• The cause is almost never on the node where the symptoms appear, and the relationship between the nodes is rarely easy to work out.
• System call tracing is almost always needed to uncover that relationship.

On to the next question.

How would you deal with a “disk full” situation?

First, two kinds of resources can run out and cause a “disk full” condition:

• File system blocks (the actual storage of contents).
• Inodes (metadata).

So diagnosing the problem takes two commands:

# df
# df -i

The approach depends on the cause. Running out of space can usually be traced to one or a few very large files, while running out of inodes is caused by huge numbers of small files.

Finding the largest files is a job for find:

# find / -type f -size +10M -printf '%k\t%p\n'

This prints the on-disk size (in KB) and path of every file larger than 10 MB. There are usually a few outliers of several gigabytes or more. Deleting them is easy but doesn’t always fix the problem, because a program may still have them open and be writing into them.

To check, use lsof:

# lsof | grep file_name

If you can’t stop or restart the program after deleting the file, truncate may help:

# truncate -s 0 file_name

You may also meet the opposite case: a very large file was deleted but is still held open by a program. The find command above won’t show it, but lsof will, and truncate then finishes the job:

# lsof | grep "deleted"
...
# truncate -s 0 /proc/pid/fd/xx

Inode exhaustion is harder to fix. There are no outliers, just hundreds of thousands of files that may need deleting. Locating them takes some detective work: figure out what software runs on the node and whether anything changed in the last few days. find helps again:

# find / -mtime -1 -print

This lists files created or modified in the last day. For a more complex deletion example, see the second question in this interview series.

That’s all for today. Thanks for reading!