The "Systems" Interview - a Few Pointers

Note: This article is about Systems Programming on Linux platforms.

This kind of interview checks what you know about what happens beneath the command-line surface. Down there, processes are created and terminated, output is collected, and system calls are made. Things get complicated quickly for anyone who is unprepared.

$ ls

This classic interview question has been asked for more than 15 years. It may not come up much anymore (as of 2016), but the answer is still instructive. The question is not about the file listing that appears on screen, but about everything that happens to produce it.

1. The shell parses the command line and identifies the command and its arguments (none here).
2. The shell works out whether the command is:
   • a built-in (internal) command,
   • an alias, or
   • an external binary that must be located.
3. On most systems, ls is an alias for the real binary plus an option such as --color=auto.
4. As a safety step, the current directory is validated with stat(".") before anything is executed.
5. The shell searches for the binary by walking through each directory in the PATH environment variable. It appends the command name to each one and calls stat() on the result. ls is usually in /bin, so the search stops once /bin/ls is confirmed as executable.
6. The shell creates a new process with fork(), then calls an exec()-family function to run the binary with the arguments determined earlier.
7. Boilerplate startup code runs: shared libraries are loaded, memory is allocated, signal masks are set, signal handlers are installed, and so on.
8. Now the interesting part begins:
   • ls reads /proc/filesystems and loads the current locale.
   • It opens the current directory with opendir().
   • It enumerates the entries with readdir() and closes the directory with closedir() (more memory may be allocated along the way).
   • It writes the directory contents to standard output with write(1, ...).
9. Finally, ls releases its memory and exits.

It’s more involved than it first looks, isn’t it?

$ ls | tee list.txt

The basic idea is simple: the standard output of ls is piped into the standard input of tee.

NB: stderr is not redirected in this case, so error messages will most likely go straight to the console.

The shell still parses and executes everything. The earlier details still apply: built-in vs. alias vs. external command, and a stat() in each PATH directory until the binary is found. The difference is that the shell now identifies and launches two external commands (ls and tee), each in its own process.

The “magic” happens before fork() and exec(). Roughly:

• The shell asks the kernel for an unnamed pipe with the pipe() system call, which returns a read end and a write end.
• The write end of the pipe is duplicated with dup2() onto the stdout of the process that will run ls.
• The read end of the same pipe is duplicated with dup2() onto the stdin of the process that will run tee.
• The shell collects the exit codes of both child processes. Even if the first one produces no output, the second one still runs, with empty input.

An interviewer who knows this area well will probe how fork() and exec() behave at the kernel level. A strong candidate should be able to explain the following.

• fork() duplicates the current process. Logically, nearly everything is copied: memory, file descriptors, and kernel resources such as references to shared memory segments or semaphores. Only the PID differs. If the process has several threads, only the thread that calls fork() is duplicated, which can cause many hard-to-find bugs.
• Copy-on-write: the memory is not actually copied up front. Pages are marked read-only, and if either parent or child tries to modify one, a page fault occurs. The kernel then quietly duplicates the page and lets the write proceed.
• exec() replaces the process image with that of the new binary, and the kernel frees the previously allocated memory. File descriptors without the close-on-exec flag (O_CLOEXEC) stay open and are available to the new program. By default, stdin, stdout, and stderr do not have this flag set.

As you can see, this kind of interview can get tough quickly. Fortunately, only a few companies run such “systems interviews,” and only for certain roles. If you have one coming up and spent a few minutes reading this, you’re likely better prepared than before. At least, that’s the hope. Thanks for reading!