Linux Performance Monitoring (an Introduction)

“Performance” is a broad term, so it helps to split it into categories first. Two questions do most of the work:

1. What do we want to monitor? The whole system, or one particular application?
2. What kind of monitoring do we need? Basic statistics gathered by the kernel, or in-depth analysis?

Combining the answers gives four categories, each with its own set of tools:

Notes:

• netstat does more than report interface and protocol statistics. It can also monitor individual connections.
• dtrace and SystemTap can also trace individual applications.

Where do the tools in the first column get their numbers? The short answer is “the kernel”, but the actual interface is the /proc file system. For every process, the kernel exposes a stat file containing all the collected data:

# cat /proc/15588/stat
15588 (httpd) S 30837 30837 30837 0 -1 4202816 1845 0 0 0 2 1 0 0 20 0 1 0 381733777 671944704 4446 18446744073709551615 140071612739584 140071613080548 140735088901328 140735088899312 140071585579283 0 0 16781312 201344747 18446744071580587657 0 0 17 0 0 0 0 0 0

This format suits software far better than people: without the documentation, nobody can tell which number is which. Fortunately, a human-readable version exists:

# cat /proc/15588/status
Name:   httpd
State:  S (sleeping)
Tgid:   15588
Pid:    15588
PPid:   30837
TracerPid:      0
Uid:    48      48      48      48
Gid:    48      48      48      48
Utrace: 0
FDSize: 64
Groups: 48
VmPeak:   656708 kB
VmSize:   655684 kB
VmLck:         0 kB
VmHWM:     17920 kB
VmRSS:     17752 kB
VmData:    87788 kB
VmStk:       100 kB
VmExe:       336 kB
VmLib:     62692 kB
VmPTE:       872 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/63581
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 000000018c0046eb
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   ff
Cpus_allowed_list:      0-7
Mems_allowed:   00000000,00000000,...,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        166
nonvoluntary_ctxt_switches:     0

The kernel also provides system-wide statistics:

# cat /proc/stat
cpu  20887396 0 10349385 3015436747 407563 3166 1467937 0 0
cpu0 10678475 0 3883808 362454876 232413 3166 1390223 0 0
cpu1 1661441 0 1011415 378955537 31275 0 11441 0 0
cpu2 746371 0 330726 380344321 23829 0 11456 0 0
cpu3 519138 0 214791 380738302 16767 0 10813 0 0
cpu4 5386158 0 3986695 371475949 54473 0 12596 0 0
cpu5 682554 0 324926 380359994 18308 0 11663 0 0
cpu6 705281 0 343150 380407382 15057 0 10218 0 0
cpu7 507974 0 253871 380700384 15438 0 9525 0 0
intr 5529843084 126 2 0 0 0 0 0 0 1 0 ... (long run of mostly zeros omitted)
ctxt 5379451608
btime 1451035714
processes 6474048
procs_running 1
procs_blocked 0
softirq 11537062915 0 4052674276 583728909 992215181 16867459 0 13667315 1762309239 2086018 4113514518

There is no human-readable version of the system-wide /proc/stat, which is why you need the tools listed above to interpret it.

Tracing is where things get more complicated.

Userspace tracing. You can monitor individual processes with signals such as SIGTRAP. This makes tracing possible with userspace-only tools, but the overhead is significant, so it is not advisable for performance-sensitive work. Still, strace and gdb remain the best tools for debugging and spotting erratic behaviour in individual programs.

Kernel-assisted tracing. More advanced tools such as dtrace rely on the kernel to report events for one process or for all of them. This is sometimes called “zero-cost” tracing, and it can be used in production environments.

dtrace and SystemTap both include scripting languages for handling the results when a “probe” fires, for example when a particular system call runs or execution reaches a certain line in the monitored program. This makes it possible to:

• Process data in advanced ways (e.g. count specific behaviours)
• Aggregate statistics
• Perform “open-heart surgery” on programs, intercepting data flows or replacing data that would otherwise trigger undesired behaviour

Static vs. dynamic tracing. Tracing falls into two categories:

• Static tracing looks at invariants such as system calls or well-known points within a feature. These are well documented and don’t change.
• Dynamic tracing is like setting breakpoints on individual lines of code. Such breakpoints work for specific versions and may not apply to later versions of the program.

Example: a dtrace one-liner that prints a timestamp for every system call made by a given application:

$ dtrace -n 'syscall::: /execname=="myapp"/ { printf("%d", timestamp); }'

More one-liners are available here: DTrace One Liners.

That wraps up my introduction to performance monitoring. I hope you enjoyed it!