Frequently Asked Question

What lives in /proc/stat and /proc/loadavg, and how do tools turn it into rates?

/proc/stat is the kernel's cumulative CPU accounting since boot. The first line, cpu, is the sum across all cores; subsequent cpu0, cpu1, … are per-core. Each row has ten counters (user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice) in units of USER_HZ (usually 1/100 s). Below the CPU rows you get aggregate counters for context switches, processes forked, processes running, and processes blocked. The values only ever increase.

Tools like top, vmstat, and mpstat work by reading /proc/stat twice with an interval between, subtracting, and dividing by the elapsed time to get a rate. That is why the first sample of vmstat 2 shows averages since boot, there is no previous sample to subtract. /proc/loadavg is much simpler: five fields, the three load averages, the runnable/total task count, and the most recently created PID. Knowing the shape of these files demystifies the tools that read them and lets you write your own monitoring scripts in five lines of Python or Bash.