Textbook of Linux

perf is the Linux kernel's built-in performance analysis tool. It uses hardware performance counters and kernel tracepoints to profile CPU usage, cache behaviour, context switches, branch predictions, and much more, with very low overhead. Where strace is for deep per-syscall inspection, perf is for understanding where a program spends its time at the instruction level.

sudo perf top                           # live CPU profile
sudo perf stat ls                        # summary stats for a command
sudo perf record -g cmd                  # record profile with call graphs
sudo perf report                          # interactive view of recording
sudo perf sched record -- sleep 5        # scheduler trace
sudo perf trace                           # strace-like, lower overhead

perf's killer feature is flame graphs: hierarchical visualisations of stack traces where the width of each bar shows how much CPU time is spent in each function. Generated from perf record data by Brendan Gregg's open-source tools, flame graphs make it immediately obvious where a program's time is going.

perf requires kernel support and linux-tools-* packages on Debian/Ubuntu or perf on Fedora. For extremely low-overhead observability, perf pairs well with eBPF tools (bcc, bpftrace) that share the kernel infrastructure. Between them, they are the modern Linux performance analyst's two most powerful instruments.