Frequently Asked Question

What is ftrace and when do I reach for it instead of perf or bpftrace?

ftrace is the Linux kernel's internal function tracer, built in since 2.6.27 and maintained by Steven Rostedt. It lives entirely inside the kernel, exposes itself through the tracefs virtual filesystem mounted at /sys/kernel/tracing/, and can record every function the kernel calls, every scheduler event, every wakeup, every interrupt, and every tracepoint, with timestamps, optionally limited to particular functions, CPUs, or PIDs. The front-end command-line tool is trace-cmd and there is a GUI called kernelshark.

You reach for ftrace when you need to understand what the kernel itself is doing in fine detail: where is vfs_read spending its time, what sequence of functions handles a network packet, why does this driver IRQ sometimes take a millisecond. perf samples; bpftrace aggregates; only ftrace gives you the per-function, per-call timeline. The overhead is modest because the tracing code is built into every function as a five-byte NOP that becomes a call to the tracer only when enabled.