Frequently Asked Question

What is a flame graph and how do I read one?

A flame graph is a stack-sampled profile drawn as a wide, layered diagram. The x-axis is the population of samples (alphabetical within each layer, not time-ordered), the y-axis is stack depth, and each rectangle is a function. A function's width is the fraction of samples in which it appeared on the stack; its colour is usually decorative. Functions that were on-CPU when sampled appear at the top of their stack, those flat plateaus along the top edge are what your CPU was actually doing.

To read one: scan along the top for the widest flat tops, then look down to see which call paths led there. A tall, narrow tower is deep recursion that isn't hot. A wide pyramid is a hot subsystem. You generate them from perf record data using Brendan Gregg's FlameGraph scripts, or directly from bpftrace profile aggregations. Differential flame graphs compare two profiles (before-and-after a change) and colour growth red, shrinkage blue the most direct way to see where a regression came from.