Frequently Asked Question

Why does my pipeline behave differently when output goes to a terminal vs a file?

The C standard library buffers stdio differently depending on what the stream is connected to. When stdout is a terminal it's line-buffered, the library flushes whenever you write a newline, so output appears promptly. When stdout is a pipe or a file it's block-buffered, the library only flushes when its internal buffer fills (typically 4 KB or 8 KB) or the program exits. This is what makes grep foo logfile | tee output.txt sometimes feel oddly silent for seconds at a time even though grep is finding matches: those matches are stuck in libc's buffer waiting for the buffer to fill.

The fix when you need real-time output is stdbuf -oL command ..., which sets stdout to line-buffered, or --line-buffered on tools that support it (grep does). Some programs offer their own flag, like -u for Python or unbuffer from the expect package which runs the command under a pseudo-terminal so libc thinks it's writing to a TTY. Behaviour of stderr is different, it's unbuffered by default, which is why error messages tend to appear immediately even when stdout is stuck.