Frequently Asked Question

What is GGUF?

GGUF is the model file format used by llama.cpp, and by extension across most of the local-inference ecosystem. A single GGUF file contains the weights at a chosen quantisation level together with the metadata needed to run them: the tokeniser, the architecture parameters, and the prompt template.

Its significance is interoperability. Because it is self-contained and widely supported, you can download one file and have several different programs run it, in much the way a single container image runs under Docker or Podman. That has made it the de facto interchange format for open-weight models.

llama.cpp itself is worth knowing about beyond the format. Written in C++ by Georgi Gerganov, it has minimal dependencies, compiles with make, runs on CPU, and supports CUDA, ROCm, Metal, and Vulkan as optional backends. Its portability is a direct result of that restraint, and it is the engine running underneath Ollama.