Frequently Asked Question

What is quantisation and does it hurt quality?

Quantisation reduces the numerical precision used to store a model's weights. Training typically uses 16-bit floating point, which costs two bytes per parameter. Quantising to 8 bits halves that, and to 4 bits quarters it, making models runnable on hardware that could not otherwise hold them.

There is a quality cost, and it is not linear. The drop from 16-bit to 8-bit is usually negligible in practice. The drop to 4-bit is measurable but modest for most tasks, and is nearly always the right trade: a larger model at 4-bit generally outperforms a smaller model at full precision on the same memory budget.

Below 4 bits, degradation becomes obvious quickly. Formats such as Q4_K_M in the GGUF ecosystem apply different precision to different parts of the model, keeping more bits where they matter most, which is why they perform better than uniform quantisation at the same average size.