Frequently Asked Question

Can I run a large language model without a GPU?

Yes. Inference engines such as llama.cpp, which sits underneath Ollama and most of the local-model ecosystem, run entirely on the CPU using system RAM. The constraint is speed rather than possibility.

A quantised model on CPU typically produces a few tokens per second, against tens on a GPU that holds the whole model in video memory. That is tedious for conversation and perfectly workable for a batch job: summarising a directory of documents overnight does not care whether each answer takes two seconds or thirty.

Some engines split a model between GPU and CPU, keeping as many layers as will fit in VRAM and running the rest on the processor. This degrades gracefully rather than failing outright, which is why a model slightly too large for your card still runs, just more slowly.

If you have no discrete GPU, stay at the small end of the model range and be patient.