Running Local Models on Linux

Everything so far in this part has depended on a network connection and someone else's hardware. This chapter removes both. Modern open-weight models are small enough, and the inference software efficient enough, that a capable assistant runs on an ordinary Linux machine with no account, no API key, and no packets leaving the building.

That is a genuinely new capability, and it is one that arrived on Linux first.

Why Run a Model Locally

Four reasons, in rough order of how often they turn out to be decisive.

Privacy. Text sent to a hosted model leaves your machine. For a hobby project that is unremarkable; for patient records, legal documents, unpublished research, or a client's proprietary source code it may be prohibited outright. A local model resolves the question by removing the transfer.

Offline operation. A model on local disk works on a train, on a ship, in a hospital basement, and in the field. Chapter 21 described AI as a product of vast connected infrastructure, which makes it easy to forget that the trained artefact is just a file.

Cost shape. Hosted inference is priced per token. Local inference costs electricity and the hardware you already own. For high-volume, repetitive work the arithmetic can flip decisively.

Control. The model does not change underneath you. A hosted endpoint may be updated or retired; a file on your disk is the same file next year, which matters for reproducible research.

The honest counterweight: a model you can run on a laptop is meaningfully less capable than the largest hosted ones. The gap narrows every year but it is real. Use a local model because one of the reasons above applies, not because it is uniformly better, because it is not.

The Hardware Question

Model size is quoted in parameters, and memory requirement follows from parameters and precision. Full precision uses two bytes per parameter, which puts even a small model out of reach of most machines. Quantisation reduces the precision of the stored weights, trading a little quality for a large reduction in memory.

Table 25.1: Approximate memory needed, by parameter count and quantisation

These are rough figures and exclude the additional memory consumed by the context window, which grows with the length of the conversation. Treat them as a guide to what will fit, not a specification.

A model that fits in GPU memory runs fast. A model that does not can still run on the CPU using system RAM, typically at a few tokens per second rather than tens, which is usable for a batch job and tedious for a conversation. Some inference engines split a model across both. The practical advice is simple: pick the largest model that fits comfortably in your GPU's VRAM, and if you have no discrete GPU, stay at the small end and be patient.

Check what you have:

nvidia-smi                      # NVIDIA: driver, VRAM, running processes
rocm-smi                        # AMD, with ROCm installed
free -h                         # System RAM
lscpu | grep -E 'Model name|CPU\(s\)'

The free -h and lscpu commands are the ones from Chapter 19. Capacity planning for a language model is capacity planning.

Ollama

Ollama is the most direct route to a working local model on Linux. It wraps the inference engine, handles downloading and caching models, and exposes an HTTP API.

curl -fsSL https://ollama.com/install.sh | sh

The same reservation as ever applies to piping a script into a shell, and the manual path exists for the same reason. It unpacks a tarball into /usr:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst \
  | sudo tar x -C /usr

An arm64 archive is published alongside it, and a separate ROCm package adds AMD GPU support. Started by hand, the server runs in the foreground:

ollama serve

Running It as a Service

Leaving a server running in a terminal is not how Linux does daemons, and Chapter 13 covered the alternative. The installer creates a unit file; if you installed manually, write /etc/systemd/system/ollama.service yourself. The important properties are a dedicated unprivileged user, a restart policy, and enabling it at boot:

sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl start ollama
sudo systemctl status ollama

The service runs as a dedicated ollama user whose home directory is /usr/share/ollama, and restarts automatically on failure. Downloaded models live under that home directory when running as a service, and under ~/.ollama when you run the server as yourself. Models are large, so check that whichever filesystem holds them has room before pulling several:

df -h /usr/share/ollama

Logs go to the journal, exactly like any other unit:

journalctl -u ollama -f

Using It

Pull a model and talk to it:

ollama pull <model>
ollama run <model>

The available models and their tags are listed at ollama.com/library, and the set changes often enough that naming specific ones in a book is a poor idea. Choose by the size that fits your hardware. Housekeeping commands follow the pattern you would expect:

ollama list          # models on disk
ollama ps            # models currently loaded in memory
ollama rm <model>    # reclaim the disk space

Local Models in Pipelines

The interesting part, for a Linux user, is that Ollama serves an HTTP API on localhost:11434. That turns a language model into an ordinary local service, and everything from Chapter 7 and Chapter 12 applies.

curl -s http://localhost:11434/api/generate -d '{
  "model": "<model>",
  "prompt": "Explain what a zombie process is, in two sentences.",
  "stream": false
}' | jq -r .response

Because it is just a service on a port, it composes:

journalctl -u nginx --since "1 hour ago" --no-pager \
  | curl -s http://localhost:11434/api/generate -d @- ...

Ollama also exposes an OpenAI-compatible endpoint, which means tools and libraries written for hosted APIs can often be pointed at your own machine by changing a base URL and nothing else. That is a quietly significant piece of interoperability: it lets you develop against a local model and deploy against a hosted one, or the reverse.

A small shell function makes it habitual:

ask() {
  curl -s http://localhost:11434/api/generate \
    -d "$(jq -n --arg p "$*" '{model:"<model>", prompt:$p, stream:false}')" \
    | jq -r .response
}

Put that in your ~/.bashrc and ask "what does the sticky bit do" becomes a command. This is the Chapter 14 lesson applied: if you do something twice, write a function for it.

llama.cpp and GGUF

Underneath Ollama, and underneath most of the local-inference ecosystem, sits llama.cpp: a C++ implementation of transformer inference written by Georgi Gerganov, originally to run a model on a MacBook and now the substrate for local inference nearly everywhere. It is worth knowing about for two reasons.

The first is GGUF, its model file format. A GGUF file holds the weights along with the metadata needed to run them, in a single file at a chosen quantisation level. It has become the interchange format for local models, which is why you can download one file and have several different programs run it.

The second is that llama.cpp is a lesson in the value of the plain approach. It has minimal dependencies, compiles with make, runs on CPU, and supports CUDA, ROCm, Metal, and Vulkan as optional backends. It is the kind of project that could have been written in 1995 in style if not in substance, and its portability is a direct consequence of that restraint.

If you want to work at that level, clone it and build it. If you want a working assistant, use Ollama and know that llama.cpp is what is running.

Choosing Between Local and Hosted

Table 25.2: When each approach fits

The two are not mutually exclusive, and the most sensible arrangement is often both: a local model for bulk work, routine classification, and anything sensitive, with a hosted model reserved for the problems that genuinely need the extra capability. Deciding which is which is an engineering judgement, and it is one you are now equipped to make, because it rests on the things this book has been about all along: what runs where, who can read what, what happens when it fails, and what it costs.

A Final Word

This book set out to explain the operating system that runs the internet, and it has ended up at the operating system that runs artificial intelligence, which turns out to be the same one. That is not a twist. It is the point.

Over twenty-five chapters we have gone from the kernel to the shell, through the filesystem, permissions, processes, networking, and services, into scripting, editors, and version control, out to containers, security, and observability, and finally to the machine learning tooling now built on top of all of it. The newest layer is genuinely new. The layer beneath it is not: a process is still a process, a file still has an owner and a mode, and an agent that runs shell commands is confined by the same namespaces that confine anything else.

That is the durable thing worth taking away. Tools arrive and are superseded, and the ones in this final part will look dated sooner than anything else in the book. The substrate is slower moving and much older. Every command-line assistant in Part 6 reads standard input and writes standard output because Doug McIlroy gave Unix pipes in 1972, and in more than fifty years nobody has found a better idea.

Get fluent in that substrate and you inherit the accumulated wisdom of more than fifty years of Unix and thirty-five years of Linux: a body of practice, tooling, and ideas so deep and so productive that engineers have built the modern internet on it, and are now building something else on it too. Welcome to the club. There is much to explore.