Frequently Asked Question

How much VRAM do I need to run a model locally?

Memory requirement follows from parameter count and quantisation level. As a rough guide, at 4-bit quantisation a 3-billion-parameter model needs around 2 GB, a 7-to-8 billion model around 5 GB, a 13-to-14 billion model around 8 GB, and a 70 billion model around 40 GB. Doubling to 8-bit roughly doubles those figures; full 16-bit precision roughly quadruples them.

Add headroom for the context window, whose memory use grows with the length of the conversation. A model that exactly fills your VRAM at load time will run out part way through a long session.

The practical rule is to choose the largest model that fits comfortably rather than exactly. On an 8 GB card, a 7-to-8 billion parameter model at 4-bit is a good fit; a 13 billion model at 16-bit is not, and will fall back to CPU.

Note that VRAM capacity matters more than raw compute throughput for this workload. A slower card with more memory beats a faster card with less.