Answers · a straight answer, sourced
How much RAM do I need to run a local LLM?
A dependable rule of thumb: at 4-bit quantisation a model needs roughly 0.6 GB of memory per billion parameters, and you can count on about 60% of your system RAM being usable for it — which nets out to a simple test, a model fits comfortably when its size in billions of parameters is below your RAM in gigabytes.
Worked examples at 4-bit (Q4): an 8B model needs about 5 GB and is comfortable on 16 GB; a 27B model needs about 16 GB and is tight on 32 GB but comfortable from 48; a 70B-total model needs about 42 GB and wants 96–128 GB. Add headroom for the context window — a long context can cost several more gigabytes.
Two refinements. First, quantisation is the dial: the same model at 8-bit needs roughly double, at 2-bit roughly half (with more quality loss). Q4 is the usual sweet spot. Second, sparse MoE models are sized on total parameters for RAM even though only a fraction is active per token, because the whole weight set still has to live in memory — see running a 120B model on a gaming GPU.
You do not have to do this arithmetic by hand. SovereignAI’s starter shelf sizes every model against the machine it is running on before anything downloads, and `sovereign doctor` will tell you whether your configured default model actually fits — "~16 GB at Q4 against ~19 GB of usable RAM: tight; comfortable from 48 GB" — so you find out before the download, not after.
Own the answer, not just read it
SovereignAI is private AI you run yourself: local models, memory with receipts, disclosure at the moment anything leaves, and an exit you can verify. The trial is one Docker command, and deleting the volume deletes every trace of you.