Answers · a straight answer, sourced
Can I run a 120B model on a gaming GPU?
Yes — if the model is a sparse mixture-of-experts (MoE) model, an engine like FreeToken keeps all the expert weights in system RAM and streams only the three-to-five billion parameters active per token to the GPU, so a single gaming NVIDIA card plus enough host RAM can run a 120B-total model. The real limit is not the GPU; it is that you need the RAM for the full weight set and, today, Linux with an NVIDIA card.
A dense model wants all its weights in GPU memory, which is why a 120B dense model needs a datacentre card. A sparse MoE model is different: it has 120B parameters total but only activates a small subset per token. FreeToken exploits that — the experts live in system RAM, the hot ones are cached on the GPU, and the rest stream over PCIe as tokens generate. The GPU only ever holds the active set, which is a few gigabytes.
So the sizing flips. Host RAM has to hold the total weights: roughly 0.6 GB per billion parameters at 4-bit, so about 70 GB for gpt-oss-120b, meaning 128 GB of RAM. GPU memory only has to hold the active set — a 4 GB card is enough for many of these. On the starter shelf, gpt-oss-20b and 120b, Qwen3.6-35B-A3B and Gemma 4 26B-A4B all run this way.
The honest limits: FreeToken is Linux x86_64 with an NVIDIA driver only today — not Apple Silicon, not the ARM-based DGX Spark, not AMD or Intel GPUs. On a MacBook you are on unified-memory dense models instead. And "runs" is not "runs fast": streaming experts over PCIe is slower than everything resident on the GPU. It is the difference between a class of model being impossible on your hardware and being usable on it.
Own the answer, not just read it
SovereignAI is private AI you run yourself: local models, memory with receipts, disclosure at the moment anything leaves, and an exit you can verify. The trial is one Docker command, and deleting the volume deletes every trace of you.