Blog · Guide · August 25, 2026

FreeToken in SovereignAI: frontier-class sparse models on the GPU you have

FreeToken is an engine from a UC Berkeley lab that runs 20B–120B mixture-of-experts models on one consumer NVIDIA card by keeping the experts in host RAM. SovereignAI recognizes it as a first-class local engine. This guide takes you from a bare machine to gpt-oss or Qwen3.6 answering in your own command center — and is explicit about what FreeToken is not.

By the SovereignAI team at Unify Dynamics · about ten minutes · commands checked against FreeToken's docs on the date above

Why a second engine at all

Ollama is the default engine for a reason: it runs everywhere and the small shelf fits any laptop. But the models that changed the conversation this year are sparse — gpt-oss-20b and 120b, Gemma 4 26B-A4B, Qwen3.6-35B-A3B — tens of billions of parameters in total with only three to five billion active per token. A dense engine wants all of those weights in GPU memory. FreeToken does not: it keeps the expert weights in system RAM, caches the hot ones on the GPU, and streams the rest over PCIe as tokens are generated. The result is that the machine you already own — a 4 GB laptop card and 32 GB of RAM, say — runs a class of model that used to need a workstation.

SovereignAI treats FreeToken the way it treats Ollama: a local engine, chosen per persona, disclosed on screen, never a dependency. If you never install it, nothing changes.

What it needs

Linux x86_64 — the command-line engine's only supported platform. Not the DGX Spark's ARM CPU, not Apple Silicon. Windows users get a desktop app instead (below).
An NVIDIA GPU with driver r580 or newer (CUDA 13), and the CUDA 13 toolkit with nvcc on your path — kernels are compiled on first use, so the first start of each model takes a while.
Python 3.10+, with uv recommended.
Host RAM sized to the model's total parameters, because every expert lives there: think 32 GB for the 20–26B class, 48 GB for Qwen3.6-35B-A3B, and 128 GB for gpt-oss-120b. GPU memory sized to the active set: a few gigabytes is enough.

Step 1 — install and serve

uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"

# serve a Hugging Face id or a local path; dtype and cache sizes resolve from the checkpoint and your GPU
ft serve --model Qwen/Qwen3.6-35B-A3B
# ... wait for: API server is ready to serve on 127.0.0.1:1919

Check it from another terminal. The model id it reports is the one SovereignAI will need:

curl http://127.0.0.1:1919/v1/models      # the served model id (defaults to the basename of --model)
ft ctl health                              # status, model, load progress

Windows: FreeToken ships a desktop app for Windows and Linux at flashml.ai that sets the engine up and gives you a GUI for running and tuning models; it exposes the same local API. Everything below applies unchanged.

Two flags worth knowing: --served-model-name if you want a short id, and --host 0.0.0.0 only when SovereignAI runs in a container on the same box (Step 4) — FreeToken has no request authentication, so never expose it beyond a machine you trust.

Step 2 — enable it in SovereignAI

Run the doctor first. If a FreeToken engine is serving on the default port and you have not enabled it yet, the doctor says so rather than staying silent:

sovereign doctor
# [info] FreeToken — running at http://127.0.0.1:1919 (Qwen3.6-35B-A3B) but not enabled
#        Enable FreeToken in Settings → Providers to chat with the model it is serving.

If this is a fresh install and the engine is already serving, the first-run wizard finds it: the "pick its intelligence" step shows a FreeToken · running on this machine choice with the model it is serving, and picks it for you when no Ollama is available. Nothing is contacted but loopback, and finding the engine does not enable it — you still choose.

On an install that is already set up, do it yourself: Settings → Providers → FreeToken. Tick enabled; the base URL defaults to http://127.0.0.1:1919; there is no API key field, on purpose. Set the default model to the id from /v1/models — FreeToken serves one model per process, so this is a single choice, not a list. Save. The provider pill turns ready with the engine's version and the model it is serving; if it is still loading, the pill says so with a percentage, because FreeToken's health endpoint reports load progress and SovereignAI reads it.

Prefer the shelf? Model Studio → the starter shelf → "Frontier-class, locally (sparse MoE)" lists the FreeToken-served entries with two badges — the RAM badge for the total weights and the GPU badge for the active set — and a one-click use as default model. If FreeToken is not enabled yet, the button tells you what to do instead of silently failing.

Step 3 — pick the model for your RAM

The shelf's rule of thumb is about 0.6 GB per billion parameters at four-bit, against roughly 60% of your memory; below is how the FreeToken-served tier lands on common machines. All figures are the model cards' parameter counts.

32 GB of RAM gpt-oss-20b (21B total, 3.6B active — OpenAI's open reasoning model in its native MXFP4) and Gemma 4 26B-A4B fit; Qwen3.6-35B-A3B is borderline.
48 GB of RAM Qwen3.6-35B-A3B — the model FreeToken was built around — sits comfortably; everything above it too.
128 GB of RAM gpt-oss-120b (117B total, 5.1B active): its experts are about 70 GB at four-bit and every one of them lives in host RAM. Near-frontier reasoning on a single desktop GPU.
GPU memory The active set is what matters: ~1.8 GB for a 3B-active model, ~3 GB for gpt-oss-120b's 5.1B. A 4 GB card handles the 3–4B-active models; 8 GB handles all of them.

FreeToken's known-good list also includes Qwen3.6-27B dense, DeepSeek-V4-Flash, GLM-5.2 and GLM-4.7 in NVFP4, MiniMax-M2.5, and Muse-Glimmer-30B; other checkpoints of the same architectures work too. Two things it does not serve, so use Ollama for them: Qwen3.8-27B and Nemotron 3.5 Lightning.

Step 4 — the Docker wiring

If SovereignAI runs in the trial container and FreeToken runs on the host, loopback inside the container is not the host's loopback. Bind FreeToken to all interfaces and hand the container the host alias:

ft serve --model openai/gpt-oss-20b --host 0.0.0.0

docker run -d --name sovereign -p 127.0.0.1:4321:4321 -v sovereign:/state \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -e FREETOKEN_BASE_URL=http://host.docker.internal:1919 \
  -e SOVEREIGN_TOKEN=pick-a-long-secret ghcr.io/mlmrx/sovereignai:latest

Setting FREETOKEN_BASE_URL both points at the engine and enables the provider — setting the URL is the intent. Because --host 0.0.0.0 opens an unauthenticated port to your network, do this only behind a firewall you trust; SovereignAI still treats the host alias as local, so the customs declaration will not ask before sending to it.

What you will see in chat

The sparse models think out loud, and FreeToken streams that reasoning on the wire. SovereignAI shows it in a collapsible panel above the answer — Reasoning · shown live, not saved — that folds away when the answer starts. None of it is written to the database, extracted into memory, or exported; only the answer is. Every reply from a local model also notes the model that answered, and because FreeToken is local, nothing leaves your machine and the outgoing-preview dialog never appears for it.

When it goes wrong

"still loading (…%)" — the engine is compiling kernels or paging the checkpoint in; run ft ctl health and wait. First start of a model is the slow one.
"not serving right now (rebuilding / stopping)" — the engine is between states; try again in a moment.
"unexpected shape — is this a FreeToken server?" — something else answered at that URL. Check the base URL in Settings; Ollama lives on 11434, FreeToken on 1919.
Default model missing — the id in Settings does not match what /v1/models reports. Copy it exactly, or use --served-model-name to make it short.
Slow generation — check RAM first: if the experts do not fit, the engine is swapping. Drop to the next model down rather than fighting it.

The limits, stated

FreeToken is weeks old and NVIDIA-only. It has no request authentication, so keep it on loopback unless you have a reason and a firewall. It serves one model per process. Multimodal models are served text-only. Our sizing is a rule of thumb, not a measurement — the shelf links to independent benchmarks for the measured numbers. And the model layer remains borrowed: open weights are still someone else's artifact, whichever engine runs them.

The frontier tier comes home.

Two processes on one machine, nothing in the cloud, and a ledger that says what each one guarantees.

Run it now

Sources

FreeToken — repository, install, quick start, CLI reference, supported models.
Parameter counts — the model cards for gpt-oss-120b, gpt-oss-20b, gemma-4-26B-A4B-it, Qwen3.6-35B-A3B.
How SovereignAI integrates it is decision record ADR-25 in the product's architecture notes; the Sovereignty Ledger records the model-layer caveats.