Local LLMs on a 4-core VPS: what actually runs, and how fast

Benchmarked 2026-07-17 on the binary.ovh VPS. Every number below was measured on this exact box — not copied from a leaderboard. Try the results live at [brain.binary.ovh](https://brain.binary.ovh).

TL;DR

• Seven models tested, from 0.8B to a 1-bit-quantized 27B. Six are genuinely usable; one is a fascinating science experiment.
• Best daily driver: Qwen3.5 2B (14 tok/s) with thinking disabled, or LFM2 2.6B (13 tok/s) if you want answers that start instantly.
• Best quality that fits: Qwen3.5 4B at 6.6 tok/s — fine for "ask and wait", not for interactive back-and-forth.
• Biggest surprise: thinking models are a trap on CPU. Qwen3.5's reasoning mode burned 1,500–2,500 tokens of internal monologue before answering — that's 2–7 minutes of silence at CPU speeds. Disabling it (--reasoning-budget 0) matters more than picking the "best" model.
• Bonsai 27B (1-bit) actually loads and runs in 3.6 GB — a 27B-class model on a 7.6 GB VPS that keeps its other services running. Speed: see below.

The host

What this hardware implies:

1. Weights must fit in ~4.5 GB to leave room for KV cache, compute buffers, and the rest of the server.
2. Generation is memory-bandwidth-bound. Tokens/sec scales inversely with model bytes — a 2× smaller quantized model is roughly 2× faster, which is why aggressive quantization (Q4, and now 1-bit) is the whole game on CPU.
3. Prompt processing is compute-bound on 4 AVX2 cores. Long prompts (RAG, big pastes) hurt more than long answers.

Results: speed (llama-bench, 4 threads, pp512 / tg128)

Reading speeds: >12 tok/s reads faster than most people; 6 tok/s is a slow typist; below ~3 tok/s you'll tab away.

Results: quality spot-checks

Three tasks per model (temperature 0.2): write a dedupe function with doctests, the chickens-and-rabbits puzzle, and strict-JSON contact extraction.

"Died thinking" means: the model solved the problem inside its reasoning trace but exhausted the token budget (1.5–2.5k tokens) before emitting a final answer. On a GPU you'd never notice — at 14 tok/s you absolutely do. With thinking disabled, the Qwen models answer these correctly and immediately — that's now the default in the webapp.

The quiet winner is LFM2 2.6B: zero thinking overhead, correct on all three, cleanest structured output ("intent": "demo" — exactly the field you'd want in a CRM). Liquid AI built its hybrid conv-attention architecture specifically for CPU/edge, and it shows.

Bonsai 27B: the "27B on a phone" experiment

PrismML's Bonsai 27B (July 2026) compresses Qwen3.6-27B to 1.125 bits per weight — every weight is a single sign bit, with one FP16 scale per 128 weights. The whole 27B model is 3.6 GB, small enough to run in a browser via WebGPU, on an iPhone (~11 tok/s), and — the question that matters here — on a humble 4-core VPS.

• It needs PrismML's llama.cpp fork (custom Q1_0 kernels; upstream llama.cpp refuses the file). They ship prebuilt Ubuntu binaries, so no compilation needed.
• It loads in ~4 GB RSS alongside Atlas, herdr, Caddy and Docker on this 7.6 GB box — nothing had to be shut down. That alone would have been fantasy a year ago: 27B-class weights previously meant ~16 GB minimum at Q4.

Measured here: 2.4 tok/s prompt processing, 1.8 tok/s generation (1.4 tok/s in a live chat context with KV overhead). And the quality is real — asked the chickens-and-rabbits puzzle, it set up simultaneous equations in a clean reasoning trace and answered exactly 23. Correct, coherent, no quantization gibberish.

But here is the catch, in one number: that single puzzle took 19 minutes end-to-end. The model is a thinking model (Qwen3.6 lineage), so it reasons for a few hundred tokens before answering, and at 1.8 tok/s every hundred tokens of thought costs a minute. A 27B-class intellect that communicates at roughly the speed of a telegraph operator.

So the honest verdict for a 4-core Haswell VPS:

• ✅ It fits. 3.6 GB of weights, ~4 GB RSS, production services untouched. The 1-bit compression claim fully holds.
• ✅ It's coherent. No "!!!!" collapse, no garbage — the failure modes some HN users hit on other quants didn't appear in the Q1_0 CPU path.
• ❌ It's not usable interactively. 1.8 tok/s is batch-job territory. On this box the same 19 minutes buys you ~40 answers from Qwen3.5 2B.
• The speed marketing (44–87 tok/s) is Apple-silicon/GPU reality; binary weights don't add compute units to a 2014-era CPU core.

Community consensus (HN) matches what we saw: the compression is real and remarkable, the marketing tok/s numbers are GPU/Apple-silicon numbers, and CPU-server use is at the edge of practical. Quality-wise PrismML claims ~90% of FP16 across 15 benchmarks (coding holds up best at 81.9 HumanEval+-class); community testing found more hallucination and occasional reasoning loops than the benchmarks suggest.

Use-case guide for this box

What runs in a browser or phone now

The same 1-bit wave that made Bonsai fit this VPS is making real models run client-side: Bonsai 27B runs in Chrome via WebGPU (WGSL compute shaders) and on iPhone via MLX at ~11 tok/s. Gemma 4 E2B is explicitly designed for phones ("runs on 5 GB RAM at 4-bit"). The gap between "server model" and "device model" is collapsing — this VPS sits at exactly the hardware class where the two meet.

The webapp

Everything above is now a living thing: [brain.binary.ovh](https://brain.binary.ovh) (also ai.binary.ovh) — a model manager + chat for this box:

• Host capability detection — reads CPU/SIMD/RAM live and shows which models currently fit.
• On/off toggle per model — starts/stops a llama-server per model under a 4.5 GB RAM budget; turning on a big model auto-evicts smaller ones (oldest first). Prod services always keep ~800 MB headroom.
• Chat with streaming, per-model speed badges from the benchmarks above.
• 🌐 Web search grounding — optional DuckDuckGo search injected into the prompt with numbered citations.
• OpenAI-compatible API — POST /v1/chat/completions with Authorization: Bearer <password> proxies to whatever model is running.

Stack: FastAPI + vanilla JS (no build step), llama.cpp servers on localhost, Caddy in front. ~450 lines total.

Method notes

• Speed: llama-bench -t 4 -p 512 -n 128, 2 repetitions (1 for Bonsai), idle production services running.
• Quality: llama-server + /v1/chat/completions, temperature 0.2, max_tokens 1500–2500.
• Quantization: Q4_K_M everywhere it exists (Gemma ships q4_0 QAT, Bonsai ships Q1_0).
• All models pulled from Hugging Face: unsloth (Qwen3.5), google (Gemma QAT), ggml-org (SmolLM3), LiquidAI (LFM2), prism-ml (Bonsai).