Qwen3.6-35B-A3B on a 12GB RTX 3060 with llama.cpp

Qwen3.6-35B-A3B on a 12GB RTX 3060 with llama.cpp

Source: https://x.com/i/status/2077472690156511643

📌 Alibaba’s sparse MoE model Qwen3.6-35B-A3B (~35B total, ~3B active) can run fully locally on an RTX 3060 12GB plus 16GB RAM via llama.cpp. Under-discussed flags—single-slot context, CPU MoE offload, and quantized KV—make a model that “won’t fit” practical on consumer hardware.

🖥️ The headline run

July 2026 demo (Doğukan / @DogukanUrker): Q4_K_XL GGUF via llama-server on RTX 3060 12GB + 16GB RAM. Claimed ~45 tok/s, 150K context, and vision with the multimodal projector kept on CPU—no cloud or API.

🧮 Memory math, not magic

Default n_parallel=4 multiplies KV VRAM (~4×). -np 1 keeps one slot so long context fits. --n-cpu-moe 26 parks expert weights in system RAM; attention/shared path stays on GPU. Full quantized footprint is often ~20GB+, but hybrid offload fits 12GB.

⚙️ Key llama.cpp flags

-np 1; --n-cpu-moe 26; --cache-type-k/v q8_0; --no-mmproj-offload; -ngl 99; flash attention on. Quantized KV halves cache; no mmproj offload saves VRAM for vision.

🔑 Why these flags matter

-np 1 stops parallel KV slots from OOMing long context. --n-cpu-moe offloads MoE expert tensors to RAM so the GPU only holds what must stay hot. Same pattern works on other 8–12GB cards (often with slower prefill).

💡 Broader takeaway

Sparse MoE + hybrid CPU/GPU offload + single-slot long context + KV quant turns mid-range GPUs into hosts for strong agentic/coding multimodal open models. Less “needs 24GB,” more “flags make consumer gear enough.”

Key facts

Details

A July 2026 X post showed that Qwen3.6-35B-A3B—a sparse mixture-of-experts model with ~35B total parameters and ~3B active—can run fully on a single RTX 3060 (12GB) plus 16GB system RAM. The setup used a Q4_K_XL GGUF through llama-server and claimed roughly 45 tokens/s at 150K context with vision enabled via a multimodal projector kept on the CPU. The point of the post was that “everyone says it won’t fit,” but a small set of under-discussed llama.cpp flags makes local use practical without cloud or API access.

What matters is memory accounting. llama.cpp defaults n_parallel to 4, which allocates four concurrent KV-cache slots and can roughly quadruple KV VRAM; -np 1 keeps a single slot so long context fits. --n-cpu-moe 26 parks many MoE expert weights in system RAM while attention and the shared path stay on the GPU, which is why a model whose full quantized footprint is often cited around 20GB+ can still run on 12GB. Quantized KV (q8_0 for K and V) further halves cache memory, and --no-mmproj-offload avoids burning VRAM on the vision projector.

That pattern is now common in the local-LLM community: hybrid CPU/GPU MoE offload, single-slot long context, and KV quantization. Others report similar wins on 8–12GB GPUs, sometimes trading lower prefill speed. The takeaway is less “35B needs a 24GB card” and more “with sparse MoE and careful llama.cpp flags, consumer hardware can run models that used to look server-only.”

Sources

• Running Qwen3.6-35B-A3B on a single RTX 3060 (original post)
• Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All
• Qwen/Qwen3.6-35B-A3B on Hugging Face
• Qwen3.6 - How to Run Locally (Unsloth)
• New llama.cpp options make MoE offloading trivial: --n-cpu-moe
• Understanding MoE Offloading (DEV Community)
• Performant local mixture-of-experts CPU inference with GPU acceleration (Hugging Face blog)
• Qwen3.6-35B-A3B-APEX / 128K ctx on RTX 3060 12GB (community benchmark)