Quantising Qwen3-32B by hand – a 32B model on 16 GB of VRAM

Running a dense 32-billion-parameter model on a GPU with 16 GB of VRAM sounds like one size too big – and in full precision it is. With manual quantisation via llama.cpp and an importance matrix, though, Qwen3-32B fits onto the card just fine. Here is the complete path I took – including the maths for why it works at all.

The VRAM problem

Qwen3-32B has ~32.8B parameters spread across 64 transformer layers. The raw weight size scales linearly with bits per weight:

(Figures rounded – actual file sizes vary slightly.)

That leads to the real decision: Q4_K_M (~20 GB) only runs with partial offload (some layers on the CPU), while IQ3_M (~15 GB) fits almost entirely on the GPU. And this is exactly where the importance matrix matters.

Step 1: Fetch the model & convert to GGUF

# Original weights from Hugging Face
huggingface-cli download Qwen/Qwen3-32B --local-dir ./Qwen3-32B

# Convert to a full-precision GGUF (the starting point for quantisation)
python llama.cpp/convert_hf_to_gguf.py ./Qwen3-32B \
  --outfile qwen3-32b-bf16.gguf \
  --outtype bf16

The bf16.gguf is the ~66 GB intermediate – it only lives on disk, not in VRAM. Every quantised variant is derived from it.

Step 2: Compute the importance matrix (imatrix)

Naive quantisation treats every weight equally. The imatrix instead measures, on real text, which weights matter for the outputs and protects those during quantisation. At low bit widths (anything under 4 bit) this is the difference between “usable” and “spouts nonsense”.

# Calibration text: a few hundred KB, representatively mixed
#   – prose, code and a slice of your own domain
./llama-imatrix \
  -m qwen3-32b-bf16.gguf \
  -f calibration.txt \
  -o qwen3-32b.imatrix \
  --chunks 200 \
  -ngl 20

What matters is the composition of calibration.txt: if you later use the model for code, put code in; if you work in German, German text. The imatrix optimises for exactly what it gets to see.

Step 3: Quantise

# Fully on GPU – the smallest usable level, thanks to the imatrix
./llama-quantize \
  --imatrix qwen3-32b.imatrix \
  qwen3-32b-bf16.gguf \
  qwen3-32b-IQ3_M.gguf \
  IQ3_M

# Higher quality, but requires partial offload
./llama-quantize \
  --imatrix qwen3-32b.imatrix \
  qwen3-32b-bf16.gguf \
  qwen3-32b-Q4_K_M.gguf \
  Q4_K_M

Budgeting VRAM & choosing -ngl

Weights are not the only thing that has to go into VRAM. The real budget:

usable VRAM ≈ 16 GB
           − KV cache        (grows with context length)
           − compute buffer  (~0.5–1 GB)
           − OS/driver       (~0.5–1 GB)

The KV cache is the biggest variable item. Qwen3-32B uses grouped-query attention with 8 KV heads – that keeps the cache small, but at 8k context in fp16 it is still ~1–2 GB. Two levers cut it down:

--flash-attn          # more efficient attention, smaller buffer
-ctk q8_0 -ctv q8_0   # quantise the KV cache itself to 8 bit → ~half the need

That leaves ~13.5–14 GB for the weights. The layer maths for Q4_K_M:

weights ≈ 20 GB / 64 layers ≈ 0.31 GB per layer
13.5 GB usable / 0.31 GB    ≈ 43–44 layers on the GPU

So -ngl 44: 44 layers run on the GPU, the rest on the CPU. With IQ3_M (~15 GB), by contrast, practically all 64 layers fit – -ngl 99 (llama.cpp caps at the maximum) – and it runs entirely without CPU offload.

Step 4: Launch

# Variant A – IQ3_M, fully on the GPU (fast)
./llama-server \
  -m qwen3-32b-IQ3_M.gguf \
  -ngl 99 \
  -c 8192 \
  --flash-attn \
  -ctk q8_0 -ctv q8_0 \
  --host 0.0.0.0 --port 8080

# Variant B – Q4_K_M, partial offload (higher quality, slower)
./llama-server \
  -m qwen3-32b-Q4_K_M.gguf \
  -ngl 44 \
  -c 8192 \
  --flash-attn \
  -ctk q8_0 -ctv q8_0

Optional: wire it into Ollama

Since the rest of my setup (the LLM wiki à la Karpathy) runs on Ollama, the finished GGUF moves there via a Modelfile:

# Modelfile
FROM ./qwen3-32b-IQ3_M.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 99

ollama create qwen3-32b-local -f Modelfile
ollama run qwen3-32b-local

Result & takeaways

Qwen3-32B runs on 16 GB – and does so usefully. What I took away:

1. Fully on the GPU beats partial offload. A lower, completely loaded quant (IQ3_M) is noticeably faster than a higher quant that computes half on the CPU – the per-token PCIe transfer costs more than the better quant gains back.
2. The imatrix is not optional below 4 bit. Without it, IQ3_M is markedly dumber; with proper calibration it lands surprisingly close to Q4.
3. KV-cache quantisation + flash attention are the quiet win: they create the headroom to fit either more context or a few more layers onto the GPU.
4. Calibration data is a domain matter. The imatrix is only as good as the text you show it – for coding tasks, code belongs in there.

Manual quantisation is ultimately a budgeting problem, not magic: trade bits per weight against KV cache against layer offload until the big model fits the small card.

[Top]