Evaluating quantised models: perplexity, KL divergence & real task evals
How do I know IQ3_M is still good enough? Measuring the quality loss of a quantisation – with llama-perplexity, KL divergence against the base model, and a small task eval of your own.
In the quantisation post I got Qwen3-32B onto 16 GB – but one question stayed open: how do I know IQ3_M isn’t secretly dumber? “Runs and sounds plausible” is not a measurement. Here are the three levels at which I actually evaluate a quantisation.
Level 1: Perplexity – useful but deceptive
Perplexity (PPL) measures how “surprised” the model is by real text – lower is better. llama.cpp ships the tool:
# Grab a test text (e.g. wikitext) once, then measure per quant
./llama-perplexity -m qwen3-32b-bf16.gguf -f wiki.test.raw -ngl 99
./llama-perplexity -m qwen3-32b-IQ3_M.gguf -f wiki.test.raw -ngl 99Typically you get something like PPL = 6.41 (bf16) vs. PPL = 6.78 (IQ3_M). The trap: PPL is an average over generic text. A model can look almost identical on wikitext and still collapse on your tasks. PPL is a smoke detector, not a verdict on quality.
Level 2: KL divergence – the honest comparison
Far more telling is the question: how far does the quantised model’s probability distribution deviate from the original’s? That is exactly what KL divergence measures – and llama.cpp can compute it directly against a base model.
# 1. Record reference logits from the full model once
./llama-perplexity -m qwen3-32b-bf16.gguf -f eval.txt \
--kl-divergence-base qwen3-32b.kld -ngl 99
# 2. Compare the quantised model against that reference
./llama-perplexity -m qwen3-32b-IQ3_M.gguf -f eval.txt \
--kl-divergence-base qwen3-32b.kld --kl-divergence -ngl 99Among other things this gives Mean KLD and – the value I care about most – the share of tokens where the top prediction changes. Rule of thumb from my practice:
Mean KLD < 0.1→ practically indistinguishable from the original.0.1–0.5→ usable, slight drift.> 0.5→ visible quality loss, quantised too aggressively.
Unlike PPL, KLD does not say “is the model good” but “how much of the original is left” – exactly the question with quantisation.
Level 3: A real task eval – the only one that counts
In the end, what matters is whether the model solves my tasks. For that I keep a small set of fixed prompts with a checkable answer – a mini eval I run against every quant. Via the llama-server endpoint that takes a few lines of Python:
import json, urllib.request
CASES = [
{"prompt": "Return ONLY the number: 17 * 23 =", "expect": "391"},
{"prompt": "Answer in one word: capital of Australia?", "expect": "Canberra"},
{"prompt": "Valid JSON for {name: Alain, year: 2026}, object only:",
"expect": '{"name": "Alain", "year": 2026}'},
]
def ask(prompt: str) -> str:
body = json.dumps({"prompt": prompt, "temperature": 0, "n_predict": 32}).encode()
req = urllib.request.Request("http://localhost:8080/completion", body,
{"Content-Type": "application/json"})
return json.loads(urllib.request.urlopen(req).read())["content"].strip()
passed = sum(c["expect"] in ask(c["prompt"]) for c in CASES)
print(f"{passed}/{len(CASES)} passed")The key is temperature: 0 – deterministic, otherwise you’re measuring noise. The cases should hit your domain: if you use the model for code, put code tasks in (just like the imatrix calibration text in the quantisation post). For softer tasks you swap the exact match for an LLM-as-judge – but then deliberately with a stronger model as the grader.
The workflow in one sentence
PPL as a fast smoke detector, KL divergence as an honest measure of “how close to the original”, and your own task eval as the last word. Only this trio turns “feels okay” into a decision: keep IQ3_M or go one level higher after all.
Related: quantising Qwen3-32B by hand produces the models that go on the test bench here.