# INT8 Quantization the Second Time Around

I did per-row scalar quantization once already, on Lattice — my vector database — compressing stored vectors down to int8 so the index takes a quarter of the memory. This time it's the same technique, aimed at something with a very different failure mode: the actual weights of an LLM, not the vectors it searches over.
 
I want to walk through what carried over, what didn't, and the number I almost led with before double-checking it turned out to be the wrong one.
 
## Why this needed doing
 
Right now my engine loads Qwen3-0.6B's weights as f32. The file itself ships in bf16, so every weight gets widened on load — simpler math, but it means the "0.6B parameter model" actually sits at somewhere north of 2.4GB in memory before you've generated a single token. INT8 cuts each weight to a quarter the size. The question worth answering honestly wasn't "does this save memory" — obviously it does — it was "how much does it cost, and does the model still work."
 
## Per-row, not per-tensor
 
Each row of a weight matrix gets its own scale factor, rather than one scale for the whole matrix. The reason is that output channels genuinely differ in magnitude — some rows in a projection matrix carry values an order of magnitude smaller than others. A single scale sized to fit the loudest row crushes every quiet row down into a handful of usable int8 steps.
 
I checked this wasn't just theoretical. Built a synthetic weight matrix with half its rows scaled 50x smaller than the other half — the exact situation per-row scaling exists to handle — and measured both approaches:
 
```
per-row scales : rel RMS error = 0.80%
one global scale: rel RMS error = 1.33%
```
 
Real gap, not a rounding difference. The cost is four bytes of extra storage per row, which on a [3072, 1024] matrix is 12KB against 3MB of weights — free, essentially.
 
## Symmetric, and why the rounding function matters
 
No zero-point offset — zero maps to exactly zero, which fits transformer weights well since they're roughly centered around it already. Asymmetric quantization handles skewed distributions slightly better but adds a term to every dot product, and the skew isn't there to justify it.
 
The detail I hadn't thought about until I wrote it: `(int)value` truncates toward zero, which biases *every single weight* slightly downward. Across 28 stacked layers, a consistent directional bias compounds. Round-to-nearest-even has no such bias — `lrintf` instead of a cast, and it's the difference between random noise and a systematic drift that gets worse the deeper the stack goes.
 
## The scale factors out of the matmul entirely
 
This is the part that makes quantized inference actually make sense rather than just being a storage trick:
 
```
sum_k a[k] * w[j][k]  ==  scale[j] * sum_k a[k] * q[j][k]
```
 
The scale can be pulled out of the inner loop and applied once per output element instead of once per multiply-accumulate. On a [1, 1024] × [3072, 1024] matmul that's 3072 multiplications instead of 3.1 million.
 
## Real numbers, on the real model
 
Synthetic tests are a start, not the answer. I ran the actual quantizer over Qwen3-0.6B's real weights — all 196 projection matrices across 28 layers:
 
```
across all quantized weight matrices:
  f32:  1761.61 MB
  int8: 441.78 MB
  3.99x smaller
  worst tensor: model.layers.19.mlp.down_proj.weight at 1.2017%
```
 
Every one of the 196 tensors landed under 1.3% relative error, tightly matching the synthetic tests — real weights quantize just as cleanly as the Gaussian approximation predicted, which is itself a small useful result: I didn't need to special-case anything for the real distribution.
 
## The number I almost led with, and the bug that stopped me
 
3.99x is a good headline. It's also not the true whole-model number, and I caught that before writing it anywhere permanent.
 
The embedding table turns out to be the single largest tensor in the entire model — 622MB on its own — and it stays f32 on purpose, since embeddings are more precision-sensitive and comparatively small individually. Once you count it, "whole model" and "quantized matrices" stop meaning the same thing.
 
Then I found something worse while doing that math: my loader was loading `lm_head.weight` as a full separate copy of `embed_tokens.weight`, even though the config says `tied_embeddings: true` — meaning the two are numerically identical. An unrelated bug from days earlier, quietly costing an extra 622MB for nothing, that I only noticed because I was being careful about a completely different number.
 
```
As currently loaded (with the duplicate):  3006.5 MB -> 1686.7 MB   (1.78x)
If the lm_head duplication were also fixed: 2384.2 MB -> 1064.3 MB  (2.24x)
```
 
The honest number is 1.78x today, 2.24x once that bug's fixed — not 3.99x. I'd rather publish the number that's actually true than the one that photographs better.
 
## The test that actually mattered
 
Same prompt, same greedy sampling, f32 weights against int8 weights:
 
![The generate CLI producing identical output from f32 and int8 weights](https://cdn.hashnode.com/uploads/covers/6a7e8ed6e6bb04b206807b7b/544828f6-0b68-4ba2-9529-2c300874ed81.png)
 
Identical. Word for word. That's the real bar quantization has to clear — not "does the error percentage look small," but "does the model still say the same thing."
 
## What this didn't buy
 
Decode throughput barely moved — 1.83 to 1.88 tokens per second, essentially flat. Worth saying plainly rather than letting the memory number imply a speed win that isn't there: the current quantized matmul converts each int8 weight back to float before multiplying, with no vectorized int8 dot product underneath it. Today's result is memory, full stop. A genuinely faster quantized matmul is separate work, not something this claims.
 
![Weight memory before and after, quantized matrices vs the honest whole-model number](https://cdn.hashnode.com/uploads/covers/6a7e8ed6e6bb04b206807b7b/17dae1e0-1313-4937-833d-6b3223b88f7a.png)
 
## What carried over from Lattice, and what didn't
 
The technique is identical — per-row int8, symmetric, same math. What's different is what's actually at risk when it goes wrong. Lattice quantizes the *data being searched*; if the error creeps up, recall quietly drops — you get slightly worse search results. Here it's quantizing the *weights doing the computing*; error doesn't just sit there, it compounds through 28 layers of matrix multiplication before it ever reaches an output. Same tool, genuinely different failure mode, which is why the identical-output test mattered more here than a recall benchmark ever needed to on Lattice.
 
## Where it stands
 
Model runs correctly at a quarter the weight memory for the parts that matter most, on real hardware, with a number I trust because I checked it against a bug it accidentally exposed. Next up is CUDA — the kernels are already written and verified on a T4, the actual wiring into the forward pass is what's left, and that's where the quantized matmul finally gets to be fast instead of just small.