What actually survives INT8 on the edge

Quantization is the difference between a model that runs in a datacenter and one that runs on the device in someone’s hand. The theory is simple: store weights and activations as 8-bit integers instead of 32-bit floats, and you get roughly 4x smaller, 2-4x faster, and a fraction of the power draw. The practice is where the interesting failures live.

Where INT8 breaks first

Not all layers quantize equally. In our experience the casualties are predictable:

• Attention softmax: the dynamic range of logits is brutal; naive INT8 here collapses long-range dependencies.
• LayerNorm / residual adds: small errors accumulate across depth.
• The first and last layers: input embeddings and the output head are disproportionately sensitive.

The usual fix is mixed precision: keep the sensitive layers in INT16 (or FP16) and quantize the bulk to INT8. On Qualcomm’s stack this maps cleanly onto the QAIRT graph.

Measuring on the device, not the notebook

A model that scores well on your validation set can still fall apart on the NPU, because the on-device kernels round differently than your training framework. We always close the loop on real hardware:

# Compare on-device INT8 output against the FP32 reference, per layer.
def layerwise_drift(fp32_ref, int8_dev, atol=1e-2):
    drift = {}
    for name, ref in fp32_ref.items():
        dev = int8_dev[name]
        rel = (ref - dev).abs().mean() / ref.abs().mean().clamp_min(1e-6)
        drift[name] = rel.item()
    # Flag the layers that moved the most - those are your INT16 candidates.
    return dict(sorted(drift.items(), key=lambda kv: -kv[1]))

Run that once and the layers to promote to higher precision fall out of the top of the list. It turns “quantization is an art” into a short, ranked worklist.

If it doesn’t run on the device, it isn’t done. A benchmark on an A100 tells you nothing about what ships.

The short version

Technique	Cost	When it pays off
Post-training INT8	Low	Vision backbones, robust CNNs
Mixed INT8 / INT16	Medium	Transformers, attention-heavy models
Quantization-aware	High	Last mile on accuracy-critical tasks

Start cheap, measure on-device, and only pay for QAT when the drift report says you have to. That discipline is most of the job.