LLM Quantization:
A Comprehensive Guide

Introduction

Llama 3.1 405B (Dubey & others, 2024) stores 405 billion parameters as 16-bit floats. That is 810 GB of weights, enough to fill nine 94 GB H100 GPUs (NVIDIA, 2023) before the model has read a single token. Store each parameter in 4 bits instead, and three GPUs are enough.

That is quantization: storing a model’s numbers with fewer bits, typically 8 or 4 instead of 16. Memory drops by half or by three quarters. Generation gets faster too, because producing a token is mostly a matter of reading the weights from memory, and smaller weights take less time to read.

The price is precision. A 4-bit number can take only 16 values, so every weight gets rounded, and a few billion rounding errors add up. The craft lies in choosing those 16 values so well that the model’s answers barely change.

One problem runs through all of it: outliers, the few values far larger than the rest, which stretch every grid they share. Most of the methods in this article are, at heart, ways of dealing with them.

We start with the bill, what a model costs in memory and bandwidth (Why do we need quantization?). Then we open up the numbers: how formats spend their bits, and how a scale maps real values onto a coarse grid (Basic concepts). A taxonomy of quantization methods lays out the choices every method makes. Advanced post-training quantization covers the methods behind most published 4-bit models: GPTQ (Frantar et al., 2023) and AWQ (Lin et al., 2024) for weights, SmoothQuant (Xiao et al., 2023) for activations, rotations that spread outliers out, SINQ (Müller et al., 2025), which needs no sample data, and KIVI (Liu et al., 2024) and TurboQuant (Zandieh et al., 2025) for the attention cache.

Every figure computes its values in your browser. Change the inputs and watch each method succeed, or fail.

Why do we need quantization?

Serving a model costs two things: memory to hold it and time to run it. Both grow with the number of bytes each parameter takes, and that is exactly the number quantization cuts.

Computing memory requirements

The memory bill is a single multiplication. A model with $P$ parameters, each stored in $B$ bytes, needs

The number format sets $B$: 4 bytes for FP32, 2 for FP16 and BF16 (Kalamkar & others, 2019), 1 for FP8 (Micikevicius & others, 2022) and half a byte for INT4. For now only their size matters, and Numerical formats explains the rest. A 70B model in BF16 takes $70 \times 10^9 \times 2 = 140$ GB, already more than a single H100 can hold.

For the largest Llama 3.1, the same arithmetic turns into a hardware bill.

Each halving of the bits roughly halves the hardware.

KV cache requirements

The weights are not alone in memory. Every token a model reads or writes adds to the KV cache, the attention keys and values of all past tokens, which each layer keeps so that it does not recompute them at every step. For a model with $L$ layers and $n_h$ key-value heads of dimension $d_h$, serving $N$ sequences of $T$ tokens, the cache takes

The 2 counts keys and values, and every other factor multiplies: double the context or the batch, and the cache doubles. Grouped-query attention (Ainslie et al., 2023) keeps $n_h$ smaller than the number of query heads. All Llama 3.1 models use 8 key-value heads of dimension 128 (Dubey & others, 2024).

At long contexts, the cache rivals the weights. Llama 3.1 70B has 80 layers, so one sequence of 131,072 tokens fills 42.9 GB of BF16 cache, a third of its 140 GB of weights. Serve eight such sequences and the cache reaches 344 GB, more than twice the weights. For more details, see What can be quantized?.

The memory calculator adds the two up. Pick a model, a GPU (an H100 or an H200 (NVIDIA, 2024)), a context length and a batch size. Each bar is one precision for the weights, and the dashed lines mark how much one, two, three or more GPUs hold.

Compute-bound and memory-bound phases

Inference has two phases:

• Prefill processes the prompt. All its tokens go through each layer together, so every weight read from memory is reused once per token. Attention adds work that grows with the square of the prompt length. Prefill is compute-bound: the arithmetic units set the pace.
• Decode generates the answer one token at a time. Each step multiplies a single vector by every weight matrix, so each weight makes the trip from memory for just two operations, a multiply and an add. Decode is memory-bound: the arithmetic units wait for data.

The roofline model makes this precise (Pope et al., 2023; Williams et al., 2009). A kernel has two speed limits: the GPU’s peak arithmetic, and its memory bandwidth times the kernel’s arithmetic intensity, the operations it does per byte it loads. The lower of the two wins.

A linear layer that processes $n$ tokens per weight read does $2n$ operations per $B$ bytes. An H100 SXM moves 3.35 TB/s and computes 989 TFLOPS in BF16 (NVIDIA, 2023), so it needs about 295 tokens per weight read before arithmetic becomes the limit. Decode a single sequence and the GPU uses 0.3% of its compute. The rest sits idle, waiting for memory.

The roofline plots this for three weight formats. Each solid line is the lower of the two limits. Highlight a format to draw both limits as dashed lines, with the ridge where they cross, and the table below the chart shows which resource is saturated: one of compute and bandwidth is always at 100%.

The memory bandwidth bottleneck

For a single sequence, the speed limit takes a simple form. Each decoding step reads every weight once, so it takes at least the size of the weights divided by the bandwidth:

Halve the bytes and you double the ceiling. Llama 3.1 8B in BF16 takes 16 GB, so an H100 SXM generates at most about 209 tokens per second with it. In INT4 the model takes 4 GB, and the ceiling rises to about 840. Real systems fall short of these numbers, since they also read the KV cache and pay for attention and kernel launches, but the ratio holds.

Basic concepts

Fewer bits have a price: every number gets rounded to one of a few values. How much that hurts depends on two choices, the number format and the scale.

Numerical formats

A floating-point number has three fields: a sign bit, an exponent and a mantissa. Its value is

The exponent picks the power of two, and the mantissa the position between that power and the next:

• Exponent bits set the dynamic range, the span from the smallest to the largest number.
• Mantissa bits set the precision. With $m$ of them, neighbouring numbers are at most $2^{-m}$ of the number apart.

Deep learning uses a handful of formats (IEEE, 2019; Micikevicius & others, 2022; Open Compute Project, 2023):

BF16 keeps the exponent of FP32, and with it the range, so activations that overflow FP16 fit in it. That made it the default for training (Kalamkar & others, 2019). FP8 comes in two variants (Micikevicius & others, 2022): E5M2 favours range, for gradients, and E4M3 precision, for weights and activations. FP4 has only 15 distinct values (Open Compute Project, 2023).

The format explorer shows the bits each format stores for a number you type.

At 70000, FP16 and E5M2 overflow to infinity, while E4M3 and FP4 clamp to their largest value. At 0.00001, E4M3 and FP4 round to zero, and E5M2 keeps only a subnormal, 53% off.

Integers are the other option. INT8 stores whole numbers from −128 to 127, so on its own it cannot hold a weight like 0.013. Quantization pairs it with a scale, the real value of one step: with a scale of 0.001, the integer 13 stands for 0.013. Unlike a float, whose gaps grow with the magnitude (roughly a logarithmic grid), an integer grid has the same gap everywhere, so small values suffer the largest relative error (uniform and non-uniform quantization).

Quantization and dequantization

Quantization maps a real number $x$ onto an integer grid with three parameters: the bit width $b$, the scale $s$ and the zero point $z$.

Divide by the scale, round to a whole step, shift by the zero point and clamp to the grid. Dequantization goes back:

Only the integers $x_q$ and the pair $(s, z)$ are stored, and kernels compute $\hat{x}$ on the fly. The difference $\epsilon = |x - \hat{x}|$ is the quantization error.

The mapping figure quantizes eight values: real values on the top axis, integer codes on the bottom one.

Each extra bit halves the step, and with it the worst-case error. At 2 bits most values collapse onto code 0, and at 8 bits the largest error is below 0.06.

Bit width, zero point and scale

The bit width $b$ sets the number of levels: $[0,\ 2^b - 1]$ for unsigned integers, $[-2^{b-1},\ 2^{b-1} - 1]$ for signed ones. Symmetric schemes often drop the most negative code, $[-2^{b-1} + 1,\ 2^{b-1} - 1]$, so that the grid mirrors around zero. Values outside the grid are clipped to the nearest end.

The zero point $z$ slides the grid along the real line, and makes a real 0 map exactly onto an integer. That matters because zeros are everywhere (padding, ReLU outputs), and an error on each would bias every sum they enter (Jacob et al., 2018).

The scale $s$ is the size of one step. To cover a range $[\alpha, \beta]$ with $b$ bits,

A wider range, or fewer bits, means bigger steps and bigger rounding errors.

How to choose α and β

For values $\mathbf{x} = \{x_1, \ldots, x_N\}$, three choices are common (Nagel et al., 2021):

• Min-max sets $\alpha = \min(\mathbf{x})$ and $\beta = \max(\mathbf{x})$. Nothing is clipped, but one outlier coarsens the steps for every other value.

• MSE (mean squared error) picks the range with the lowest reconstruction error:

  $$\alpha^*, \beta^* = \arg\min_{\alpha, \beta} \frac{1}{N} \sum_{i=1}^{N} \left(x_i - \hat{x}_i(\alpha, \beta)\right)^2$$

• Percentile sets the range from a quantile, such as the 99.9th, and clips the rest. It is more robust than min-max and cheaper than MSE.

The clipping-rounding trade-off

The range trades one error for another:

1. Clipping error hits values outside the range. A larger range clips less.
2. Rounding error hits values inside it, at most half a step, $s/2$. A smaller range rounds less.

A value is either clipped or rounded, so the two squared errors add up:

The clipping figure plots both on 4,000 synthetic values: a standard normal bulk, plus 0.5% outliers six times wider.

At 4 bits, the lowest error comes from clipping at about 40% of the largest value, which cuts the error of min-max by more than three times. At 8 bits the steps are already fine, and the optimum moves back to about 97%.

Microscaling formats

At 4 bits, one scale per tensor is not enough: values that differ by orders of magnitude cannot share 15 levels. Microscaling (MX) formats (Rouhani & others, 2023) give each small block its own scale. The low-bit elements carry the local detail, and the shared scale carries the magnitude, for a few extra bits per block.

MXFP4

MXFP4, defined by the Open Compute Project (Open Compute Project, 2023), shares one scale across blocks of $k = 32$ elements:

Each $x_q$ is an FP4 (E2M1) value, and $S$ an 8-bit power of two (E8M0), so applying the scale is just an exponent add. The catch is that a power of two can miss the ideal scale by up to a factor of two, and one outlier sets the step for its 31 neighbours.

NVFP4: more local, more precise scales

NVFP4 keeps the FP4 elements and changes the scales (NVIDIA, 2025):

1. Smaller blocks of $k = 16$, so an outlier affects fewer neighbours, for twice the scale overhead.
2. A finer scale, an FP8 E4M3 number instead of a power of two, applied by multiplication.
3. Two levels: an FP32 scale per tensor brings the block scales into FP8’s range.

The microscaling figure stores 64 synthetic values in four groups of 16 with very different magnitudes, and one outlier in the first group.

In MXFP4, the small second group shares a block with the large first one, and most of its values round to zero. NVFP4 gives it its own scale and cuts the error by more than an order of magnitude. Its error also barely moves as the outlier grows, since the outlier sets the scale of its own block of 16 and nothing else. The table compares the two formats.

A taxonomy of quantization methods

Every quantization method answers the same six questions. Where does the grid sit? Are its levels evenly spaced? How many values share a scale? Is the scale fixed in advance? Which tensors get quantized? And does the model train through the rounding?

Symmetric and asymmetric quantization

Symmetric quantization centres the grid on zero: $z = 0$, the grid runs from $-q_{\max}$ to $q_{\max}$, and the largest magnitude sets the step, $s = \max_i |x_i| / q_{\max}$. It is cheap, but it wastes levels on lopsided data. ReLU outputs are never negative, so half the grid goes unused.

Asymmetric quantization fits the grid to the actual range $[\alpha, \beta]$, with the zero point

which puts the top code on $\beta$ and makes every code available to the data.

The mapping figure returns with non-negative values, as a ReLU produces.

In symmetric mode, the negative codes go to waste. Asymmetric mode halves the step and more than halves the largest error.

Symmetric quantization survives because zero points cost extra work when the matrix multiply runs on integers, as in W8A8 (8-bit weights and activations).

Uniform and non-uniform quantization

Uniform quantization spaces the levels evenly, so the absolute error is at most half a step everywhere. For small values that is a large relative error: anything below half a step rounds to zero.

Non-uniform quantization uses steps that grow with the magnitude, which keeps the relative error more even. Floating-point grids are non-uniform by construction: every power of two holds the same number of values, so their spacing is roughly logarithmic. So is NF4, the 4-bit format of QLoRA (Dettmers et al., 2023), whose levels follow the quantiles of a normal distribution.

The level-spacing figure compares three 4-bit grids, each with eight non-negative levels up to 6.

Below about 0.4, the uniform grid returns zero, a 100% error. The power-of-two grid keeps the relative error at or below a third for all but the smallest values, and FP4 sits in between.

Uniform grids still dominate for weights, because they are cheap to decode and every kernel supports them. Non-uniform grids need a lookup table, unless the hardware supports them natively, as Hopper GPUs do for FP8 (NVIDIA, 2023) and Blackwell GPUs for FP4 (NVIDIA, 2025).

Quantization granularity

The scale and zero point can be shared by many values or by few:

• Per-tensor: one $(s, z)$ for the whole tensor. Two extra numbers, but one outlier sets the step for everything.
• Per-channel: one $(s_c, z_c)$ per output channel, that is, per row of a weight matrix $\mathbf{W} \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}}}$.
• Per-group: each row is split into groups of $g$ consecutive values, for $C_{\text{out}} \times \lceil C_{\text{in}} / g \rceil$ pairs.

Finer granularity is cheap. With FP16 scales and groups of 128, a common choice for 4-bit weights (Lin et al., 2024), the scales add $16 / 128 = 0.125$ bits per weight, plus the zero points for asymmetric groups.

The granularity figure quantizes a synthetic 8 × 32 weight matrix with two outliers. Outlines mark which weights share a scale, and the lower heatmap shows the error.

Per channel cuts the per-tensor error by almost four, and groups of 8 by another four, because each step confines the outliers to fewer neighbours.

Activations are harder. Their outliers sit in a few input channels (Dettmers et al., 2022), and when the multiply runs in low precision, those channels cannot get their own scales. A scale per output channel multiplies a finished output, so it can be applied after the integer multiply. A scale per input channel would change inside the sum, so it cannot be factored out. Activations therefore get one scale per tensor or per token, which the outliers stretch for everyone. SmoothQuant (Xiao et al., 2023) works around this.

Static and dynamic quantization

Weight scales are computed once, offline. Activation scales can be:

• Static: fixed before deployment by running a small calibration set of representative inputs through the model. Free at run time, but inputs outside the calibrated range get clipped.
• Dynamic: computed at run time for each batch or token. Always fitted to the data, at the cost of a reduction before every quantized multiply.

What can be quantized?

Three kinds of tensors can be stored in low precision. W4A16 means 4-bit weights and 16-bit activations, and W8A8 means 8 bits for both.

• Weights are the easiest and most common target: fixed and smoothly distributed. They are converted back to 16 bits right before each multiply, so weight-only quantization saves memory and bandwidth without changing the arithmetic.
• Activations, quantized with the weights, let the whole multiply run on low-precision tensor cores. They are harder, because a few channels carry outliers far larger than the rest, which become systematic beyond a few billion parameters (Dettmers et al., 2022).
• The KV cache grows with context and batch (KV cache requirements). Quantizing it fits longer sequences and larger batches on the same GPUs. Its keys and values have outliers too (Quantizing the KV cache).

The dataflow diagram follows one decoding step under each scheme.

In W4A16, a dequantization step feeds the weights to a 16-bit multiply. In W8A8, the multiply itself runs in 8 bits and its result is rescaled. With an 8-bit KV cache, the cache is converted on every write and read.

PTQ, QAT and QAD

The last question is how much the model takes part in its own quantization: not at all, by training, or by training with a teacher.

Post-training quantization (PTQ)

PTQ quantizes a finished model: compute the ranges (min-max, MSE or percentile), usually on a calibration set, and round.

• Pros: no retraining, works on any checkpoint, takes minutes to hours.
• Cons: accuracy drops quickly at 4 bits and below, especially for activations, and depends on how representative the calibration set is.

Quantization-aware training (QAT)

QAT simulates quantization during training or fine-tuning, so that the model learns to live with it. Each quantized tensor goes through a fake quantization step in the forward pass:

The loss sees the rounded weights $\hat{W}$, while the optimizer updates the full-precision weights $W$. Round has zero gradient almost everywhere, so the straight-through estimator (STE) (Bengio et al., 2013) treats it as the identity inside the clipping range:

• Pros: much better accuracy than PTQ at 2–4 bits, because the weights adapt to the grid.
• Cons: needs a training loop with its data and compute. The STE biases the gradients, and results are sensitive to hyperparameters.

Quantization-aware distillation (QAD)

QAD adds knowledge distillation (Hinton et al., 2015) to QAT: a full-precision teacher guides the quantized student.

$\mathcal{L}_{\text{KD}}$ is usually the KL divergence between the two output distributions, and $\lambda$ sets the balance. In self-distillation, as in LLM-QAT (Liu & others, 2023), the teacher is the full-precision model itself. Its full distribution over the next token carries more signal than a single correct label.

• Pros: a richer training signal, which matters most at aggressive bit widths.
• Cons: the teacher adds its memory and a forward pass to every training step, and $\lambda$ needs tuning.

The workflow figure puts the three side by side.

QAT and QAD add a training loop to PTQ’s straight line. The loop buys accuracy, but costs data, compute and tuning, so most deployments start with PTQ.

Advanced post-training quantization

At 8 bits, rounding to nearest with a good range is usually enough. At 4 bits, or with activation outliers, it is not. Each method below fixes a different weakness of plain rounding:

• GPTQ (Frantar et al., 2023) corrects rounding errors by adjusting the weights it has not quantized yet.
• SmoothQuant (Xiao et al., 2023) moves activation outliers into the weights.
• AWQ (Lin et al., 2024) protects the few weights those outliers make important.
• Rotations spread outliers over all channels, taking weights, activations and the KV cache down to 4 bits.
• SINQ (Müller et al., 2025) gets much of AWQ’s benefit without calibration data.
• KIVI (Liu et al., 2024) and TurboQuant (Zandieh et al., 2025) target the attention cache.

GPTQ

GPTQ (Frantar et al., 2023) quantizes one linear layer at a time, and judges the result by the layer’s outputs, not its weights. Given weights $\mathbf{W} \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}}}$ and $N$ calibration inputs stacked as the columns of $\mathbf{X} \in \mathbb{R}^{C_{\text{in}} \times N}$, it solves

Inputs are correlated, so the rounding error on one weight can be cancelled by nudging the weights that multiply correlated inputs. GPTQ quantizes the matrix one column at a time, and after each column compensates by updating the columns not yet quantized.

The objective is quadratic, with Hessian $\mathbf{H} = 2\mathbf{X}\mathbf{X}^\top$. Let column $j$ be next, and $F$ the columns not yet quantized, $j$ included. Rounding column $j$ leaves an error $\mathbf{w}_j - \hat{\mathbf{w}}_j$, one value per row, and the best correction to $F$ is

where $\mathbf{H}_F^{-1}$ is the inverse Hessian restricted to $F$. Each row spreads its error over its remaining weights, in proportions set by how the inputs correlate. This is the update of Optimal Brain Surgeon (Hassibi & Stork, 1993), adapted to quantization by Optimal Brain Compression (Frantar & Alistarh, 2022).

The GPTQ figure runs the algorithm on a small layer with correlated inputs, one column per step.

The bottom heatmap shows how far GPTQ has moved each weight: the columns not yet quantized keep changing as errors are pushed onto them. After all twelve columns, GPTQ’s output error is less than half of round-to-nearest’s.

Every row is quantized in the same column order, so all rows share one inverse Hessian. Updates are applied in blocks of columns to keep the GPU busy, and a Cholesky decomposition keeps the inverse Hessian numerically stable. Together, these let GPTQ quantize 175B-parameter models to 3 or 4 bits in about four GPU hours (Frantar et al., 2023).

H⁻¹ = (2XXᵀ + λI)⁻¹
U   = upper Cholesky factor of H⁻¹        (H⁻¹ = UᵀU)
for each block of k columns:
    for each column j in the block:
        q_j = quant(w_j)                   quantize the column
        e_j = (w_j − q_j) / U[j, j]        scaled quantization error
        W[:, j+1 : block end] −= e_j · U[j, j+1 : block end]
    W[:, after the block] −= E · U[block, after the block]

SmoothQuant

GPTQ leaves activations alone, and for W8A8 they are the hard half. Their outliers sit in a few input channels that cannot get their own scales (granularity), so a per-tensor scale wide enough for the outliers leaves the other channels a few of the 256 levels.

SmoothQuant (Xiao et al., 2023) divides each activation channel by a smoothing factor $s_j$ and multiplies the matching weight row by the same factor, which leaves the product unchanged:

The factor for channel $j$ is

The migration strength $\alpha$ sets how much difficulty moves. At $\alpha = 0.5$, each activation channel and its weight row end up with the same maximum: the activations become smooth, and the weights get slightly wider, which they tolerate. The paper uses 0.5 by default and larger values for models with stronger outliers (Xiao et al., 2023). The factors are folded into the previous layer, so smoothing is free at run time.

The SmoothQuant figure applies this to synthetic activations with two outlier channels.

At $\alpha = 0.5$, each channel’s largest activation and largest weight match, and the output error is about seven times lower than without smoothing. Toward 0 the outliers stay in the activations, toward 1 they move into the weights.

Activation-aware weight quantization (AWQ)

The few channels with large activations also make some weights matter more: every error on them gets multiplied by a large input. AWQ (Lin et al., 2024) calls these salient weights, only 0.1–1% of the total.

Keeping them in FP16 would break the regular layout fast kernels need. Instead, AWQ scales each input channel by $s$ before quantization and divides the input by the same $s$:

A group’s step is set by its largest weight, so scaling up a channel that does not become the largest leaves the step, and the absolute rounding error, unchanged. The weight grows $s$ times, so its relative error $\Delta w / w$ shrinks by $s$. Scale too far, and the group’s range widens for every other channel.

AWQ sets $\mathbf{s} = \mathbf{s}_{\mathbf{x}}^{\alpha}$, where $\mathbf{s}_{\mathbf{x}}$ is the average activation magnitude per channel, and grid-searches $\alpha \in [0, 1]$ for the lowest output error on calibration data. Searching for scales, rather than reconstructing outputs as GPTQ does, also overfits the calibration set less (Lin et al., 2024).

In the paper’s ablation, OPT-6.7B (Zhang & others, 2022) with 3-bit weights in groups of 128 reaches a WikiText-2 (Merity et al., 2016) perplexity (lower is better) of 23.54 with round-to-nearest, against 10.86 in FP16. Multiplying the 1% salient channels by 2 brings it to 11.92, close to the 11.39 of keeping them in FP16 (Lin et al., 2024).

The AWQ figure starts at $\alpha = 0$, plain round-to-nearest.

The best $\alpha$ cuts the output error to about a third. Weight error on the salient channels drops from 24% to 10%, while the others rise from 22% to 28%, a good trade because the salient inputs are 12 times larger. Keeping the salient channels in FP16 (dashed) is lower still, but here 2 of 32 channels (6%) are salient, far more than in a real model.

Rotations: spreading outliers across channels

SmoothQuant and AWQ rescale single channels. Rotations mix all channels, so that an outlier’s energy is shared by every channel instead of sitting in one. Any orthogonal matrix $\mathbf{Q}$ ($\mathbf{Q}\mathbf{Q}^\top = \mathbf{I}$) can be inserted into a matrix multiply for free:

A rotation preserves each vector’s length but spreads it over the coordinates: two huge coordinates become many moderate ones, which a uniform grid handles well. QuIP (Chee et al., 2023) introduced the idea for weights as incoherence processing.

The usual choice is a Hadamard matrix. Its entries are all $\pm 1/\sqrt{d}$, so it mixes every input into every output equally, and like an FFT it runs in $O(d \log d)$. Flipping the signs of its rows at random gives a randomized Hadamard transform, which spreads outliers wherever they sit.

The rotation figure rotates activations with two outlier channels, then quantizes activations and weights to 4 bits.

After rotation, every channel has a similar range, and the 4-bit output error falls to less than a third. As the outliers grow, the unrotated error grows two to three times, while the rotated error stays flat.

QuaRot (Ashkboos et al., 2024) rotates a whole transformer. It relies on the computational invariance shown by SliceGPT (Ashkboos, Croci, et al., 2024): once the RMSNorm (B. Zhang & Sennrich, 2019) scales are folded into the next weights, the residual stream can be rotated and every weight that reads it counter-rotated, with no change to the output. Most rotations are folded into the weights offline. A few run online: one before the feed-forward down-projection, and others in attention, including one on queries and keys so that the cached keys are rotated too. Every matrix multiply then runs in 4 bits, KV cache included, and 4-bit Llama 2 70B (Touvron & others, 2023) loses at most 0.47 WikiText-2 perplexity (Ashkboos, Mohtashami, et al., 2024).

Random rotations are not equally good. SpinQuant (Liu et al., 2025) finds they can differ by up to 13 points of zero-shot accuracy, and learns them instead, closing up to 45% of QuaRot’s gap to full precision on Llama 3 8B. FlatQuant (Sun et al., 2025) replaces rotations with a learned affine transform per layer, kept cheap by a Kronecker factorization, and reports less than 1% accuracy loss for W4A4 on Llama 3 70B.

SINQ: activation awareness without calibration

AWQ needs calibration data to find the channels with large activations. SINQ (Müller et al., 2025) finds a proxy in the weights alone: in trained models, the weights that read a large input tend to have a small spread. In the $C_{\text{out}} \times C_{\text{in}}$ layout, each input channel is a column of $\mathbf{W}$, and across the models SINQ examines, the inverse of a column’s standard deviation predicts its average input magnitude (Müller et al., 2025).

Scaling columns alone would create new outliers within rows, so SINQ uses two scale vectors, one per row ($\vec{s}$) and one per column ($\vec{t}$):

where $\mathbf{W}_q$ is the quantized matrix, with its usual scale and zero point per row. An outlier $W_{ij}$ can be absorbed by raising $s_i$ or $t_j$. To find the scales, SINQ iterates in the style of Sinkhorn-Knopp matrix balancing (Sinkhorn & Knopp, 1967), dividing each row and column by its standard deviation, and keeps the scales with the lowest imbalance:

the ratio of the largest to the smallest row or column standard deviation. The balanced matrix is then rounded to nearest. At run time, $\vec{t}$ multiplies the layer’s input like AWQ’s scale, so the overhead is one element-wise product.

The SINQ figure runs the iteration on a synthetic matrix whose column spreads follow the paper’s observation.

Unscaled, the row and column spreads differ by 12×. Within a few iterations they flatten toward 1, the output error falls well below round-to-nearest’s, and the column scales track the input magnitudes without ever seeing an input.

The synthetic matrix makes the gain larger than on a real model. On the Qwen3 family (Yang & others, 2025) and others, at 3 and 4 bits, SINQ more than halves the perplexity gap to 16 bits left by other uncalibrated uniform quantizers (Müller et al., 2025). It processes each layer on its own, so it works on any architecture, and combines with calibrated methods such as AWQ.

Quantizing the KV cache

The KV cache is not just another activation. It is written once per token and read at every later step, its keys and values have different statistics, and at long contexts it can outweigh the model (KV cache requirements).

Keys per channel, values per token (KIVI)

KIVI (Liu et al., 2024) measures that keys have a few fixed channels with very large magnitudes in every token, while values have no outlier pattern. So it groups them differently:

• Keys per channel, so a large channel no longer coarsens the others.
• Values per token, since the attention output mixes value rows, and per-row quantization keeps each token’s error from leaking into the others.

Per-channel groups need several tokens, while generation adds one at a time, so KIVI keeps the most recent tokens in full precision until a group is complete.

The KIVI figure quantizes the keys of one synthetic attention head both ways.

Per token, each scale stretches to the three large channels and every key is rounded coarsely. Per channel, the ordinary channels get scales fitted to their own range, and the attention-score error drops 6–9× at every bit width.

With this layout, KIVI stores the cache in 2 bits with almost no quality loss on Llama, Falcon (Almazrouei & others, 2023) and Mistral (Jiang & others, 2023) models. Peak memory, weights included, falls by 2.6×, allowing up to 4× larger batches and 2.35–3.47× higher throughput (Liu et al., 2024).

Rotation plus an unbiased residual (TurboQuant)

TurboQuant (Zandieh et al., 2025) rotates each vector instead, in three steps:

1. Rotate. Store the vector’s length, and multiply the unit vector by a random rotation. Every coordinate then follows the same known distribution (close to a normal with variance $1/d$), whatever the input.
2. Round to a fixed codebook. Since the distribution is known, its best codebook is computed once with the Lloyd-Max algorithm (Lloyd, 1982; Max, 1960). No calibration, no scale per group.
3. Correct the bias. A quantizer that minimizes squared error shrinks inner products toward zero, and attention scores are inner products. TurboQuant spends one bit on the rounding residual $\mathbf{r}$: it stores the signs of $\mathbf{S}\mathbf{r}$ for a random Gaussian matrix $\mathbf{S}$, plus the residual’s length. This 1-bit quantized Johnson-Lindenstrauss (QJL) transform (Zandieh, Daliri, & Han, 2025) makes every inner-product estimate unbiased.

The TurboQuant figure shows the first two steps on one key, then repeats the whole procedure over 150 random rotations to show the third.

After rotation, every coordinate falls in the same narrow band, so one codebook fits them all. The MSE-optimal quantizer underestimates the inner product consistently, while TurboQuant is right on average, with a wider spread. Random errors partly cancel across the many keys a query attends to, but a bias shifts every score the same way and flattens the attention weights.

On real models, TurboQuant keeps full quality with 3.5 bits per channel and loses little at 2.5 (Zandieh, Daliri, Hadian, et al., 2025). The fractional budgets come from giving outlier channels more bits. The paper also proves its distortion is within a factor of about 2.7 of the best any quantizer can achieve.

How low can the cache go?

A 4-bit cache preserves accuracy, but a 2-bit one often degrades it, especially on long-context reasoning (Xia et al., 2025). Kitty ranks key channels by sensitivity and keeps a small fraction at 4 bits, cutting KV memory by nearly 8× with negligible accuracy loss on Qwen3 and Llama 3 (Xia et al., 2025).

Comparing the methods

These methods combine more often than they compete. GPTQ (Frantar et al., 2023) or AWQ (Lin et al., 2024) quantizes the weights, rotations or SmoothQuant (Xiao et al., 2023) make activations quantizable, and KIVI (Liu et al., 2024) or TurboQuant (Zandieh, Daliri, Hadian, et al., 2025) handles the cache. The table lists what each one needs.

Putting it together

Quantization works because decoding is limited by memory: fewer bytes per weight mean fewer GPUs, more room for the KV cache and faster decoding. The price is rounding error, and outliers make it expensive. Clipping gives them up, small blocks fence them in, SmoothQuant and AWQ rescale around them, rotations spread them out, and KIVI and Kitty give them their own scales or extra bits. GPTQ corrects whatever error remains.

The recipe below follows from these mechanisms, not from benchmarks, so check it against your own evaluation.

1. Find your bottleneck. If the model does not fit, or single-stream generation is too slow, quantize the weights. If large batches or long prompts saturate the GPU’s arithmetic, quantize the activations too. If long contexts or large batches run out of memory, quantize the KV cache.
2. Start with post-training quantization, calibrated on data that looks like your traffic.
   • Weights at 4 bits: GPTQ (Frantar et al., 2023) or AWQ (Lin et al., 2024) with groups of about 128. Without calibration data, SINQ (Müller et al., 2025).
   • Weights and activations at 8 bits: FP8 where the hardware supports it, SmoothQuant (Xiao et al., 2023) for INT8 or for models with strong activation outliers.
   • Activations at 4 bits: rotate first, as QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2025) do.
   • KV cache: 8 bits is safe and 4 bits preserves accuracy. 2 bits takes per-channel keys, as in KIVI (Liu et al., 2024), and extra bits for the most sensitive key channels, as in Kitty (Xia et al., 2025), to protect long-context reasoning.
3. Evaluate on your task, not only on perplexity. Quantization error can be small on average and still concentrate on long contexts or multi-step reasoning.
4. Escalate only when needed. If accuracy drops too far, try smaller groups or a microscaling format first, then quantization-aware training or distillation.

The surveys by Gholami et al. (Gholami et al., 2021) and Liu et al. (K. Liu et al., 2025) cover more methods, including the extreme low-bit ones left out here. The white paper by Nagel et al. (Nagel et al., 2021) goes deeper into the fundamentals, and the ApX course on practical LLM quantization (ApX Machine Learning, 2026) shows how to apply them.