Quantization, Step by Step

Quantization, Step by Step

A model is billions of numbers. Quantization stores each one with fewer bits. These six stations follow one number into its bits, then a few numbers into integer codes and back, and show what gets lost on the way, and one way to lose less.

What you will learn on this page

  1. A number in 16 bits is three fields read separately: a sign, an exponent (the scale) and a mantissa (the detail). FP16 and BF16 split the bits differently, and the space to the next number that can be stored grows with the number. → section 1
  2. Absmax: the largest absolute value sets one scale; every value is multiplied by it and rounded to an integer code. With fewer bits there are fewer codes, and different values start to share one. → section 2
  3. One outlier stretches the range, so the ordinary values share a few codes. Clipping caps the range: one large error for the outlier, smaller errors for all the others. → section 3
  4. When all values are 0 or more, symmetric codes for negative values go unused. Asymmetric quantization maps the smallest value to code 0, and the zero point is the code that stands for 0.0. → section 4
  5. GPTQ rounds the weights one at a time and moves each rounding error into the weights not yet rounded (here, simply the next one), so the output stays closer than when each weight is rounded on its own. → section 5
  6. A scale for each block of weights keeps the small weights from rounding to 0; each extra scale costs 16 bits, as in GGUF files. → section 6

1One number in 16 bits

Click a bit to flip it, or type a number. The three colored fields are read separately, then multiplied.

Examples:

Sign

Exponent: the scale

Mantissa: the detail

Play walks the exponent: 1.5, then × 2 and ÷ 2, one exponent step at a time. The line below is a scale of sizes: the mark 2^k stands for 2 to the power k, and one power to the right is twice as large. Watch the space to the next number.

2Absmax: from floats to integer codes

Nine values, stored as signed integers. The largest absolute value, α = 10.8, is mapped to the largest code. Every value is multiplied by the same scale s and rounded. Fewer bits: fewer codes, and values start to share one.

Bits:
a value a code shared by two values: they can no longer be told apart

Click a row to see its arithmetic

3One outlier stretches the range

Twenty-four ordinary values between −2 and 2, and one value of 256. The picture zooms in on the ordinary values. The vertical ticks are the codes: with the outlier, absmax spreads the codes so far apart that only three of them fall here. Clipping caps the range, so the codes move closer together.

Bits:
Clip at:

Clipping trades one error for many: the outlier is stored as the clipping value, and every ordinary value gets closer to a code. Which trade is right depends on how much the outlier matters.

4Lopsided values: the zero point

These nine values are all 0 or more, like the outputs of ReLU, a step inside networks that replaces every negative value with 0. Symmetric absmax keeps codes for negative values that never occur. Asymmetric quantization maps the smallest value to code 0 and the largest to the last code; the zero point z is the code that stands for 0.0.

Move all values by: · 4 bits

Careful with s: here it is the step between two neighboring codes (a value per code), so a value is divided by it. In section 2, s was codes per unit (127 ÷ 10.8), so a value was multiplied by it. The same name, inverse numbers.

5GPTQ: round one weight, then correct the next

One output of a learned table: six weights (the model's learned values, its parameters), one input of six numbers x₁ … x₆, and one output, their dot product y = w₁x₁ + … + w₆x₆. The weights can only be stored in steps of 0.25 (the gray lines). Rounding each weight on its own lets the errors add up. GPTQ rounds the weights one at a time, and moves each rounding error into the next weight, before that one is rounded.

the original weight the weight now, after corrections rounded and stored ▲ the next weight to be rounded

Real GPTQ computes the corrections for all the weights not yet rounded, from a small set of real inputs (calibration data), so that the output stays close on average over those inputs. Here there is one input, and the whole error goes into the next weight.

6One scale per block

Sixteen weights of very different sizes, stored with 4 bits. With one scale for all of them, the large weights set the range and the small ones round to 0. Give each block of weights its own scale, and the small ones keep their values. Each scale costs 16 more bits.

Weights per block:
the weight what is stored, after rounding a weight rounded to 0 Bar heights: square root of the size, so the small weights stay visible.

Files in the GGUF format (llama.cpp) use this idea. Type Q4_0, for example, stores blocks of 32 weights at 4 bits, plus one 16-bit scale per block: 4 + 16 ÷ 32 = 4.5 bits per weight.