Microcredencial Algoritmos verdes para la IA: diseño e implementación

Green NLP: Quantization

Natural language processing · Topic 2 · Instructor: Olga Zamaraeva

Before the activity

The task: answering questions about documents

documents
+ question

→

language
model

→

answer

→

right or wrong?

Input: a few short documents about green AI, and a question. The model must answer from the documents, in one short sentence.
Model: Qwen2.5-1.5B-Instruct, a small general-purpose language model
Score: an answer is right if it contains the reference answer as whole words, ignoring capital letters. Accuracy = right answers ÷ all questions.
Two runs: with the documents, and without them. What do the documents add to accuracy, and to energy?

question	What does the acronym PUE stand for?
reference	power usage effectiveness

The check is simple and cheap. It also misses right answers that use other words.

Activity · 60–75 minutes, individual

Notebook 01a: Baseline measurement

Answer questions about a small set of documents, measured with CodeCarbon: energy, CO₂eq and quality, as rows of your leaderboard
The same quiz with no documents: what does the model know on its own?

Notebook 01a, in Colab, with a GPU

In Colab: Runtime → Change runtime type → T4 GPU
Download leaderboard.csv at the end: the next notebooks add rows to it

Open in Colab: beginner Open in Colab: advanced

The plan

The same tasks, the same model, one change

Baseline
baseline-fp16
16 bits per number

→

Quantized
gptq-int4 or bnb-int8-fallback
4 or 8 bits per number

The same documents, the same questions, the same scoring rule
The same model, Qwen2.5-1.5B-Instruct
One change: how many bits store each of the model's numbers. This is quantization.

Each run is one row of your leaderboard

quality	accuracy: right answers ÷ questions
energy	kWh, measured by CodeCarbon
carbon	g CO₂eq, from that energy
time	seconds for the whole quiz
memory	peak GPU memory, GB (in the quantized rows only)

A model is a collection of numbers

A computer stores every number as bits

Binary: the same place values as decimal, with powers of 2

A bit is 0 or 1
Each position is worth twice the one to its right: 1, 2, 4, 8, …
A model's parameters are numbers: billions of them, all stored as bits

Integers

Can integers store the weights?

What larger weights would do to attention

Real attention of Qwen2.5-1.5B, one head of the first layer, in “The capital of Galicia is Santiago de Compostela.”

A floating-point number has three parts

The IEEE 754 standard: sign, exponent, mantissa

value = sign × (1 + mantissa) × 2^{exponent − bias}

FP16: a 16-bit floating-point number. FP32 uses 32 bits.
The next slides show where the 1 and the bias in this formula come from

Moving the point

The place values continue past the point: each position is worth half the one to its left. The same happens in decimal: 13 = 1.3 × 10¹

Storing 13: the leading 1

Storing 13: the bias

Two exponent values are kept for special cases

One real weight in FP16

Floating point: fine steps for small numbers

How fine are the steps in FP16?

Formats side by side

A scale of sizes: each gridline is × 10¹⁰. 1 is in the middle, and 0 is not on it. Each float format stores the same sizes with a minus sign too
Mantissa bits buy precision, how close two numbers can be: every doubling holds 8,388,608 values in FP32, 1,024 in FP16, 128 in BF16

Interactive

Flip the bits

Open the full page

What quantization is

Memory for the weights of our model

Qwen2.5-1.5B-Instruct: 1.5 billion parameters, from here on also called its weights (not the attention weights of Topic 1)

6 GBFP32
32 bits

3 GBFP16
16 bits

1.5 GBINT8
8 bits

0.75 GBINT4
4 bits

The weights only. Running the model also needs memory for its intermediate values.

Try it · about 10 minutes

Which models fit on a 16 GB GPU?

The Python here runs in your browser. The first Run loads Python, which takes a few seconds.The same exercise as a notebook: beginner · advanced (Colab)

Replace each ___ with code. Run shows what your code prints. Run the tests also checks it and says what to fix.

Exercise 1 · Weight memory

Complete weight_memory_gb: memory in gigabytes = parameters × bits ÷ 8, divided by 10⁹ bytes per GB. Then the loop checks 7B, 32B and 70B models in FP16, INT8 and INT4 against the 16 GB of a Colab T4. Ignore everything except the weights.

Hints

params_billion * 1e9 is the number of parameters.
Bits ÷ 8 gives bytes; bytes ÷ 1e9 gives gigabytes.
A model fits if its weights take less than the GPU's memory.

When quantization decides whether a model fits

Weight memory, against the 16 GB of a Colab T4

7B: all three fit. FP16, 14 GB, leaves almost no room for anything else.
32B: only INT4 comes close, and it takes the whole GPU: in practice it does not fit.
70B: none fits. Even INT4 needs more than twice the GPU.
Past some size, quantization is the only way to run a model on given hardware; past a larger size, it is no longer enough.

What you gain and what you risk

From weights to integer codes

Each weight is replaced by the code of its strip. Stored: the small integers, plus one number to get back. This is why it is lossy: inside a strip, weights can no longer be told apart

Absmax quantization

Six real weights and INT4: the codes run to 7. With INT8 they run to 127, and the steps are 18 times finer. Zero stays exactly zero

Interactive

Fewer bits, fewer codes

Open the full page

Try it · about 15 minutes for both exercises

Absmax from scratch

The Python here runs in your browser, with NumPy. The first Run loads Python, which takes a few seconds.The same exercises as a notebook: beginner · advanced (Colab)

Replace each ___ with code. Do the exercises in order: the second one uses your first function.

Exercise 2 · Quantize and dequantize

Complete absmax_quantize and absmax_dequantize. qmax is the largest code of a signed bits-bit integer: 127 for 8 bits. alpha is the largest absolute value. The scale maps alpha to qmax. The cell should print the codes 64, 36, −89, 0, −22, −54, 127, 36, −23.

Hints

A signed bits-bit integer goes up to 2^{bits − 1} − 1.
np.abs(x) gives absolute values; np.max the largest one.
scale = qmax / alpha. Quantize: multiply by the scale, then round. Dequantize: divide by it.

Exercise 3 · Sweep the bit widths

Quantize the same values with 8, 4, 3 and 2 bits. For each, count the distinct codes the values use, and compute the mean and the largest absolute error.

Hints

set(q.tolist()) keeps each code once; len counts them.
The error of each value is values - dq; take np.abs of it first.

Lopsided values: asymmetric quantization

An illustration, with values from −1 to 3

Both rows use INT4 codes. Weights center on 0, and absmax suits them; one-sided values do not, like the outputs of ReLU, a common step inside networks that replaces every negative value with 0

One large weight stretches the range for all

Absmax on the real model with one scale for all its weights, in INT4

Interactive

Clip the range

Open the full page

Clipping: a narrower range

Real weights, INT4, one scale for all the weights. Error: the root mean squared difference between each weight and its stored value

Choosing the range is called calibration

Clipping trades one large error for many small ones
Calibration: choosing the range, often on a sample of real inputs

Calibration rule	The range keeps…
percentile	all values except the rarest extremes, e.g. the top 0.1%
mean squared error	the one that makes the average squared error smallest
KL divergence	the one that keeps the shape of the values' histogram (how many values fall in each small interval) closest

input

×

weights
fixed after training

→

activations
new for every input

Post-training quantization	scale and zero point	trade-off
dynamic	computed during each run	adapts to every input; extra work
static	fixed in advance, from calibration data	faster; needs representative data

Try it · about 20 minutes for both exercises

The zero point and clipping

These exercises use your absmax_quantize and absmax_dequantize from Exercise 2. Run that cell first.

Exercise 4 · Asymmetric quantization

Complete asymmetric_quantize and asymmetric_dequantize. The codes run from 0 to 2^bits − 1. The scale is the width of the range divided by the number of steps: the step between two codes, so values are divided by it (absmax's scale was the inverse, codes per unit). The zero point is the code that stands for 0.0. The cell compares both methods on one-sided values, at 4 bits.

Hints

The width of the range is xmax - xmin.
Quantize: divide by the scale, then add the zero point.
Dequantize: subtract the zero point, then multiply by the scale.

Exercise 5 · Clip before quantizing

Complete quantize_with_clipping: limit every value to [−clip, clip], then quantize with absmax. The loop compares no clipping with three clipping ranges on the 24 ordinary values and the outlier.

Hints

np.clip(x, low, high) replaces values below low with low and above high with high.

GPTQ: round one weight, then correct the others

One output of a learned table: the dot product of two weights with the input x = (2, 1); weights rounded to steps of 0.25

The goal: keep the output (the dot product) close to the original
GPTQ rounds the weights one at a time; after each, it adjusts the weights not yet rounded to make up for the error
The adjustment is computed on a small set of real inputs, the calibration data
4 bits per weight with little loss, without retraining

Frantar et al., “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” ICLR 2023

Interactive

Watch GPTQ correct the next weight

Open the full page

GGUF: one scale per block of weights

Tick marks: the codes. One large weight spoils the range only in its own block. The file format of llama.cpp, for running models on laptops and CPUs; type Q4_0 is shown

Interactive

One scale, or one per block

Open the full page

Quantization-aware training

Train with the rounding in place: the model learns to live with it

full-precision
weight

→

quantize

→

dequantize

→

forward pass
and loss

→

update the
full-precision weight

↺

Post-training quantization (PTQ)

Train in full precision, quantize at the end
Cheap: minutes to hours
The GPTQ checkpoint used here is PTQ

Quantization-aware training (QAT)

The forward pass already sees the rounding error
Usually more accurate at low precision
Expensive: it needs training

BitNet: weights of −1, 0 or +1

A dot product of inputs and weights, with ordinary weights and with ternary weights (three possible values); inputs (2, 5, 3)

× +1: keep the input; × −1: flip its sign; × 0: skip it. No multiplication left, only additions.
Trained this way from the start, as in QAT
One weight is coarse; thousands of them together still give a fine range of sums

Ma et al., “The era of 1-bit LLMs: all large language models are in 1.58 bits,” 2024

Which method, when

Method	When it is used	Cost
Absmax, zero point (PTQ)	8 bits; the basic building block of every method	almost none
Dynamic PTQ	activations whose range changes a lot from input to input	extra work in every run
Static PTQ	fast inference, when representative calibration data exists	a calibration run
GPTQ	4 bits on a GPU, without retraining	calibration, layer by layer
GGUF block formats	running models on laptops and CPUs	one scale per block
QAT	the best accuracy at low precision, when training is affordable	training
BitNet	a model designed for ternary weights	training from scratch

What measurements show

Two recent studies, on very different hardware

Study and setting	Finding
Shi & Ding (2025): 11 quantization methods, models of 7B to 70B parameters, A100 and H100 GPUs	energy efficiency up to 30% better (34B models)
	waiting time (latency) of the slowest responses up to 70% shorter
	code generation (HumanEval): share of generated programs that pass their tests, down by up to 92% for the 13B model; smaller models lose more quality
Husom et al. (2025): 28 quantized models, a Raspberry Pi 4	3- and 4-bit models used up to 79% less energy than 16-bit ones
Husom et al. (2025): 28 quantized models, a Raspberry Pi 4	for Qwen2.5 1.5B, our model: every 3-bit version used more energy per token than the 4-bit ones

Shi & Ding, “Systematic characterization of LLM quantization: a performance, energy, and quality perspective,” arXiv:2508.16712; Husom et al., “Sustainable LLM inference for edge AI,” ACM Transactions on Internet of Things 6(4), 2025, doi:10.1145/3767742

The hardware has to support the format

Peak throughput of the NVIDIA T4, the GPU in Colab, from its datasheet

FP328.1 TFLOPS

FP1665 TFLOPS

INT8130 TOPS

INT4260 TOPS

TFLOPS, TOPS: trillions of floating-point or integer operations per second

The T4 has special circuits, tensor cores, for 16-, 8- and 4-bit numbers
These are peak numbers: real runs are slower, and memory is often the limit
A format the hardware or the software cannot use directly is unpacked first: smaller, but not faster

Discussion · 10–15 minutes

Is quantization a free lunch?

When would quantization not be worth it? Think of:

hardware or software that cannot compute in the lower precision
too few bits for the task: code, arithmetic, rare words
a small model, already close to its quality limit
calibration data unlike the real inputs
a quality check too weak to notice what broke

Discuss with your neighbor, then with the room.

What we run

The same model family, two ways to load it with fewer bits

config in the leaderboard	what is loaded
baseline-fp16	Notebook 01a: Qwen2.5-1.5B-Instruct, 16 bits
gptq-int4	first try: Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int4, quantized by the Qwen team with GPTQ
bnb-int8-fallback	if GPTQ does not install: the 16-bit model, quantized to 8 bits as it loads, with the bitsandbytes library

The same layers, the same tokens (pieces of words), the same training: only the stored numbers change
The software that computes with them changes too: the configuration is recorded with every row
gptq-int4 and bnb-int8-fallback are two different configurations: compare like with like

Activity · 90–125 minutes, individual

Notebook 02a: Quantized measurement

Upload your three CSV files: leaderboard.csv (Notebook 01a), mt_leaderboard.csv and mt_eval_leaderboard.csv (Notebook 01e); without them, the notebook starts new ones
Load the quantized model: GPTQ first, then the fallback
Run the same eight questions with CodeCarbon, and score them the same way
Append a row to leaderboard.csv and compare it with your baseline
Translate the Galician passage again; grade it with BLEU, BERTScore and two LLM judges

Notebook 02a, in Colab, with a GPU

In Colab: Runtime → Change runtime type → T4 GPU
After the install cell: Runtime → Restart session, once
Download the three CSV files again before you close Colab

Open in Colab: beginner Open in Colab: advanced

Activity · about 40 minutes, if time allows

Notebook 02c: Where does quantization pay off?

Part A: peak GPU memory, quantized and 16-bit, for one question
Part B: memory as the batch grows from 1 to 128 questions at once
Part C: a 7B model on the same GPU, in 16 bits and in 4 bits: the chart from “When quantization decides whether a model fits”, measured

Notebook 02c, in Colab, with a GPU, after 02a

Open in Colab: beginner Open in Colab: advanced

Share-out: what changed?

Compare your two rows

Which configuration loaded: gptq-int4 or bnb-int8-fallback?
Accuracy: the same as the baseline?
Energy and CO₂eq: lower, the same, higher?
Time? Peak memory (quantized rows only)?
The translation: did it change in a way a reader would notice?

Then compare with your neighbor

Same configuration, same numbers?
Run the quiz twice: how much do your own numbers vary between runs?

Topics 2 and 3

Topic 2

A number is stored as bits; fewer bits, less memory
Quantization: scale, round, store the codes; the error is the price
The range decides the error: outliers, clipping, the zero point, blocks
Savings and losses depend on the model, the task and the hardware: measure them

Topic 3: retrieval

Topic 2
baseline

→

Topic 2
fewer bits

→

Topic 3
less text

The same 16-bit model, given only the relevant documents

Thank you for your attention

catedrainditexalgoritmosverdes.com

Start

Green NLP: Quantization

Notebook 01a, in Colab, with a GPU

Step 1 · Weights are numbers

Step 2 · Quantization and memory

Exercise 1 · Weight memory

Step 3 · Absmax quantization

Exercise 2 · Quantize and dequantize

Exercise 3 · Sweep the bit widths

Step 4 · Choosing the range

Exercise 4 · Asymmetric quantization

Exercise 5 · Clip before quantizing

Step 5 · Methods used in practice

Step 6 · Does it pay off?

Your turn

Notebook 02a, in Colab, with a GPU

Notebook 02c, in Colab, with a GPU, after 02a

What comes next

Thank you for your attention