Start
Microcredencial Algoritmos verdes para la IA: diseño e implementación
Green NLP: Retrieval
Natural language processing · Topic 3 · Instructor: Olga Zamaraeva
Content warning
- The translation part uses sentences from the novel Speechwriter
- Two of them contain strong swearing, as in the original
- We keep them: translating swearing with the same strength is hard for translation systems
Where we left off
Two topics, two changes to the same pipeline
baseline-fp16
reads all documents
quantized
fewer bits per number
retrieval
reads only the relevant documents
| The model gets | The task is called |
|---|---|
| the question and all the documents, with every answer in them | reading comprehension |
| the question only | closed-book question answering |
| the question and the documents a retriever found for it | open-book question answering |
- Closed book, in our Topic 1 test run: 2 of 8 right; with the documents, 8 of 8
- Its wrong answers sounded just as sure: it called RAG (retrieval-augmented generation, next screen) “Retrieval-Augmented Learning”
- A small model alone is not a reliable source of facts. Documents supply the facts; retrieval chooses which.
Step 1 · Retrieve, then read
Retrieval-augmented generation (RAG)
searches the documents
question + top documents
model
- First, find the documents most related to the question; then put them into the prompt, next to the question.
- The model answers from the prompt and from what it learned in training
- The facts only need to be in a document the retriever can find: a small model can answer about facts it never saw in training
Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” NeurIPS 2020
Where the documents come from
The system builders choose the collection; the model never sees the search
1. Once, before any question
documents
chunks
each chunk
an index
2. For every question
question
the index
into the prompt
the answer
- The collection, the knowledge base: help-desk articles, contracts, regulations, …
The same idea in a chat assistant
The question: “How much energy does one Gemini prompt use?”
- The model only writes text; a search request is text in a special format
- It was trained on examples where a question about a specific or recent fact is followed by a search request
- An uploaded file works the same way: cut into chunks, embedded, searched
- Real systems add: approximate search over millions of chunks, word and embedding search together, a second model that reorders the top results
Step 2 · Dense retrieval
How a retriever finds documents
The cosine similarity of Topic 1, for whole texts
- Turn every document into a vector, an embedding. Once, in advance.
- Turn the question into a vector, with the same embedding model (ours: MiniLM, from Notebook 01c, trained so that texts with the same meaning get close vectors)
- Compare the question's vector with each document's: cosine similarity
- Keep the k most similar documents. Only those go into the prompt.
cos(q, d) = q · d|q| × |d|
- q · d: multiply matching values, add them up
- |q|: the length of the vector, √(sum of squares)
- 1: the same direction; 0: a right angle; below 0: pointing away
Ranking documents by cosine similarity
Notebook 03b's toy vectors (a text's vector: the average of its known words'); the question “what does it cost in electricity to get an answer”, whose only known word is electricity
Move the question
Step 3 · Sparse retrieval
The older method: count shared words, weighted by rarity
TF-IDF; an improved version, BM25, is still used in search engines
- A word in every document does not help to choose one; a word in only one document points straight at it
- tf, term frequency: how many times the word is in this document
- idf, inverse document frequency: how rare the word is in all the documents
idf(word) = ln number of documentsdocuments that contain the word
tf-idf(word, doc) = tf(word, doc) × idf(word)
A document's score for a question: the sum of tf-idf over the question's words.
TF-IDF, worked out
The five toy documents of Notebook 03b
Why the logarithm?
An imagined news archive with N = 1,000,000 articles
| The word is in | N ÷ that number | ln of it |
|---|---|---|
| 1 article | 1,000,000 | 13.8 |
| 1,000 articles | 1,000 | 6.9 |
| every article | 1 | 0 |
- Without the logarithm, the rarest word counts 1,000 times more than a word in 1,000 articles: one rare word, even a typo, would decide the ranking
- With it: only twice as much
- With 5 documents the ratio is at most 5: the logarithm matters in large collections
Sparse and dense vectors
The picture of Topic 1, now for whole documents
Sparse: one value per word of the vocabulary (its tf-idf in the document), mostly 0
- Finds exact word matches only
- Needs only counting
Dense: a few hundred learned values, all filled in
- Matches different words with the same meaning
- Needs an embedding model to run
The two questions of the notebook
| question | words shared with doc_energy | finds doc_energy? |
|---|---|---|
| A: how much energy does running a model use | energy, running, a, model | sparse ✓ · dense ✓ |
| B: what does it cost in electricity to get an answer | it | sparse: a tie with two others · dense ✓ |
Shared words, weighted
Build both retrievers
Replace each ___ with code. Do the exercises in order: later ones use your earlier code.
Exercise 1 · Rank by shared words
Complete word_overlap_score: the number of distinct words that the question and the document share, ignoring capital letters. Question A should rank doc_energy first.
Hints
set(text.lower().split())is the set of a text's words.a & bis the intersection of two sets;lencounts it.
Exercise 2 · TF-IDF
Complete idf and tf, then tfidf_score: the sum of tf × idf over the distinct words of the question that are in the collection. The cell prints the board's numbers, 0.51 and 1.61, then ranks the documents for both questions.
Hints
math.logis the natural logarithm.- idf divides the number of documents by the number that contain the word.
list.count(word)counts how often the word is in a list.- A word in no document would divide by 0: skip it.
Exercise 3 · Dense retrieval with toy vectors
Complete cosine_sim and text_vector (the average of the vectors of the known words), then rank the documents for question B. doc_energy should come first, with 0.984.
Hints
np.dot(a, b)is the dot product;np.linalg.norm(a)the length.np.mean(list_of_vectors, axis=0)averages vectors value by value.
Both retrievers on the course quiz
Step 4 · What retrieval saves
Fewer tokens to read
The 26 documents, cut into tokens the way Qwen2.5 cuts them, without the question; top 2 by the dense retriever: the average over the 26 questions (TF-IDF's: 311)
- Attention compares every token with every token: twice the tokens, four times the pairs (Topic 1)
- The rest of the model grows in step with the tokens
- Writing the answer: one run of the model per written token, as many either way; only the attention to the input gets cheaper
- So expect an energy saving smaller than the token saving: the notebook measures it
Long documents must be split
The embedding model reads at most 128 tokens
- The rest of a document is cut off before its vector is computed: 23 of our 26 documents are longer than 128 tokens
- Real systems split long documents into chunks, embed each chunk, and retrieve chunks; the chunk size is a design decision
Reading everything, or retrieving?
Li et al. (2024): GPT-4o, GPT-3.5-Turbo and Gemini-1.5-Pro on long-document question answering
| long context | RAG | |
|---|---|---|
| the prompt holds | the whole source text: on average 7,000 words (LongBench) or 100,000 tokens (∞Bench) | the top 5 chunks of 300 words: 5 × 300 = 1,500 words |
| average quality | higher, for all three models | lower |
| the same answer | for 63% of the questions | |
Self-Route: RAG first; the model says if the chunks are enough; if not, long context
| model | cost (tokens the model reads) compared with long context |
|---|---|
| Gemini-1.5-Pro | −61% |
| GPT-4o | −39% |
with quality close to long context
Li, Li, Zhang, Mei & Bendersky, “Retrieval augmented generation or long-context LLMs? A comprehensive study and hybrid approach,” EMNLP 2024 Industry Track
Step 5 · What can go wrong
The retriever can miss
Then the answer is not in the prompt at all
- The baseline read every document: the right one was always in the prompt
- Now a wrong answer has two possible causes: the retriever, or the model
- Retrieval accuracy: how often the right document is in the top k
- Answer accuracy: how often the final answer is right
The quiz, top 2, computed with the notebook's models
| retriever | right document found |
|---|---|
| dense (MiniLM) | 23 of 26 |
| sparse (TF-IDF) | 26 of 26 |
| dense miss | why |
|---|---|
| What does the acronym PUE stand for? | an acronym: three letters to the embedding model |
| the training-emissions question | the answer is past the 128-token window |
| the RAG question | two other retrieval documents rank higher (the RAG paper, kNN-MT) |
Two levers, measured separately
| Topic 2: quantization | Topic 3: retrieval | |
|---|---|---|
| changes | bits per number | tokens to read |
| keeps | the text | the precision |
| new risk | rounding error | the retriever misses |
- Independent: a system can use both. Notebook 03c runs them side by side on one GPU.
Discussion · 10 minutes
- When would you give the model everything, on purpose?
- When would you trust the retriever's top document enough to show the model nothing else?
- Our 26 documents are short and each is about one topic. What goes wrong first with a large, messy collection?
Step 6 · Retrieval for translation
Retrieval works for translation too
kNN-MT: a lookup at every word
model
translations
- At each word it writes, it compares its decoder state with the states stored from earlier translations
- It mixes the words written after the k closest states into its own probabilities
Khandelwal, Fan, Jurafsky, Zettlemoyer & Lewis, “Nearest neighbor machine translation,” ICLR 2021
Ours: a small translation memory, one lookup
| Speechwriter sentence in the memory | cosine |
|---|---|
| O meu curmán Fernando aguantaba o trebón… | 0.40 |
| Xuro que fun capaz de ler o pensamento… | 0.33 |
| Estou rodeado de merda, carallo! … | 0.32 |
| Pero que dúas horas nin que merda! … | 0.29 |
| Á noitiña, xa máis calmo, pagou os cubatas. | 0.25 |
| Un paso máis atrás, o alcalde suaba puro medo. | 0.18 |
Similarity to our passage, with MiniLM. The top sentence and its English translation go into the prompt as an example.
Your turn
The same parts in our notebook
| In a real system | In Notebook 03a |
|---|---|
| knowledge base | CORPUS: 26 documents, Topic 1's eight plus 18 new |
| embedding model | paraphrase-multilingual-MiniLM-L12-v2 |
| index | doc_embeddings, computed once |
| search | retrieve(): cosine similarity, keep the top 2 |
| augmented prompt | build_prompt(): the documents above the question |
| language model | Qwen2.5-1.5B-Instruct, 16 bits, as in Topic 1 |
The quiz, three times: 26 questions each
no documents
all 26 documents
the top 2
- The prompt says: answer only from the documents. An instruction, which the model may or may not follow.
- Earlier rows had 8 questions, so the notebook compares cost per question
Notebook 03a: Retrieval measurement
- Part 1: the quiz three ways, measured with CodeCarbon; answer accuracy and retrieval accuracy; three rows for your leaderboard
- Part 2: dense and sparse retrieval on the same quiz, and the questions where they disagree
- Part 3: our Galician passage translated with and without a retrieved example, graded with four metrics
Notebook 03a, in Colab, with a GPU
- In Colab: Runtime → Change runtime type → T4 GPU
- Upload your three CSV files as saved at the end of Topic 2 (leaderboard.csv, mt_leaderboard.csv, mt_eval_leaderboard.csv) before you start
- Download them again at the end
Notebook 03c: three levers, one run
| configuration | model | the model reads |
|---|---|---|
| baseline, closed book | 16 bits | the question only |
| quantized, closed book | GPTQ, 4 bits | the question only |
| retrieval | 16 bits | the question + the top 2 |
The same 26 questions, in one Colab session, on one GPU; the translation too. Unlike the baseline row of Notebook 01a, this baseline reads no documents.
- What does quantization do to what the model knows?
- What does retrieval add to it, and at what cost?
The course, in one screen
Three topics, one pipeline, one leaderboard
language models
measure the baseline
quantization
fewer bits per number
retrieval
fewer tokens to read
- Measure first: energy, carbon, time and memory, always next to quality
- Change one thing at a time, on the same task, with the same check
- Savings and losses depend on the model, the task, the data and the hardware: published numbers are a starting point, your own are the answer
