A retriever picks the documents the model will read. These four stations show how: by comparing vectors, by counting rare shared words, on the course's own 26 documents, and where it goes wrong.
What you will learn on this page
Dense retrieval: a text becomes a vector (here, the average of its words' vectors), and the documents are ranked by cosine similarity with the question, which depends only on the angle between the vectors (1: same direction, 0: a right angle). → section 1
The retriever keeps the top k documents; only they go into the prompt the model gets. → section 1
Sparse retrieval (TF-IDF) scores the words a document shares with the question, each weighted by its rarity: a word in every document scores 0. → section 2
On the course's 26 documents and 26 questions, both retrievers can be checked: is the document with the answer in the top k? Each extra document adds tokens for the model to read, and attention's pairs grow with the square of the tokens. → section 3
The dense retriever's model reads only the first 128 tokens of each document; it never sees the rest. Real systems therefore split long documents into shorter chunks. → section 4
1Dense retrieval: the angle between vectors
Each word we know has a 2-D vector; a text's vector is the average of its known words' vectors. The retriever ranks the documents by cosine similarity with the question: 1 means the same direction, 0 a right angle. Pick a question, type your own, or click on the plane.
Keep the topdocuments
a document (thick: in the top k)the question, and its angle with the selected document
Known words: energy (4, 2), electricity (4, 3), carbon (3, 4), water (1, 4), retriever (0, 5), memory (2, −3).
The prompt the model gets
2Sparse retrieval: shared words, weighted by rarity
Word overlap gives one point per shared word. TF-IDF gives each shared word tf × idf: how often it is in the document, times idf = ln(5 ÷ the number of documents that contain it). A word in every document scores 0; a word in one document scores ln 5 ≈ 1.61.
word shared with the questionword shared, and in only one document
Score of each document
Click a document to see its TF-IDF sum
3Both retrievers on the course quiz
The 26 documents and 26 questions of Notebook 03a. The bars are each document's similarity to the question. The green bar is the document that holds the answer; the dark bars are the top k, the only documents the model will read.
Keep the top2
Tokens: the question plus the documents, cut into tokens (pieces of words) the way Qwen2.5 cuts them; the prompt's instructions add a few more. Attention compares every token with every token, so the pairs grow with the square of the tokens.
The sparse scores are scikit-learn's TF-IDF, the retriever of Notebook 03a: a smoothed idf, English stop words left out, each document's vector scaled to length 1. With section 2's formula and the same stop words, the top document is the same for all 26 questions; lower ranks differ.
4The embedding model reads only 128 tokens
The dense retriever's model, MiniLM, reads at most 128 tokens of a text; the rest is cut off before the vector is computed. Each bar is one of the 26 documents. The hatched part is never seen by the retriever.
MiniLM reads every document from its start; where the window ends, it stops. The lengths are in MiniLM's own tokens: it cuts words into pieces differently from Qwen2.5, so they differ from section 3's counts.
Real systems split long documents into chunks shorter than the window, embed each chunk, and retrieve chunks.
Ask Claude
Ask about anything on this page: why a document ranks first, what idf does, why the dense retriever misses a question. Claude sees the page's current state.