A language model predicts the next word as a probability for every word in the vocabulary. An n-gram model gets those probabilities by counting sequences of words in a corpus and dividing. We start with Shakespeare: first single words counted play by play, then word sequences across all his works. Then we take the counting apart on three tiny sentences.
Count how often each word occurs in each play. Read down a column and a play becomes a list of numbers: a vector. Read across a row and a word becomes a vector too. Similar plays use words in similar proportions, so their vectors point in similar directions.
The drawing shows two dimensions at a time. Similarity below uses all of them.
Similarity of every pair (click a cell)
The arithmetic for the selected pair
Table on the slide: Jurafsky & Martin, Speech and Language Processing, lecture slides on vector semantics. Counted columns: Project Gutenberg eBook #100, so a few numbers differ from the slide's edition.
Four models counted from the complete works of Shakespeare. Each one writes a sentence by drawing the next word from the counts, given the last n−1 words. The longer the context, the more it sounds like Shakespeare, and the more it simply quotes him.
Reading the complete works…
Click a word to see the model's choices at that step
How sparse are the counts?
There are far more possible word sequences than Shakespeare could ever write, so a long context has usually been seen once, with a single continuation. The 4-gram model then has no choice but to copy.
Example after Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Ch. 3, Fig. 3.4. Generating text to see what a model has learned goes back to Shannon (1948) and Miller & Selfridge (1950). Text: Project Gutenberg eBook #100, lower-cased, punctuation counted as words, speaker names and stage directions removed.
One line is a real sentence from the plays and poems, the other was written by the model. Pick the real one.
The same model, small enough to see every count: three sentences about an aunt and an uncle. Edit the corpus here and every section below updates.
Corpus
One sentence per line. Try adding my uncle spoke kindly and watch P(·|my) change.
Count(w1 w2): rows are the first word, columns the next word
The last column is Count(w1): how often the row word is followed by any word. Click a row to divide.
The formula, with this row's numbers
Counting, then dividing. That is the whole model. Every word in the vocabulary gets a probability; most of them get 0 because the pair never occurred.
Smoothing: add k to every count
Total probability is always 1. Smoothing takes some of it from the pairs the corpus saw and spreads it over the pairs it never saw. Part of it goes to <UNK>, the slot for every word that is not in the vocabulary: that share is P(<UNK> | word).
Start from a word, look up its row, pick a next word, repeat. Greedy takes the most frequent next word; sampling draws one with the row's probabilities. Both use the smoothing set in section 4.
The probability of a sentence is the product of one probability per pair of neighboring words. If the corpus never saw one of the pairs, that factor is 0, and so is the whole product. Generation stopped at “her” for the same reason: a count of 0. A word outside the vocabulary is read as <UNK>; try her aunt stayed quiet.
Ask about anything on this page: what a panel shows, why a vector moved, what a number means. Claude sees the page's current state.
Answers use your own Claude account.