How a neural model translates: where its vectors come from, how the encoder reads a Galician sentence into one summary, and how the decoder writes English from it. The running example is the start of a sentence from the novel Speechwriter: Unha tía miña, que todo o solucionaba con ditos e refráns, dicía algo así como «un é escravo do que di e dono do que cala».
What you will learn on this page
A neural translation model has two parts. The encoder reads the Galician words into a summary, a vector; the decoder starts from that summary and writes the English words one at a time. Every word in this model is a 2-D vector in one of three tables, and all of them start random: before it can translate, the model has to learn them from examples, by training. Here the 16 pairs are taken one at a time, round after round, and after every pair the tables are updated a little. Section 2 shows the encoder's rule with simple made-up numbers; section 3 takes the trained encoder and decoder apart.
The training data: 16 pairs, each an input (a Galician phrase) → its gold output (the right output: its English translation, by a person)
Encoding: the inputs are mixed, one by one, into one vector, the summary (new = ½ × old + ½ × vector of the input; section 2 shows the rule)
green: the probability of this pair's gold translation · red: of the translation the model writes · orange line: where the cycle is now. Green climbs because every update of this pair moves the tables toward its gold translation; the other 15 pairs' updates share words with it and mostly help too.
B. The whole training: the same procedure as part A, seen a round at a time (1 round = one pass over the 16 pairs); and, for comparison, the other procedure
the training data: an input (a Galician phrase) → its gold output (the right output: its English translation, by a person) · click a cell to follow that pair in part A, with the tables as they are after this round
one output at a time, the one with the highest probability, mixed into the state before the next · the gold output · a different one · click one to follow it in part A, translating with these tables
training changes only the numbers in the three tables; how they are used (summary, score, probability, what is mixed in) never changes
Inputs translated right, over the rounds
count:C. Training data and development data: the same training again, with one pair held out
Same random start, same rule, same 3000 rounds as part B; only unha tía miña → an aunt of mine is left out. The 15 pairs trained on are the training data. The held-out pair is development data: the model never sees it in training, and we use it to check the model while we build it. A failure here is not the summary losing words: in 3 words, tía is still a quarter of it. It is a model that learned the pairs it was shown, not how to combine familiar words anew. The other weakness, words fading from the summary of a long sentence, is section 2's; the next page takes it up.
each gold output filled by the probability the model gives it · grayed, with a tag: the development pair, held out of training
one output at a time, the most probable, mixed into the state before the next · the gold output · a different one · section 3 walks the development pair output by output, with either model
unha tía miña (development): gold vs written translation, over the rounds
D. Training longer, checked on the development pair
Part C stopped at 3000 rounds. Here the same training on the same 15 pairs goes on to 6000 rounds; unha tía miña is still never trained on. The rest of the page keeps the models of 3000 rounds.
unha tía miña (development): gold vs written translation, over 6000 rounds
green: the probability of the gold translation · red: of the translation the model writes · orange line: this round · one curve where the model writes the gold translation
What the marks show
2-D vectors; stochastic gradient descent, one pair per update, learning rate 0.2, 3000 rounds over the 16 pairs (part B can also train all 16 at once: the 16 gradients added up, learning rate 0.1, 3000 rounds, replayed by the page as well); torch.manual_seed(0); mt_s2s_train.py records the random start, and the page replays the training with the same rule, so that every single update can be shown.
From here on, inference: a trained model translating. No gold output, no update; the tables stay as they are (section 1's "when translating" switch showed this inside training).
A translation model starts by reading the Galician sentence. Each word has an embedding, a row in a table like p on the skip-gram page; here it has 2 dimensions, so every word is a point. The encoder reads the words in order and keeps a summary: one more vector, updated after every word. In this simplified encoder, each new word moves the summary halfway toward that word's point.
The update, with this step's numbers
How much of each word is in the summary now
The word vectors are made up for this example. Real encoders, recurrent neural networks, learn their update rule instead of always taking half and half, and they can learn to keep some things longer. But their summary is still one vector of a fixed size, whatever the length of the sentence.
Still inference: a model from section 1 translates, word by word, as in section 1 part A's “when translating”. Choose the model and the phrase: a training pair, the development pair, or a test phrase neither model ever saw.
The model of 16 writes all its training phrases right; it fails on phrases it never saw (the test phrases; for the model of 15, also the development pair) and on long sentences, where early words fade from the summary (section 2; the next page shows this with other models, and the remedy, attention).
a. One plane for both languages: how the next word is chosen
Every word's score, on one line
b. How the state was recomputed
What is in the state now
In long sentences the first words fade far more (section 2: tía is 0.534 of the summary). Sutskever and colleagues (2014) fed sentences in reverse to keep the first words fresh; the next page shows the remedy that made the trick unnecessary.
d. Is “an aunt of mine” the best translation?
The toy model learned to write “an aunt of mine” for unha tía miña, and the real model Helsinki-NLP/opus-mt-gl-en writes the same (a slide in the lesson looks at why). It is correct and faithful: “an” keeps the new character that unha introduces. But English rarely says it. For every 100 “my …”, English books write “a … of mine” this often (log scale):
“A … of mine” picks one out of several, which is natural for friends and marked for close family. A translator would rebuild the sentence: “I had an aunt who…”. No word of the Galician corresponds to “I had”, and our model never saw such a sentence in training, so it cannot write it. Section 4 trains a model that does. Counts from Google Books; the n-gram page, section 7, has the full chart and how it was counted.
Same kind of model as sections 1–3, different training data. Here the English follows how often English books use each construction: “a friend of mine” stays, but for aunts and brothers the English is rebuilt the way a translator would, “I had an aunt who sang”. The decoder writes I, had and who, words with no Galician counterpart, because its training data has them.
A. The data: 8 patterns, each with three verbs (cantaba sang, bailaba danced, ría laughed): 24 pairs; part C holds one out as development data
kept as English books have it
rebuilt, as a translator would
B. Training on all 24 pairs
Both runs start from the same random vectors and train the same way: all their training pairs at once, with a refinement of gradient descent that adapts the size of each step (Adam), 2000 steps, every 25th recorded. Section 1 showed how one pair is decoded and how an update moves the vectors; here only the outcome.
one output at a time, the most probable · the gold output · a different one · click a sentence: its words on the plane
C. Training on 23 pairs: unha tía miña cantaba held out as development data
The same training without one pair. The held-out sentence is development data: the model never sees it, and we check what it writes for it. Its bailaba and ría versions, and un irmán meu cantaba (I had a brother who sang), are still trained on.
one output at a time, the most probable · the gold output · a different one · click a sentence: its words on the plane