Skip-gram, Step by Step

Skip-gram, Step by Step

The model only ever tries to predict a word's neighbors. Watch the embedding table change while it does: words with the same neighbors end up pointing the same way.

What you will learn on this page

  1. Skip-gram learns one vector per word by predicting its neighbors: the input word's vector, times the second table, gives a score and then a probability for every word as a neighbor. → the multiplication
  2. Training takes one pair at a time (16 pairs = one round): the input's vector moves a little toward its gold neighbor's vector and away from the average guess. → the plane
  3. Words with the same neighbors get the same nudges, so they end up pointing the same way (cats and dogs), although the model never sees our colors. → the plane
  4. The cosine reads similarity off the direction. One-hot vectors are all at right angles, so they have no similarity; trained vectors do. → section 2
  5. With fewer dimensions, the similarity shows sooner: fewer random starting values to overwrite. → section 2
round 0

The training data: each sentence of the corpus gives 4 pairs, an input word and one neighbor to predict, the gold output (window = 1: the word just left or just right of the input)

wordinput wordneighbor to predict wordnot in this pair ✓trained on in this round just trained on next

Training goes through the 16 pairs in this order, again and again; once through all 16 is one round.

Inside skip-gram: what gets multiplied

p: the embedding table, one row per input word (the dots in the plot). p′: the second table, one column per word as a neighbor to predict (the rings).

×1

Each word's vector, drawn as a point. The axes zoom out as the values grow.

a word's row in the embedding table × where that point started →word: that word's vector in p′, the second table (the word as a neighbor to predict) line: the path the point took during training (jump ahead or play a few rounds to see it)
cats, dogs chase, eat mice, balls fish, meat
starting points:

Model's prediction

Try this

  • Turn on show the second table and press Step: the green arrow adds a little of the →chase vector to cats, the red arrow subtracts a little of the average guess, and their sum, the black arrow, is the step cats takes.
  • Jump to round 3000: cats and dogs point the same way (both predict chase and eat), and so do mice and balls (only chase) and fish and meat (only eat).
  • Press new random start a few times and jump to 3000 each time: the starting points change, the grouping comes back. The model never sees our colors.
  • Turn on paths and jump to 100: about two thirds of the movement up to round 100 happens in the first 20 rounds.

Which words are similar?

values:−+ Each table below holds every word's vector: one row per word, one column per dimension. Rows in order: animals, chase/eat, things chased, food (the colors at the left).
cats and dogs, 2 dimensions: the same neighbors, the same direction
cats and fish, 2 dimensions: different neighbors, a wider angle
cats and dogs, one-hot: (1, 0, 0, …) and (0, 1, 0, …): always a right angle, so no similarity

Cosine compares two vectors by their direction only: 1 same direction · 0 a right angle · −1 opposite. Length does not count, so the pictures draw both vectors at the same length.

cos(cats, dogs) after each round

one-hot vectors
trained, as many dimensions as words
trained, 2 dimensions