Skip-gram, step by step

The model only ever tries to predict a word's neighbors. Watch the embedding table change while it does: words with the same neighbors end up pointing the same way.

Inside skip-gram: what gets multiplied

p: the embedding table, one row per input word (the dots in the plot). p′: the second table, one column per word as a neighbor to predict (the rings).

Each word's vector, drawn as a point. The axes zoom out as the values grow.

a word's row in the embedding table × where that point started
cats, dogs chase, eat mice, balls fish, meat

Corpus, window = 1

pass 0
starting points:

Model's prediction

What to watch

  • cats and dogs both predict chase and eat. They get the same kind of nudges and end up pointing the same way.
  • mice and balls both predict only chase. fish and meat both predict only eat.
  • Press new random start a few times and jump to 3000. The starting points change every time; the grouping by color comes back every time.
  • Press Step with show the second table on: the green arrow adds a little of the →chase vector to cats, the red arrow subtracts a little of the average guess. Their sum, the black arrow, is the step.
  • With paths on, jump to 100: almost all of the movement happens in the first 20 passes. Cats and dogs end up on the same line from the origin.
  • The model never sees the colors.

Which words are similar?

values:−+ table: one row per word, one column per dimension. Rows in order: animals, chase/eat, things chased, food.
cats and dogs, 2 dimensions
cats and fish, 2 dimensions
cats and dogs, one-hot: (1, 0, 0, …) and (0, 1, 0, …)

Cosine compares two vectors by their direction only: 1 = same direction, 0 = at a right angle, −1 = opposite. How long the vectors are does not count, so the pictures draw them at the same length.

Words with the same neighbors get the same nudges, so they end up pointing the same way: high cosine. That is how we read similarity off the table.

One-hot vectors never have a 1 in the same place, so any two of them are at a right angle. Cosine 0 for every pair: one-hot vectors cannot say that cats are more like dogs than like fish.

cos(cats, dogs) after each pass

one-hot vectors

the table

trained, as many dimensions as words

the table

trained, 2 dimensions

the table