What Does word2vec Really Learn? Researchers Finally Have an Exact Learning Theory
Berkeley Artificial Intelligence Research (BAIR)
Researchers from BAIR (Berkeley AI) have developed the first quantitative and predictive theory for word2vec's learning process. They prove that under realistic conditions, training reduces to unweighted least-squares matrix factorization, with final embeddings given by PCA of a specific matrix derived from corpus statistics.
In a new paper, researchers from BAIR provide the first closed-form theory for word2vec's learning dynamics. They show that when embeddings are initialized near zero, the model learns concepts one at a time, each corresponding to an orthogonal linear subspace, and the effective rank of the embedding matrix increments stepwise. The learned features are the top eigenvectors of a matrix M* defined by co-occurrence probabilities. The theory matches numerical experiments well: word2vec achieves 68% on analogy benchmarks, the approximate model gets 66%, versus 51% for the classical PPMI method. The work also explains the emergence of linear representations for abstract concepts like gender or tense.
- Сокращения
- PCA = Principal Component Analysis — Анализ главных компонент
- PPMI = Positive Pointwise Mutual Information — Положительная попарная взаимная информация
- BAIR = Berkeley Artificial Intelligence Research — Исследования в области искусственного интеллекта в Беркли
Source: BAIR (Berkeley AI) —
original