UUtah CS 6340 NLP | Fall 2026 | Word embeddings (Part 1)
Overview
Word2Vec’s Skip-gram with Negative Sampling (SGNS) learns dense word vectors from corpus co-occurrence, using the distributional hypothesis that words appearing in similar contexts tend to have related meanings. The lecture builds the method from positive and sampled negative word pairs through its sigmoid-dot-product objective and gradient descent, then explains embedding lookup, practical tools, and the limitation that one static vector cannot represent a word’s different senses.
Key takeaways
- Word2Vec’s Skip-gram with Negative Sampling learns embeddings by making observed target-context pairs score highly and sampled negative pairs score poorly under a sigmoid of their vector dot product.
- The distributional hypothesis lets rare or unfamiliar words acquire useful representations when they appear in contexts resembling those of better-observed words.
- A 300-dimensional embedding is dense and much smaller than a vocabulary-sized one-hot or bag-of-words representation, while still encoding learned semantic relationships.
- Negative sampling makes training practical by drawing candidate negatives from the vocabulary rather than exhaustively verifying that each pair never co-occurred; occasional noisy samples are acceptable at scale.
- Static word embeddings assign one vector to every word form, so polysemous words such as bank conflate distinct senses; contextualized models produce representations that depend on the surrounding text.
Chapters
- The practicum is scheduled for the Wednesday after Labor Day; students should prepare with the practice assignment and ask questions as needed.
- A lemma is a word’s canonical dictionary form, while inflection creates forms such as a singular noun and its plural.
- Polysemous words such as basin have multiple senses, so NLP systems need word-sense disambiguation to identify the meaning used in context.
- Bag-of-words struggles with synonyms, antonyms, and semantic relatedness because its word-count vectors do not encode meaning or relationships.
- Vector semantics aims to place semantically similar or related words near one another in a learned vector space.
- Synonyms such as canyon and gorge may occur in similar contexts, while antonyms such as hot and cold should not be treated as equivalent.
- Related words need not be synonyms: surgeon, scalpel, nurse, and hospital belong to the same semantic field.
- These representations are foundational ideas for modern NLP, even though current LLM token embeddings are produced differently.
- The illustrated embeddings have 300 dimensions, far fewer than a vocabulary that may contain 30,000 to 200,000 entries.
- Unlike sparse bag-of-words vectors, the displayed embeddings are dense, with learned nonzero values across their dimensions.
- Words such as father and uncle, or girl, niece, and daughter, cluster according to relationships in the vector space.
- The visualization shows how similarity and relatedness can be represented through distances or closeness between vectors.
- The distributional hypothesis, associated with linguistics in the 1950s, says words appearing in similar contexts tend to have similar meanings.
- Context clues such as saline environments and Great Salt Lake shallows suggest that the unfamiliar word Artemia refers to a small organism.
- A rare word can gain a useful representation by appearing in contexts similar to more frequent, better-understood words.
- Word embeddings apply this context-based signal to learn representations from large text corpora.
- Feature engineering manually specifies useful input features, while representation learning lets a model learn vector values directly from text.
- Learned dimensions may not have an obvious human interpretation, motivating interpretability research.
- A corpus is a collection of written text; its word distributions are statistical patterns in how words occur.
- An embedding is a learned mapping from input tokens into a vector representation space.
- Word2Vec refers to methods introduced by Tomas Mikolov and collaborators, with the work published in 2013 and later cited more than 40,000 times.
- Skip-gram trains a binary classifier to decide whether a target word and a candidate context word are likely to occur near each other.
- The classifier relies on the two words’ embeddings, encouraging vectors to encode useful contextual relationships.
- After training, the embeddings are retained for downstream tasks and the auxiliary classification model is discarded.
- A sliding context window supplies positive examples; the apricot example uses two words on either side of the target.
- For a target-context pair, the dot product of their embeddings is passed through a sigmoid to produce a value between zero and one.
- A higher dot product yields a higher sigmoid output, making the pair more likely under the classifier.
- Training must adjust randomly initialized embeddings because their initial dot products do not reflect observed word co-occurrence.
- Skip-gram maintains two vectors per vocabulary word: one for when the word is a target and one for when it is a context word.
- The model scores a pair using the target vector and the corresponding context vector.
- The context window size is a hyperparameter; the lecture’s example uses two words before and after the target.
- After training, the target-word embeddings are typically kept as the word representations.
- Positive pairs come directly from words occurring together in a corpus window, while negative pairs are sampled from the vocabulary.
- Negative sampling avoids checking every candidate word against the entire corpus to establish that it never co-occurred.
- The sampling strategy gives relatively more attention to rare words, which are often absent from a particular target’s context.
- Sampled negatives can include accidental true co-occurrences, but large corpora and repeated training make this noise tolerable.
- The model seeks high predicted probability for observed target-context pairs and low probability for sampled negative pairs.
- The objective uses one positive context and typically two to five negative samples, with the negative count treated as a hyperparameter.
- The embeddings are the trainable weights in this model; with vocabulary size V and dimension 300, the two embedding tables contain 2 × V × 300 values.
- Gradient descent updates the vectors to minimize the training loss.
- The likelihood combines probabilities for positive and negative examples, using an independence assumption to express the joint probability as a product.
- Maximizing likelihood is converted into minimizing its negative, matching the usual gradient-descent setup.
- Taking logarithms turns products into sums and improves numerical stability.
- The sigmoid identities express positive-pair terms using the embedding dot product and negative-pair terms using its negation.
- Gradient descent updates each embedding by subtracting the loss gradient multiplied by a learning rate.
- The learning rate is a chosen training hyperparameter; gradients specify how each vector component affects the loss.
- Positive and negative context embeddings receive updates based on their different roles in the SGNS objective.
- Repeated updates produce the final vectors, after which the target embedding table can be used as the learned word representations.
- A word such as bank receives one static embedding even when it refers to a riverbank or a financial institution.
- Different senses can pull the same vector toward incompatible contexts, making one representation a poor fit for either meaning.
- Shared contexts can bring related words closer together, while negative examples also discourage some pairings.
- Contextualized representations in later NLP models address this limitation by producing representations that vary with context.
- FastText provides efficient embedding training and extends word-based methods to subword units.
- GloVe is another widely used approach for producing word embeddings from text.
- Gensim offers tools for training and using embeddings, including models trained on specialized domain data.
- Pretrained embeddings can also be downloaded and loaded into frameworks such as PyTorch instead of implementing training from scratch.
- An embedding matrix has shape vocabulary size × embedding dimension, with each row assigned to the token whose vocabulary ID matches that row index.
- A tokenized sequence becomes a sequence of vectors by looking up each token ID’s row in the embedding matrix.
- Embedding dimension is a hyperparameter; historical Word2Vec examples commonly used 50 or 300 dimensions, while small modern language models may use around 4,000.
- The next class will cover a small neural network for binary classification and embedding analysis; the practicum follows the Labor Day break.
Transcript excerpt
0:01 So yeah, practicum is happening next Wednesday in a week. Um, you have a practice assignment to prepare for it. I will share just one message kind of collecting all the thoughts I mentioned around uh practicum today or tomorrow. Um, yeah, there's nothing else for you to do than do the practice assignment. Ask us questions uh if there are any along the way. Uh, next Monday is a Labor Day, so we won't have a class. So we meet uh next Wednesday uh when the practicum uh is today we have a lot to do a little bit uh ambitious plan on my end so I will jump right into it unless there are some pressing questions. Okay let's let's get into it then. Um
0:51 remember last time we were chatting about how uh we have a piece of text. It can be for example a prompt we give to an LLM. That piece of text is then tokenized namely split into a list of tokens in the vocabulary. And then we talked about how now we need some kind of numerical representation that can be fed into a neural network. So neural network can process it. And these numerical representations ideally would already carry information about um you know useful information about meaning of these words and their relation to other words. So just to go over some of those linguistic aspects we were kind of mentioning in Russia at the at the end of the uh last lecture those are word senses. Um here is um just a screenshot
1:40 of the uh victionary item for the word basin. Um one term I want you to know is lema. Lemma refers to the canonical or dictionary form of a word. And we have many uh word forms. These are you know
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.