UUtah CS 6340 NLP | Fall 2026 | Word embeddings (Part 2)
Overview
The lecture analyzes static word embeddings trained with skip-gram and negative sampling, comparing them with bag-of-words representations and explaining intrinsic and extrinsic evaluation, visualization, analogies, and encoded social biases. It then builds a binary sentiment classifier that averages token embeddings before an MLP, covering softmax predictions, gradient-based training, embedding fine-tuning, and classification metrics.
Key takeaways
- Averaging static word embeddings creates a compact sentence representation, but it removes word order and cannot distinguish sentences with identical tokens arranged to express different roles.
- Intrinsic embedding benchmarks test alignment with human word-similarity judgments, while extrinsic evaluation tests whether the embeddings improve a downstream task such as sentiment classification.
- t-SNE plots and analogy examples can build intuition about embedding structure, but neither is reliable proof: visualization can shift with hyperparameters, and analogy arithmetic fails on many relationships.
- Embeddings can encode gender and racial stereotypes from training data, causing allocation harm in systems such as CV screening and representational harm in generated text.
- A simple sentiment model can average token vectors, pass the result through ReLU- or GELU-based MLP layers, produce two logits, and choose the class with the highest softmax score.
- For imbalanced classification, accuracy may hide poor performance on a less common class; precision, recall, and macro F1 provide complementary views of performance.
Chapters
- Bag-of-words represents a sentence as a sparse vector with one dimension per vocabulary word, potentially 30,000 to 200,000 dimensions.
- Word embeddings map tokens to dense vectors, commonly 50 or 300 dimensions, which are more practical neural-network inputs.
- Unlike bag-of-words dimensions, embedding dimensions have no direct human-readable meaning because skip-gram learns them from context.
- Static embeddings assign one vector to each word, even when a word has multiple senses; contextualized embeddings later address this limitation.
- Averaging embeddings discards word order, so “Dave sat on Tim” and “Tim sat on Dave” can receive the same representation.
- Word embeddings capture semantic relationships more effectively than bag-of-words, but averaged representations still cannot reliably represent who did what to whom.
- Extrinsic evaluation compares downstream task performance, such as sentiment-classification accuracy, using pretrained, random, or alternative embeddings.
- Intrinsic evaluation compares embedding cosine similarities with human similarity judgments for word pairs, often using Spearman correlation.
- A strong intrinsic score does not guarantee downstream utility, so embeddings should be judged in the context of their intended applications.
- Visualizations project 50- or 300-dimensional word or phrase vectors into two dimensions for inspection.
- Nearby negative-sentiment terms such as “bad,” “worst,” and “dislike” can provide intuition about learned relationships.
- Methods such as t-SNE can produce very different plots after hyperparameter changes, so visualizations are not proof of a claim.
- Word-vector analogy tests use relationships such as king:man :: queen:woman to probe relational structure.
- A vector operation subtracts the representation for “man” from “king,” adds “woman,” and searches for the nearest vocabulary vector, which may be “queen.”
- Analogy arithmetic works for some examples but fails for many others; positive demonstrations should not be treated as universal evidence.
- Because embeddings learn associations from text, examples such as “doctor minus man plus woman” can return “nurse,” reflecting occupational stereotypes.
- Allocation harm can arise when biased embeddings influence consequential decisions, such as ranking CVs for programming jobs.
- Representational harm includes associating names commonly given to Black Americans with less pleasant words, or generating stereotyped stories about Muslim people.
- A model can amplify rather than merely reproduce a dataset’s bias; the lecture illustrates this with a hypothetical three-to-one imbalance in mentions of male versus female doctors.
- Repeated generations could make women doctors far less frequent than the original data imbalance suggests, embedding the disparity into model outputs.
- Debiasing is difficult because distributional associations can persist even when a system avoids overtly hateful language.
- An embedding matrix stores one token vector per row, with the row index matching the token’s vocabulary index.
- Embedding lookup uses token indices to retrieve and stack the corresponding rows, avoiding slow token-by-token loops.
- For a review such as “Predator is a masterpiece,” the model can average the four token embeddings to form one sequence vector for sentiment classification.
- An MLP applies linear transformations and elementwise nonlinearities such as ReLU or GELU to the averaged review vector.
- For two sentiment classes, the output projection uses a 2-by-D matrix to produce two logits, then softmax returns two scores that sum to one.
- The predicted label is the index of the largest softmax score, mapped to the application’s positive or negative label.
- Training updates MLP weights and biases using cross-entropy, also described here as negative log-likelihood, and gradient descent.
- The embedding matrix can be frozen or fine-tuned; pretrained embeddings offer a useful starting point, while random embeddings need larger updates to become useful.
- Accuracy can mislead on imbalanced data; precision measures the reliability of positive predictions, recall measures identified positives, and macro F1 gives each class equal weight.
- The lecture distinguishes skip-gram’s binary word-context prediction from the separate binary sentiment-classification task performed by an MLP.
- When input classes are not linearly separable, hidden layers can transform their representations so a final linear classifier can separate them.
- Fine-tuning pretrained embeddings makes small, task-specific adjustments through gradients, rather than requiring the model to learn useful vectors entirely from random initialization.
Transcript excerpt
0:00 Uh good afternoon everyone. Um hope you had a nice weekend. Uh yeah, it's been a while since I was teaching. How was the practical last time? I know some of you didn't have the best experience, but that's kind of to be expected. It's not the most chill of this at least. Um anything else you maybe want to share in this class setup? not not really in the class. Uh that's okay. Uh we are a little bit behind be uh with the grading mostly because of me because I'm still figuring out how to exactly grade it. But we'll try to uh be done with all of that soon. Uh but I don't know the exact date yet. I will
0:49 warn you if that uh changes. Um okay. So today we are going to continue where we left off. Uh and then once I'm done with everything I want to say about word embeddings, we are going to do a little quiz for 15 minutes and then we are going to see how much time we have left in the class, whether it makes sense to cover some more material or uh we just bounce and use the rest of our Monday. Any questions before I start? Okay, the room is big so I need to scan scan a wide space. All right. So, uh what we're going to chat now uh chat chat about now is uh the analysis of our word embeddings. We have used our skip gram. We have trained some word embeddings. Now, we have you know data structure which where for every word and
1:37 its index in the vocab. We have that embedding matrix where each row is the uh corresponding embedding of its corresponding word in the vocab. And we are going to just talk a little bit
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.