1786 lecture2 Sep15
Overview
The lecture explains how neural networks learn word embeddings by predicting which words occur near one another, turning the distributional hypothesis into a self-supervised training task. It builds a skip-gram predictor from corpus-derived target-context pairs, one-hot output labels, learned embeddings, softmax, and cross-entropy, then connects the mechanism to language models and introduces negative sampling as a faster alternative for large vocabularies.
Key takeaways
- The distributional hypothesis makes embedding learning possible: words that appear in similar contexts can be represented by related vectors, even when their meanings are not identical.
- A sentence corpus automatically supplies self-supervised target-context labels, avoiding the manual annotation required for many other prediction tasks.
- Backpropagation can update the input word embeddings themselves, allowing gradient descent to learn representations alongside ordinary neural-network weights.
- A full softmax predictor needs one output per vocabulary item, making it expensive for vocabularies that can range from tens of thousands to around a million entries.
- Skip-gram with negative sampling replaces full-vocabulary classification with binary association prediction over positive corpus pairs and randomly sampled negative pairs.
- Autoregressive language models generate text by predicting one token at a time, appending each prediction to the input, and repeating the process.
Chapters
- Assignment 1 is due Monday at 9:00 p.m.; sections 3 and 4 build on neural-network material students are expected to know already.
- Word embeddings are vectors whose values can encode meaning and relationships, even though their individual numbers are not directly interpretable.
- The lecture focuses on how neural-network training can create these vectors from language.
- Similar words can have similar vectors, while vector differences can encode relationships such as queen minus king matching woman minus man.
- Country-currency word pairs can exhibit analogous vector relationships, suggesting embeddings capture more than simple similarity.
- Earlier approaches counted word co-occurrences; the lecture instead focuses on neural-network methods associated with Yoshua Bengio and Geoffrey Hinton.
- A neural network trained to make predictions that depend on word meaning must encode useful information about meaning internally.
- The lecture frames embedding creation as a prediction task, with learned representations emerging because they help the network predict correctly.
- This compact prediction mechanism is presented as closely related to the core of large language models.
- The sentences “The mathematician went to the store” and “The engineer went to the store” place the two professions in similar contexts.
- Replacing either profession with “solved the problem” gives further contextual evidence that the words are related, without implying they mean the same thing.
- The distributional hypothesis says that words appearing in similar contexts are related, summarized by the idea that a word is known by the company it keeps.
- The training objective predicts which words typically occur together, rather than directly asking whether two words have identical meanings.
- Repeated contexts can make words such as engineer and mathematician acquire related embeddings.
- The prediction task uses surrounding words as evidence and learns representations that encode aspects of word meaning.
- Section 3 uses a limited vocabulary of about 11 words, including I, he, she, hold, rub, dog, cat, a, and the.
- The corpus contains simple sentences such as “I hold a dog” and “She holds a dog,” with forms such as holds reduced to hold.
- Students train two-dimensional embeddings so related words can be plotted and compared.
- The training task asks which words belong together, and sentences supply positive examples without human annotators labeling each pair.
- Written language provides large quantities of automatically extractable training data across many languages.
- The usefulness of a corpus depends on linguistic coherence, though grammatical errors do not necessarily eliminate meaning.
- For “I hold a dog,” a window of plus or minus two words makes I and a context words for hold.
- The same sentence also produces pairs such as hold with dog and a with dog, depending on which word is the target.
- The target word is the input-side word, while nearby words are context words used as training labels.
- The basic predictor receives word A and predicts a context word B from the same vocabulary.
- A target can have multiple valid context words, so the output must represent alternatives rather than assume one uniquely correct word.
- The window size determines which target-context pairs are generated and is a parameter students explore in the assignment.
- A vocabulary of four words requires four one-hot values, with exactly one position set to 1 for each word.
- For example, hello can be represented as 1, 0, 0, 0 and goodbye as 0, 1, 0, 0.
- One-hot encoding identifies a word by its vocabulary position but carries no semantic information; it is useful for output labels.
- With a vocabulary of 10 words and embedding dimension d = 4, each word has four embedding values.
- The values form an embedding matrix that allows the model to retrieve a vector for any vocabulary item.
- Embeddings are randomly initialized and updated during training; assignment 1 section 3 uses dimension 2, while sections 1 and 2 use dimensions from 50 to 300.
- For a target such as hold, the four associated embedding values become the neural network input.
- Backpropagation updates embedding values as well as conventional network weights, because gradients can flow back to the input representation.
- The network needs one output score per vocabulary word, then applies softmax to turn logits into a distribution over possible context words.
- A vocabulary of 10 words requires 10 output neurons, each producing a logit for one word.
- Softmax normalizes those logits into values that sum to one and can be interpreted as probabilities for candidate context words.
- For a labeled pair such as hold and dog, cross-entropy penalizes the model when it assigns low probability to the correct one-hot target.
- Backpropagation can continue through the network to update the input embeddings, not just the weights inside its neurons.
- Different examples can pull an embedding in different directions; training combines those signals into representations that generalize across contexts.
- This interaction between examples helps explain how vectors can capture recurring relationships rather than memorizing individual sentences.
- The output layer's weights can be viewed as another set of word representations, distinct from the embeddings used as inputs.
- A neuron's weighted sum computes a dot product between an input embedding and an output-side vector, making compatibility between words affect the score.
- Sharing input and output embeddings can reduce parameters and may work, but the lecture treats separate representations as the standard setup.
- A large language model also maps input tokens to a prediction over a vocabulary, though its vocabulary may contain 50,000 to 400,000 tokens.
- In autoregressive generation, the model predicts one token, appends it to the input, and predicts the next token repeatedly.
- The same broad input-to-vocabulary-output mechanism connects the small context predictor to transformer language models.
- The skip-gram approach predicts which words are associated with a target, but a full vocabulary output layer can be costly when vocabularies reach 10,000 to one million items.
- Skip-gram with negative sampling instead takes a word pair and predicts a binary yes-or-no association, using a single output.
- Positive pairs come from nearby words in the corpus; negative examples are created by randomly sampling word pairs, most of which are assumed to be unassociated.
- Assignment 1 section 3 lemmatizes holds to hold and rubs to rub so related word forms share an embedding.
- The larger corpus in section 4 also handles punctuation, while tokenization breaks text into reusable pieces to control vocabulary size.
- Students are directed to complete assignment sections 3 and 4 and consult the textbook for details on preprocessing and negative-sampling refinements.
Transcript excerpt
0:03 Good morning everyone. Welcome to ECE 1786 lecture number two. Uh the work that you have in flight is assignment one which is due next Monday at 9:00 p.m. If you're new to the course, make sure that you've looked at lecture zero, which says these are the things you need to know in order to be able to do the assignments in this course. And now there's lecture one that is sort of the beginning of the story of what makes gosh this modern world of large language models and AI uh incredible. And so we began I'll talk about that in a second. Um uh on about assignment one. It is big and long. If you haven't started it then you're you're behind because it it uh it will take you a big chunk of this week to get
0:51 it done. I told you last week you need to get going on sections one and two. Sections three and four I'm going to cover uh this week and it's going to rely on what the background material of the course uh says that you need to know. So this is not a beginning neural network course. It is a subsequent one. Um yes so last day I began by talking about uh word embeddings and vector words word embeddings/vectors it's the same same two words meaning the same thing they get used a lot uh I talked about their properties and something that I'm keen on that I put in assignment one which is to look at these vectors which are just a bunch of incomprehensible numbers and say is there a way we can do something that uh
1:41 directly creates meaning to us humans in uh by looking at those vectors just to kind of know get you used to them. And I
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.