Save this video free

1786 lecture2 Sep15

Jonathan Rose · 1:33:34 · Watch on YouTube

Part of University of Toronto ECE 1786. See all 10 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture explains how neural networks learn word embeddings by predicting which words occur near one another, turning the distributional hypothesis into a self-supervised training task. It builds a skip-gram predictor from corpus-derived target-context pairs, one-hot output labels, learned embeddings, softmax, and cross-entropy, then connects the mechanism to language models and introduces negative sampling as a faster alternative for large vocabularies.

Key takeaways

Chapters

0:00 Course Context and Why Word Embeddings Matter
2:13 Embedding Similarities, Analogies, and Historical Context
8:02 The Core Idea: Predict Meaning-Dependent Language
16:12 Distributional Hypothesis Through Engineer and Mathematician
20:00 Context Prediction Produces Meaningful Embeddings
22:02 Assignment 1 Section 3 and Its Small Corpus
26:02 Natural Language Supplies Self-Supervised Training Labels
31:01 Create Target-Context Pairs with a Two-Word Window
35:21 Define a Neural Network That Predicts Context
39:09 One-Hot Encoding Represents Vocabulary Labels
44:33 Build the Embedding Matrix from Learned Word Vectors
50:17 Feed Learned Embeddings into the Predictor
1:01:06 Ten Output Neurons, Softmax, and Cross-Entropy
1:05:19 How Gradients Reconcile Conflicting Training Examples
1:08:02 Input and Output Embeddings as Learned Comparisons
1:17:39 From Context Prediction to Autoregressive Language Models
1:22:03 Skip-Gram and Negative Sampling Reduce Output Costs
1:30:00 Lemmatization, Tokenization, and Assignment Work

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:03 Good morning everyone. Welcome to ECE 1786 lecture number two. Uh the work that you have in flight is assignment one which is due next Monday at 9:00 p.m. If you're new to the course, make sure that you've looked at lecture zero, which says these are the things you need to know in order to be able to do the assignments in this course. And now there's lecture one that is sort of the beginning of the story of what makes gosh this modern world of large language models and AI uh incredible. And so we began I'll talk about that in a second. Um uh on about assignment one. It is big and long. If you haven't started it then you're you're behind because it it uh it will take you a big chunk of this week to get

0:51 it done. I told you last week you need to get going on sections one and two. Sections three and four I'm going to cover uh this week and it's going to rely on what the background material of the course uh says that you need to know. So this is not a beginning neural network course. It is a subsequent one. Um yes so last day I began by talking about uh word embeddings and vector words word embeddings/vectors it's the same same two words meaning the same thing they get used a lot uh I talked about their properties and something that I'm keen on that I put in assignment one which is to look at these vectors which are just a bunch of incomprehensible numbers and say is there a way we can do something that uh

1:41 directly creates meaning to us humans in uh by looking at those vectors just to kind of know get you used to them. And I

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.