1786 lecture6 Oct13

Overview

Jonathan Rose connects word embeddings to neural-network text classification, then uses Assignment 3 to compare a single-neuron logistic regression model with a one-dimensional CNN for labeling sentences as objective or subjective. He explains GloVe inputs, averaging and padding, kernel feature detection, max pooling, embedding fine-tuning, and how to inspect learned weights; he also outlines course deadlines and resources for following AI developments.

Key takeaways

Assignment 3 compares a 100-dimensional GloVe sentence-average logistic-regression model with a one-dimensional CNN to classify movie-related sentences as objective or subjective.
Pang and Lee’s training-data strategy uses movie reviews as mostly subjective examples and IMDb plot summaries as mostly objective examples, accepting some label noise in exchange for a practical dataset.
A CNN kernel of size k scanning an input of n words with stride 1 produces n − k + 1 responses; max pooling retains each kernel’s strongest match before the MLP classifier.
GloVe embeddings can be fine-tuned through backpropagation for a target classification task, showing that embeddings are trainable parameters rather than merely fixed lookup values.
Nearest-word inspection of learned logistic-regression weights and CNN kernel vectors can make some neural-network features interpretable.
LLMs can classify through prompting with far fewer task-specific examples than conventional neural-network training, while still requiring concrete evaluation and expert-designed workflows.

Chapters

0:00 Approval Deadlines and Proposal Presentation Requirements

Approval in principle is due October 13, and Rose must approve a project before students proceed with the proposal.
The proposal, consisting of a document and presentation slides, is due the following Monday, with presentations scheduled for Tuesday and Wednesday.
Students may miss only one of three presentation sessions for a valid reason, submitted through the course form.

2:20 Project Approval Favors Concrete Ideas Over AI Marketing

Rose had reviewed 20 approval submissions and handled about 14, with an approval rate near 60% at that point.
Rejected proposals receive a personalized email, and students should discuss feedback with a TA and resubmit promptly.
Avoid stock-prediction projects and recipe-generation ideas; proposals should specify concrete inputs, outputs, and a substantive problem rather than say only “AI-powered.”

4:40 Course Roadmap: Embeddings, Neural Networks, and Transformers

Assignment 3, based on the lecture, is due November 2, leaving time for project work and reading week.
The course combines designing novel applications for language models with understanding how current LLMs work.
The learning sequence moves from embeddings to ordinary neural-network processing, then to transformer construction, training, and guardrails.

8:00 Word Embeddings Encode Meaning and Relationships

An embedding represents a word as a vector, with dimensions ranging from roughly 50 to 10,000 in large transformer systems.
Words with related meanings occupy related positions, and vector arithmetic can capture relationships, as explored in Assignment 1.
A single word embedding cannot distinguish senses such as a river bank, a financial bank, or a turn in banking; transformers address context-dependent meaning.

10:10 Traditional Classifiers Versus Prompted LLM Classification

Traditional machine learning maps numerical inputs to categories, such as classifying images as cats, dogs, or planes.
Assignment 2 contrasted training a classifier with prompting an LLM to judge whether a sound is soft or hard.
Prompt-based classification can require far fewer examples than training a conventional neural network, even though both approaches produce labels.

14:10 Language Classification Applications: Sentiment, Spam, and Safety

Sentiment analysis can track whether social-media posts about a company's product are positive, negative, or neutral.
Other classification tasks include spam detection, language identification, and estimating the difficulty of a user’s question.
Rose described mental-health applications such as detecting depression, suicidal ideation, and therapy-related change talk versus sustain talk.
Legal teams can classify email for privileged lawyer-client communications, reducing manual review work in litigation.

22:40 Assignment 3: Objective and Subjective Sentence Labels

Assignment 3 trains two neural networks to classify individual sentences as objective or subjective.
Pang and Lee’s dataset uses movie reviews as a source of mostly subjective sentences and IMDb plot summaries as a source of mostly objective sentences.
The labels are noisy because not every sentence in a review is opinion and not every plot-summary sentence is purely factual, but the dataset still provides a useful signal.
The assignment supplies the examples in a CSV or TSV file, so students can focus on model construction and classification.

32:00 Tokenization Breaks Text into Model-Usable Units

Modern LLMs process tokens, which are often word fragments rather than complete words and average about three-quarters of a word.
A previously unseen word can be decomposed down to individual letters, though those pieces may carry little standalone meaning.
Assignment 1’s lemmatization and punctuation removal provide simpler examples of turning raw text into model-ready units.

35:00 GloVe Embeddings and the Unknown-Word Fallback

Assignment 3 uses pretrained GloVe word embeddings, with an embedding dimension of 100 rather than Assignment 1’s 50.
Words absent from the GloVe vocabulary map to a special unknown-word entry so an unfamiliar term does not crash processing.
Each sentence becomes a sequence of 100-dimensional word vectors, one vector for every recognized word.

38:30 Binary Labels, CNN Padding, and the Two Assignment Models

The classifier outputs 1 for subjective and 0 for objective sentences.
Students build both a single-neuron logistic regression model and a convolutional neural network using the supplied training examples.
Because sentence lengths vary, the CNN input is padded to the maximum length; zero padding minimizes the influence of added positions.

43:40 What Averaging Word Embeddings Preserves or Erases

The logistic-regression model averages each of the 100 embedding coordinates across all words in a sentence to create a fixed-length vector.
Averaging can retain shared semantic features, but averaging too many word vectors can wash out distinctions; Rose noted that sentence-length inputs still work reasonably well.
The discussion connects embedding averages to research questions about whether a model’s concepts are best represented by one vector or averages across multiple vectors.

52:30 Logistic Regression: One Neuron from Average Embedding to Label

A single neuron computes a weighted sum of the 100 average-embedding values, adds a bias, and applies a sigmoid.
A decision threshold of 0.5 converts the sigmoid output to the binary class: 1 means subjective and 0 means objective.
Students implement and train this model with PyTorch, backpropagation, and gradient descent; short sentence embeddings average well enough for it to perform surprisingly effectively.

59:00 Inspecting Logistic-Regression Weights with Nearest Words

The 100 trained weights form a vector that can be compared with GloVe embeddings, much like a dot product or cosine-similarity operation.
Assignment 3 asks students to use a print-closest-words method to interpret what the learned classifier weights represent.
Nearest-word inspection offers a window into a neural network’s learned parameters, rather than treating the model entirely as a black box.

1:02:20 CNN Refresher: Image Kernels Detect Local Patterns

For image classification, a CNN scans learned kernels across pixel arrays; a 3×3 kernel can respond to a local pattern such as an angled line or whisker.
Each kernel position produces a dot product, and scanning creates a feature map that indicates where the learned pattern appears.
Training adjusts initially random kernels using labeled examples; later layers can combine simple features into larger structures.

1:10:20 A One-Dimensional CNN Scans Sequences of Word Embeddings

For text, a CNN scans left to right across the sequence of word embeddings instead of across a two-dimensional image.
A kernel spanning two words has 200 weights when each GloVe vector has 100 dimensions; larger kernels detect patterns across more words.
The kernel dot products learn local patterns of meaning, making the text CNN a useful conceptual step toward transformer mechanisms.
Assignment 3 varies the number of kernels and uses two kernel sizes, creating hyperparameters students must evaluate.

1:17:00 CNN Output Length, Max Pooling, and Assignment Constraints

With input length n, kernel size k, and stride 1, a valid scan produces n − k + 1 values; a two-word kernel therefore produces n − 1 outputs.
Max pooling selects the strongest response from each kernel’s scan, representing the strongest match anywhere in the sentence.
The pooled values feed a fully connected network, or MLP, which produces the final classification.
Students must implement the PyTorch CNN themselves; generative AI may help explain concepts but cannot answer or complete assignment questions.

1:26:30 Fine-Tuning GloVe Embeddings for a Classification Task

The CNN can either keep pretrained GloVe embeddings fixed or allow backpropagation to update them during training.
Fine-tuning makes embeddings more suitable for the specific objective-versus-subjective task and can improve performance.
The experiment illustrates that embeddings are trainable network parameters, alongside CNN kernels and MLP weights.
Transformer systems learn token embeddings as part of the model rather than requiring every input embedding to be fixed in advance.

1:29:20 Interpreting CNN Kernels with GloVe Nearest Neighbors

A one-word CNN kernel is a 100-dimensional vector and can be inspected with the same nearest-word technique used for logistic-regression weights.
A two-word kernel has 200 parameters, which can be split into two 100-dimensional vectors and examined separately.
The assignment uses these inspections to connect learned CNN features to interpretable word-level patterns.

1:31:20 Keeping Up with AI Through The Batch and Other Sources

Rose recommends Andrew Ng’s weekly newsletter, The Batch, for accessible coverage of new models, research papers, AI jobs, and societal questions.
AlphaSignal offers a daily stream of model and tooling updates, including open-weight versus closed-model comparisons and topics such as routers.
Practical AI and Ethan Mollick’s writing provide additional accessible perspectives on AI applications and the changing role of human expertise.
Rose’s key point is that models may know how to perform many tasks but still need expert guidance and harnesses to select and execute the right steps.

Transcript excerpt

0:05 Good morning everyone and welcome to ECE 1786 lecture number six. There's a bunch of work in flight. There is a lot of work in this course and I've I've newly made it not easy, not difficult. Uh and I apologize for that. Uh but I think the upside of having hard work is we're touching something that is really important that's happening right now and changing right now in all kinds of ways. So I've changed the course in a bunch of ways but I've also made it harder. Some of the timelines are a bit compressed both expanded and compressed. One thing not not everybody I think has paid attention to is that the approval in principle stage was due today October 13th. So you have to the end of the day to submit it and you need my approval

0:53 before you should really start going and working on the proposal which is due next Monday. And it's two things. It's a document and it's a bunch of slides that you're going to you're going to um present on Tuesday and Wednesday. Remember there are three sessions and if you can't make one of those sessions, there's a form to fill out. You're only allowed to miss one of them because you're going to be reviewing others in one session and presenting your own. And so there's only two people that filled out that form so far. Usually you get people do have a good reason, not a not a I don't want to go that day reason, a good reason for missing that. So, uh, back to approval and principal. Uh, I've been looking at a bunch of them. I've seen 20 submitted. I've dealt with I don't know 14 of them. The success rate so far is about 60%. So, some the

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.