ECE 1786 Lecture 1 September 8, 2026
Overview
Jonathan Rose introduces ECE 1786’s three goals: understand how large language models and transformers work, learn to build applications with them, and practice turning ideas into clear engineering projects. The first technical lecture explains why rule-based natural language processing struggles, then uses word embeddings, GloVe vectors, Euclidean distance, cosine similarity, and vector arithmetic to explore meaning, analogy, bias, and interpretability. Assignment 1 applies these ideas; students may not use generative AI to write its code, while AI use is encouraged for the course project.
Key takeaways
- Rule-based NLP struggles with polysemy, paraphrases, and context: bank has several senses, and the headline about a tumor fighting back requires background knowledge to interpret correctly.
- Word embeddings replace hand-written semantic rules with vectors learned from language data; Euclidean distance and cosine similarity provide different ways to compare those representations.
- Nearest-word searches make high-dimensional embedding spaces easier to inspect, but their results reflect both learned linguistic associations and biases in the training text.
- Vector arithmetic can encode relationships such as king minus man plus woman approximately equaling queen, but analogy results are inconsistent and can reproduce stereotypes.
- Averaging selected word vectors can create useful combined representations and support sentence classification, while averaging too many vectors may erase distinctions.
- ECE 1786 separates learning fundamentals from AI-assisted creation: generative AI is prohibited for Assignment 1 code but encouraged for the final project.
Chapters
- Jonathan Rose frames the course around LLM internals, building applications with LLMs, and engineering projects that develop ideas into working systems.
- The course includes technical lectures, four assignments, and a major project; the free Jurafsky and Martin textbook received a new edition in August 2026.
- Assignment 1 code must not be generated with generative AI; AI use is encouraged for the project, where students are expected to build ambitious ideas.
- Rose warns that Assignment 1 is lengthy, recommends starting immediately, and says its later sections test neural-network background that the course will not teach.
- Natural language distinguishes human language from computer languages such as Fortran, Python, C, and Rust.
- Rose introduces NLP as a field dating to the 1960s and asks why computers find human language difficult to process.
- The word bank can refer to a financial institution, a riverbank, a power bank, a bank shot, or actions such as banking an aircraft.
- Banking on something adds another related sense, illustrating how a single word can carry multiple meanings that depend on context.
- A procedural program would need to distinguish these uses without relying on a simple one-word-to-one-meaning mapping.
- “He decided the risk was worth taking” and “Despite the danger, he chose to go ahead” express similar ideas with different words.
- The headline “Boy paralyzed after tumor fights back to gain black belt” requires knowing that the boy, not the tumor, fights back.
- “I saw bats” can mean seeing animals or sawing sports bats, depending on context and interpretation.
- Too many linguistic variations overwhelm procedural rules and grammar-parsing approaches, despite decades of work on them.
- Rose presents encoding meaning into numbers as the foundation of modern NLP approaches based on deep learning.
- The shift became especially consequential around 2012, when neural-network NLP methods began showing strong results at larger scale.
- Earlier statistical methods assigned word vectors using counts of how often words appeared near other words.
- Yoshua Bengio explored neural-network approaches to language in the 1990s; later work helped bring neural NLP into wider use.
- Meaning can be represented at several levels, including words, sentences, paragraphs, and entire documents.
- A word embedding is a vector of numbers; words with related meanings are expected to have nearby vectors.
- A vocabulary can be represented as a table mapping each word, such as apple or banana, to a vector with coordinates such as A0 through A99.
- Euclidean distance measures separation by taking the square root of the sum of squared coordinate differences.
- Rose notes that embedding dimensions are often thousands in modern models and are not generally interpretable one by one.
- Rose cautions that intuitions about vector directions become difficult to maintain in spaces with 100, 2,000, or 15,000 dimensions.
- He connects large concept vectors to possible representations of thought in the brain, while explicitly presenting that connection as a belief rather than an established fact.
- The Assignment 1 notebook uses Stanford’s GloVe word vectors, a resource built using a combination of count-based and neural methods.
- The notebook loads more than 1.29 million word entries, with available vector sizes ranging from 50 to 300 dimensions.
- PyTorch code maps words to indices and retrieves their vectors; the notebook displays the 50-number embedding for apple.
- Cat and dog have a Euclidean distance of about 1.8, while apple and banana are about 4.7 apart, though raw distances are not easy to interpret.
- Good and bad can be close because embeddings reflect shared usage contexts as well as similarity in meaning.
- Cosine similarity compares vector directions, normalizing the dot product to a score from minus one to one.
- Vectors such as [1, 1, 1] and [2, 2, 2] have cosine similarity 1, while [1, 1, 1] and [-1, -1, -1] have similarity minus one.
- In the GloVe examples, cat and dog score about 0.94, apple and orange about 0.61, good and bad about 0.8, and good and perfect about 0.87.
- A single word embedding can combine multiple senses; transformer processing helps resolve which meaning applies in context.
- A notebook function computes distances from a supplied vector to all one million-plus GloVe entries, sorts them, and returns the closest words.
- For cat, nearest results include dog, rabbit, monkey, and pet; for nurse, results include dentist, doctor, pediatrician, and midwife.
- Nearest-word results can reveal both learned associations and characteristics of the training text, as with names clustered around Elizabeth or Chow.
- The function can also inspect arbitrary vectors, including averages of word vectors, rather than only vectors tied to single vocabulary words.
- The classic analogy is king minus man plus woman, which returns queen among the closest words; reversing the relationship can return king.
- Related patterns include big versus biggest and small versus smallest, as well as country and currency or country and capital relationships.
- Not every analogy works reliably: grandmother minus mother plus father does not return the expected grandfather, while old minus young plus father does.
- Rose demonstrates bias with doctor minus man plus woman returning nurse, reflecting gender patterns present in the training text.
- Word-vector associations can expose stereotypes inherited from training data, including the doctor and nurse example.
- Rose says Assignment 1 asks students to examine examples of embedding bias.
- A linked visualization uses principal component analysis to project 200-dimensional word vectors into three dimensions.
- Clicking harvest reveals nearby words such as wheat, autumn, and celebration, offering an intuitive but reduced view of the space.
- Rose reviews the dot product as the core calculation behind cosine similarity and connects it to convolutional neural network operations.
- The vector difference between queen and king corresponds approximately to the difference between woman and man.
- Tomáš Mikolov’s work reports other recurring vector relationships, including comparative and superlative forms and country-currency pairs.
- Rose presents these patterns as useful but imperfect clues, not as a complete explanation of how embeddings encode language.
- In word2vec, word vectors emerge as a byproduct of training a neural network to make predictions based on language.
- Words begin with a simple encoding that does not itself contain meaning; prediction training forces useful semantic information into learned representations.
- Rose previews this mechanism as the subject of the following lecture and notes that GloVe was produced by a different method.
- Rose proposes human-readable dimensions such as color, temperature, plant content, and human orientation, then assigns illustrative values to mountain, ocean, sun, judge, and grass.
- A single prompt word such as color may be ambiguous, so a more specific concept can use a cluster such as red, green, blue, and black.
- Assignment 1 Part 2 asks students to estimate interpretable properties using cosine similarity and word examples.
- The two methods are to average the example-word vectors and compare the result to a target, or average the individual cosine similarities to the target.
- Averaging a modest number of word vectors can combine concepts, but averaging too many may blur their information.
- Rose previews Assignment 3, where averaged word embeddings help a convolutional neural network classify sentences as objective or subjective.
- The linearity hypothesis proposes that some abstract concepts in neural networks combine through addition, despite the networks’ nonlinear operations; Rose describes this as an unproven idea.
- Rose again urges students to begin Assignment 1 early, noting that one graduate student started two days before the deadline and cried by Section 4.
- Google Colab is an option, but Rose recommends learning to install software on a personal computer.
- For the course’s setup instructions, Rose recommends UV as a fast alternative to pip and Conda or Anaconda.
Transcript excerpt
0:03 All right, good morning everyone and welcome to ECE 1786 lecture number one. You hopefully have seen lecture number zero. I've sent out messages on that. If you're not registered in the course, you can see everything pretty much everything that's on the main course website on this public website. If you Google ECE 1786, you usually end up in the 2025 site, but uh there is one with 2026 as well. Um So, uh make sure if you haven't, you you've gone and watched lecture zero because it describes the entire structure of this course and I'm not going to not going to really repeat that and there's a bunch of moving pieces to the course. And uh I ask you to also carefully consider if you have the right technical background for this course because the assignments require it and uh
0:52 yeah, it's the we have we don't actually have a way to enforce prerequisites in graduate school here. So, we So, uh I used to just tell look ask people to tell me about their background and kick them out if they didn't have it. Now, I let you decide for yourself mainly after finishing the last two sections of assignment one, you can you can go well, no, actually I don't know any of this. I better I better go take the courses that uh that uh teach the background material. Um I will recap lecture zero. I want to say a few things about uh the core the core items there. First of all, this course is about three things. The first thing is to teach you the inner inner workings of large language models, transformers, and how they're trained
1:40 and how they're structured. And so, to have some fundamental understanding of this thing that is revolutionizing
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.