UUtah CS 6340 NLP | Fall 2026 | Intro to vector semantics
Overview
The lecture explains why NLP systems need numerical representations that capture relationships in meaning, not just token identity. It contrasts one-hot and bag-of-words vectors, which can be sparse and fail to encode semantic similarity, with dense token embeddings; it previews Word2Vec and notes that modern LLM token embeddings use related intuition but are not identical.
Key takeaways
- A token's vocabulary ID is a lookup label, not a semantic representation: assigning puppy and kitten different IDs does not tell a model that they are related.
- One-hot vectors have vocabulary-sized dimensions and exactly one nonzero value per token, while a bag-of-words vector counts token occurrences but discards word order.
- With a vocabulary of 100,000 to 200,000 entries, bag-of-words representations are very sparse because an ordinary input contains only a small fraction of the vocabulary.
- One-hot and bag-of-words encodings cannot distinguish semantic closeness: puppy and kitten are no closer than puppy and table under this representation.
- Word2Vec offers a compact dense-vector approach, with classic representations around 50 to 300 dimensions, that motivates placing semantically related words near one another.
- Classic Word2Vec embeddings are not identical to the token embeddings used in LLMs, but the idea of encoding semantic relationships in vector space remains relevant.
Chapters
- A tokenizer applies merge rules such as Byte Pair Encoding (BPE), then maps each token to an integer vocabulary ID.
- Token IDs alone do not express relationships: a useful NLP representation should indicate that terms such as puppy and kitten are related.
- Similarity between text segments supports tasks including paraphrase generation, question matching in search, and summarization.
- One-hot encoding creates a vocabulary-sized vector with a single 1 at the token's ID; all other entries are 0.
- For the example vocabulary of five tokens, the bag-of-words representation of “the cat sat sat” is [0, 1, 0, 1, 2], counting each token without retaining order.
- Real vocabularies may contain 100,000 to 200,000 tokens, making bag-of-words vectors extremely high-dimensional and sparse.
- Neural networks typically use dense inputs; the lecture notes that sparse inputs can produce training difficulties such as messy gradients.
- One-hot vectors make puppy and kitten as orthogonal as either word is to table, despite the stronger relationship between puppy and kitten.
- The same encoding cannot show that good and bad have opposite meanings; token identity alone captures neither similarity nor semantic distance.
- The lecture previews Word2Vec, popularized around 2015, as a method for learning dense token vectors, historically around 50 to 300 dimensions rather than 100,000 or more.
- Modern embeddings may have a few thousand dimensions, and LLM token embeddings are not exactly the same as classic Word2Vec vectors, though the intuition of representing meaning through vector relationships remains useful.
Transcript excerpt
0:02 You plug in the dollar. >> Oh, no. Like, >> okay. Um, welcome back. How was that? Fine. Yeah, enough time. I think that was enough time. We all done it before uh before the end. Uh, good. Okay. So, that's roughly what you can expect from all the quizzes. When we have more content to cover it, they are probably going to be a little bit longer. we are going to adjust the time and yeah if you have any concerns send them my way uh happy to hear if something doesn't work out in this format. Um okay um I hear also maybe some of you are discussing also the uh the quiz which was the online quiz in uh canvas for checking your background. Um I don't have the
0:49 solutions handy uh just to hand them out. So yeah, I'll stick around after the lecture. If you have any questions about those, just just ask me and I'm happy to explain the answers and tell you what the right answers are. Okay. Uh moving on with uh our topics. So um this is yet another visualization of you know um prompting uh CHPT and kind of seeing what's happening after that. Uh so again you're seeing this prompt explain some superconductors like I'm five years old and now you know that there is this tokenizer which involves both the vocabulary of possible tokens and uh the segment or encoder part of the tokenizer
1:37 which has with at least with the case of BP these rules uh which are basically greedily apply these merges uh to this new piece of text. Um, and then you get tokenize text. And important thing to remember, something that I didn't quite visualize or maybe stress enough is that vocabulary is a dictionary where each
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.