Save this video free

UUtah CS 6340 NLP | Fall 2026 | BPE Tokenizer (Part 2)

UofU Data Science · 31:17 · Watch on YouTube

Part of University of Utah CS 6340. See all 20 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture completes the BPE tokenizer discussion, contrasting the classroom character-level version with production byte-level BPE over UTF-8, which handles emojis and multilingual text. It examines tokenizer evaluation through fertility and parity, then shows how tokenization can affect sentiment, code performance, model behavior, and the cost of generating in languages such as Gujarati.

Key takeaways

Chapters

0:00 BPE Practice Assignment and Practicum Preparation
4:36 Production BPE Uses UTF-8 Bytes for Unicode Coverage
7:12 Fertility Measures Tokenizer Compression, Not Model Quality
14:50 BPE Splits Can Affect Sentiment, Code, and Glitch Tokens
23:30 Gujarati Over-Fragmentation Creates a Token-Cost Fairness Gap
28:55 Tokenization Challenges and Research Beyond Separate Tokenizers

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:01 to be late to open AI and action. >> Good afternoon everyone. Let's let's get started. Hope you had a nice first week of classes. I felt my first week felt like five weeks. Uh it's hard to get into the rhythm after after the summer. Um I want to remind you in case you didn't see this so either remind you or uh let you know that we have released the first practice uh assignment and uh the idea is that you work on these assignments on your own in your own uh you know paceime you use autograder if you like uh which we definitely recommend you try out. Um and then next uh Wednesday uh we are going to meet here and you are going to uh do programming tasks related to the practice uh assignment in a person uh

0:50 using a collab that we are going to share in class. Um next Monday is holiday so we won't meet on Monday and we are going to uh meet on uh um yeah Wednesday next I mean this next Wednesday and Wednesday after um we are yet to announce our office hours. I will chat with the TAs after after the lecture today and then we will figure out those uh let's say today to to share with you all. Um one thing that I didn't say here in this announcement is uh in this practice assignment you have given uh you you were given some code and that code involves some of the helper functions. Um do not assume those will be given during the practicum. So engage with those helper functions and

1:37 understand what they are doing, why are they implemented and so on. Right? There are just two like doing the merging and counting pairs that uh that are uh

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.