UUtah CS 6340 NLP | Fall 2026 | BPE Tokenizer (Part 2)
Overview
The lecture completes the BPE tokenizer discussion, contrasting the classroom character-level version with production byte-level BPE over UTF-8, which handles emojis and multilingual text. It examines tokenizer evaluation through fertility and parity, then shows how tokenization can affect sentiment, code performance, model behavior, and the cost of generating in languages such as Gujarati.
Key takeaways
- Production byte-level BPE applies merges to UTF-8 bytes, allowing a tokenizer to cover emojis and diverse scripts without relying on a Latin-character inventory.
- Lower tokenizer fertility can shorten token sequences, but Ali et al.’s findings discussed in class show that fertility alone does not predict downstream language-modeling performance.
- A BPE split can affect model behavior: splitting a negative word into a fragment resembling “superb” can bias sentiment, while special whitespace tokens can improve code-task performance.
- A vocabulary token that is poorly represented in pretraining can have a weak embedding; SolidGoldMagikarp became a glitch token associated with erratic outputs when inserted into prompts.
- Gujarati text took about ten times longer to generate than the English comparison in the cited demonstration, illustrating how token-count premiums can make multilingual model use less affordable.
- Parity evaluates whether equivalent text in different languages receives similar token counts; a large premium over English signals unequal tokenizer efficiency, though it does not by itself explain output quality.
Chapters
- The practice assignment has no deadline, so students are expected to work through it independently before the in-class practicum.
- The instructor emphasized understanding the BPE helper functions for counting adjacent pairs and applying merges, since the practicum may require implementing them.
- The course schedule is tentative, but quizzes and practicums are intended to stay on their announced dates unless major delays require changes.
- The original 2016 BPE explanation starts from characters, but production tokenizers commonly encode text as UTF-8 and run BPE over bytes.
- Byte-level BPE can represent emojis and text from non-Latin scripts without requiring a separate character inventory for every writing system.
- The course assignment keeps character-level splitting for simplicity; Jurafsky and Martin section 2.3 is recommended for more on Unicode and tokenization.
- Fertility is the average number of tokens needed to represent a word or document; fewer tokens can reduce generation length and give frequent whole-word units more contexts.
- The lecture contrasts a five-token split of “tokenization” with a two-token split, illustrating why lower fertility can seem preferable.
- Research discussed in class, attributed to Ali et al., finds that fertility does not reliably correlate with downstream language-modeling performance, so extrinsic evaluation still matters.
- Resources mentioned include Hugging Face’s tokenizer library and notebook, OpenAI’s tiktoken visualizer, and Andrej Karpathy’s simplified BPE repository.
- A sentiment model misread “superbizarre” as positive after BPE split it into a segment resembling “superb” and another fragment, showing how token boundaries can introduce misleading associations.
- In code, treating four or eight spaces as dedicated tokens improved coding-task performance in an example from around 2022.
- The SolidGoldMagikarp glitch token appeared in a BPE vocabulary but lacked a well-trained embedding, and including it in prompts could trigger bizarre or unhinged model outputs.
- The instructor noted that modern pretraining includes more code than in 2022, while current tokenizers may still use additional rules alongside BPE.
- A PhD-defense demonstration generated an English story in about 4 minutes, while the same prompt in Gujarati took about 40 minutes, roughly ten times longer.
- Over-fragmentation means some languages require many more tokens than English to express equivalent content, increasing generation time and cost.
- The lecture cautions that response quality also reflects uneven pretraining data, especially English’s overrepresentation, so tokenization alone does not explain quality differences.
- Parity compares token counts for equivalent material across languages; the token-count ratio relative to a baseline such as English is called the premium.
- The lecture recap connects tokenization to multilingual underperformance, confusing number splits that hinder arithmetic, and glitch-token behavior.
- Andrej Karpathy’s discussion of tokenization issues is recommended as further reading.
- Models that remove tokenization as a separate stage are an active research direction, but the instructor says the approaches are not yet ready.
- The instructor briefly mentions SuperBPE in response to a question about merging across word boundaries, while keeping the assignment focused on not merging across those boundaries.
Transcript excerpt
0:01 to be late to open AI and action. >> Good afternoon everyone. Let's let's get started. Hope you had a nice first week of classes. I felt my first week felt like five weeks. Uh it's hard to get into the rhythm after after the summer. Um I want to remind you in case you didn't see this so either remind you or uh let you know that we have released the first practice uh assignment and uh the idea is that you work on these assignments on your own in your own uh you know paceime you use autograder if you like uh which we definitely recommend you try out. Um and then next uh Wednesday uh we are going to meet here and you are going to uh do programming tasks related to the practice uh assignment in a person uh
0:50 using a collab that we are going to share in class. Um next Monday is holiday so we won't meet on Monday and we are going to uh meet on uh um yeah Wednesday next I mean this next Wednesday and Wednesday after um we are yet to announce our office hours. I will chat with the TAs after after the lecture today and then we will figure out those uh let's say today to to share with you all. Um one thing that I didn't say here in this announcement is uh in this practice assignment you have given uh you you were given some code and that code involves some of the helper functions. Um do not assume those will be given during the practicum. So engage with those helper functions and
1:37 understand what they are doing, why are they implemented and so on. Right? There are just two like doing the merging and counting pairs that uh that are uh
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.