Save this video free

Mixture of experts language models

Mohit Iyyer · 1:12:25 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 11 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer explains how mixture-of-experts (MoE) transformers increase model capacity by replacing each layer’s single feed-forward network with many expert networks and routing each token through only a selected few. He details the router’s softmax and top-k selection, why hard routing complicates gradient-based training, and how sparse activation trades lower per-token computation for higher memory use, with Qwen, DeepSeek, and Switch Transformer as examples.

Key takeaways

Chapters

0:00 Why Transformer Language Models Need More Parameters
5:00 Dense Transformers Use the Same Parameters for Every Token
8:00 Mixture of Experts Adds Capacity Through Sparse Activation
12:00 Replacing One Feed-Forward Layer with Multiple Experts
15:00 Token Routing Selects Which Experts Process Each Representation
20:00 The Router Produces Scores Over Experts
28:00 Top-k Gates Weight the Active Expert Outputs
34:00 Why Sparse Top-k Routing Is Hard to Differentiate
40:00 Top-k Routing Behaves Like a Switch
45:00 Router Scores Can Change Which Experts Train
51:00 Expert Computation Can Be Batched Across GPUs
58:00 Load-Balancing Losses Prevent Unused Experts
1:03:00 Total Parameters and Active Parameters Measure Different Costs
1:08:00 MoE Inference Speed Depends on Batch Size and Memory Traffic

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:03 Okay, let's get started. So today we are going to cover a modification of transformer language models that is basically kind of has become the standard for modern language models and these are called mixture of experts um language models. So this is not a topic I've covered in previous years but since now basically every model is an instance of mixture of experts I thought it would be worthwhile to cover it especially because it's uh an architectural modification. So it nicely connects to the stuff from um last time we were talking about um transformers the giant um multi-layer perceptron or feed

0:55 forward layer that all the parameters live in. Um, so essentially here we're trying to modify that giant feed forward layer where the concatenation of all of the individual head representations go um to add more parameters into our model. And um you know we haven't yet covered scaling laws of language models. So, um it may not be super clear right now why we want to have more parameters, but um let me just write down our topic of so this is what we're covering. Um basically the highle thing that you need

1:44 to know that we will cover in depth in a later lecture is that bigger models plus more data equals lower loss. So, and this obviously equals better next token prediction. So you will just have to take on faith that this is true that bigger models and more data equals more powerful language model. We will get to the actual projections of this and how um you can kind of predict what is going to happen with a certain model size and data size

2:32 um probably next week. Um but for now let's focus on this part which is bigger models and we'll get to the data part

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.