Mixture of experts language models
Overview
Mohit Iyyer explains how mixture-of-experts (MoE) transformers increase model capacity by replacing each layer’s single feed-forward network with many expert networks and routing each token through only a selected few. He details the router’s softmax and top-k selection, why hard routing complicates gradient-based training, and how sparse activation trades lower per-token computation for higher memory use, with Qwen, DeepSeek, and Switch Transformer as examples.
Key takeaways
- An MoE transformer expands feed-forward parameter capacity by adding expert networks, while top-k routing limits each token to K of n experts.
- For an MoE layer, total parameter count is primarily a memory cost, while the number of active experts largely determines per-token computation.
- Hard top-k selection is non-differentiable: gradients can update selected gate scores and expert weights, but an expert generally receives no token-level training signal while inactive.
- Router score updates can change the expert ranking over time, enabling new experts to enter the active set and develop specializations.
- Expert imbalance can leave much of an MoE model’s capacity unused, so training commonly adds load-balancing objectives to encourage broader expert utilization.
- MoE inference speed is workload-dependent: small batches benefit from sparse computation, while expert-weight loading and routing overhead can reduce the advantage at intermediate batch sizes.
Chapters
- Mohit Iyyer frames mixture-of-experts language models as a modern transformer architecture focused on scaling model capacity.
- The lecture starts from the scaling premise that bigger models and more data tend to reduce loss and improve next-token prediction.
- Common dense-model scaling methods include adding transformer layers, widening hidden states or embeddings, and increasing attention heads.
- A dense model sends every token through the same attention and feed-forward parameters at each layer.
- Increasing depth or width raises parameter count, but also increases computation for every token and can make inference slow.
- Large dense models also require substantial memory, a concern when serving language models to many users.
- MoE models aim to store many parameters while activating only a small fraction for each token.
- Token-level routing lets a model choose experts as generation proceeds, rather than selecting one expert for an entire sequence in advance.
- The central change is to the transformer feed-forward layer, where much of the parameter count resides in large models.
- A conventional transformer layer has one feed-forward network, represented in the lecture by a single WZ matrix.
- An MoE layer has multiple feed-forward networks, each with its own WZ parameters; four experts would give four parameterized networks.
- With four experts and two active per token, each token uses only two networks rather than computing through all four.
- After masked multi-head self-attention, the concatenated head representation is the input to the MoE feed-forward stage.
- A separate router evaluates the token representation and selects experts, such as choosing two of four for each token.
- The outputs of the selected experts are combined to form the token’s layer representation, while the residual connection remains.
- For a d-dimensional token representation and n experts, the router uses a learned matrix W_R to produce n expert scores.
- A softmax over those scores gives a gate distribution for the token; this resembles attention’s use of a distribution, but the choices are experts rather than previous tokens.
- The number of active experts K is a model-design hyperparameter chosen and tuned empirically.
- The router selects the K experts with the highest gate scores and ignores the other n minus K experts for that token.
- Some implementations renormalize the selected gates so their weights sum to one among the active experts.
- The layer output is a gate-weighted sum of the selected experts’ feed-forward outputs, followed by the residual pathway.
- Only active gate values contribute to the token representation, so inactive experts receive no gradient from that token’s next-token loss.
- Using every expert with a softmax-weighted sum would make routing differentiable, but would remove the compute advantage of sparse activation.
- The top-k operation changes the selected set discontinuously, so small score adjustments often leave the active experts unchanged.
- In an example with four experts and K=2, changing a gate from 0.14 to 0.15 may not change the selected pair at all.
- Once a score crosses the ranking threshold, the active set can switch, abruptly changing which expert outputs form the representation.
- This hard selection gives the Switch Transformer its name and explains why ordinary backpropagation cannot directly differentiate through the top-k choice.
- Although the top-k selection itself is not differentiable, gradients can update the scores of experts that were selected.
- Repeated updates can change the score ranking, bringing a different expert into the active set so its parameters begin receiving training signal.
- As experts enter and leave the selected set, they can develop different specializations for token types or contexts, including patterns observed in code generation.
- Implementations can group tokens by selected expert, compute each expert only for tokens routed to it, and aggregate the resulting representations.
- Tokens can still be processed in parallel, and expert computations can be distributed across GPUs before their outputs are combined.
- Mohit Iyyer notes that efficient MoE training and inference require specialized implementation work, pointing to DeepSeek’s released code and libraries.
- Poor initialization or data ordering can cause most tokens to select only a few experts, leaving many parameters unused.
- Additional load-balancing objectives encourage the router to distribute tokens across experts instead of concentrating probability on one expert.
- Balanced expert use matters because rarely activated experts consume memory without contributing much training or inference value.
- Total parameters determine how much model weight memory must be stored; active parameters determine much of the per-token computation.
- A model with 30 billion total parameters and 3 billion active parameters, such as the Qwen example discussed, can use compute comparable to a dense 3-billion-parameter model while needing far more memory.
- MoE design also involves choosing expert count and size, active-expert count, and whether a shared expert should run for every token.
- For a single sequence, an MoE model can be faster than a dense model with the same total parameter count because it computes through fewer active parameters.
- At large batch sizes, routing can distribute work across all experts while retaining lower computation per token; at intermediate batch sizes, loading expert weights can reduce the advantage.
- Inference optimization must account for expert weight residency, routing overhead, and frequently used or shared experts; the lecture then turns to data and post-training.
Transcript excerpt
0:03 Okay, let's get started. So today we are going to cover a modification of transformer language models that is basically kind of has become the standard for modern language models and these are called mixture of experts um language models. So this is not a topic I've covered in previous years but since now basically every model is an instance of mixture of experts I thought it would be worthwhile to cover it especially because it's uh an architectural modification. So it nicely connects to the stuff from um last time we were talking about um transformers the giant um multi-layer perceptron or feed
0:55 forward layer that all the parameters live in. Um, so essentially here we're trying to modify that giant feed forward layer where the concatenation of all of the individual head representations go um to add more parameters into our model. And um you know we haven't yet covered scaling laws of language models. So, um it may not be super clear right now why we want to have more parameters, but um let me just write down our topic of so this is what we're covering. Um basically the highle thing that you need
1:44 to know that we will cover in depth in a later lecture is that bigger models plus more data equals lower loss. So, and this obviously equals better next token prediction. So you will just have to take on faith that this is true that bigger models and more data equals more powerful language model. We will get to the actual projections of this and how um you can kind of predict what is going to happen with a certain model size and data size
2:32 um probably next week. Um but for now let's focus on this part which is bigger models and we'll get to the data part
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.