EfficientML.ai Lecture 13 - LLM Deployment Techniques (MIT 6.5940 Fall 2026)
Watch on YouTube →
Overview
LLM inference is constrained by model-weight traffic, activation outliers, KV-cache growth, and inefficient request scheduling, so deployment systems combine quantization, sparsity, and serving optimizations. SmoothQuant and AWQ preserve accuracy at low precision, TinyChat and QServe turn compressed models into faster inference, while Wanda, Deja Vu, MoE, PagedAttention, continuous batching, FlashAttention, and speculative decoding reduce computation or memory costs.
Key takeaways
- SmoothQuant addresses LLM activation outliers by shifting their scale into corresponding weights, preserving linear-layer outputs and folding the adjustment into earlier layers so inference needs no extra smoothing kernel.
- AWQ uses activation magnitude, not weight magnitude alone, to find sensitive channels; preserving or scaling a small fraction of these channels can recover most of the perplexity lost to low-bit quantization.
- W4A4 is not automatically faster than W8A8: repeated per-group conversions and rescaling can overwhelm the savings, so QServe delays floating-point conversion and keeps accumulation on INT8 tensor cores.
- PagedAttention replaces large contiguous KV-cache reservations with fixed-size blocks and page tables, reducing fragmentation and supporting cache sharing across requests with a common prompt.
- Continuous batching improves utilization by changing the request group after every decoding step, while static batches make new requests wait for the longest-running sequence.
- FlashAttention and speculative decoding reduce different costs: FlashAttention avoids writing the full attention matrix to HBM, while speculative decoding verifies multiple proposed tokens in one target-model pass.
Chapters
- Quantization uses fewer bits per value to reduce model memory and memory traffic, but lower precision must preserve model capability.
- W8A8 can track FP16 on smaller models yet lose substantial accuracy around 6.7 billion parameters.
- A few large activation outliers force shared INT8 scales to cover a wide range, collapsing many small values onto the same quantization level.
- SmoothQuant identifies consistently large activation channels using calibration data, then scales activations down and corresponding weights up without changing the linear-layer output.
- The per-channel scaling can be folded into the previous layer, avoiding an extra inference kernel.
- A calibration-based balance, commonly using alpha = 0.5, makes activations easier to quantize while keeping adjusted weights manageable in INT8.
- Single-request decoding is often memory-bandwidth-bound: W4A16 reduces weight traffic, while larger batches can benefit from W8A8 INT8 tensor-core throughput.
- AWQ uses groupwise weight quantization and activation magnitudes to identify important channels; protecting just 1% of channels can bring 3-bit, group-size-128 perplexity from about 40 to about 13.
- Instead of retaining mixed-precision channels, AWQ scales important weights and inversely scales their input channels; a factor of 1.5 brought perplexity from about 43 to about 14.5.
- A small calibration-set grid search selects the scaling strength, making AWQ comparatively simple, hardware-friendly, and less dependent on calibration data than regression-based approaches.
- TinyChat is a Python-native inference engine designed for cloud, desktop, laptop, edge GPU, and CPU deployment.
- Its ARM CPU kernels rearrange packed 4-bit weights offline so inference can use shifts and masks instead of extra on-the-fly rearrangement.
- Kernel fusion combines quantization and dequantization with attention and matrix-multiplication operations; TinyChat reports roughly 3x speedups over Hugging Face and AutoGPTQ baselines.
- QServe targets W4A8KV4: 4-bit weights, 8-bit activations, and a 4-bit KV cache.
- Naive W4A4 can be slower than W8A8 because per-group conversions and rescaling add CUDA-core work inside matrix-multiplication loops.
- QServe applies group scales while converting INT4 weights to INT8, then uses INT8 tensor cores and delays floating-point output conversion until accumulation ends.
- SmoothAttention scales key-cache outlier channels; reported WikiText results improve on comparison methods, with QServe reaching roughly 2 to 3 times the speed of TensorRT-LLM baselines.
- Magnitude pruning zeros the smallest weights, but a small weight can still matter when multiplied by a large input activation.
- Wanda scores each weight using its absolute magnitude times the L2 norm of its corresponding input-channel activations, measured on calibration data.
- The activation-aware score can preserve a weight of magnitude 1 over a weight of magnitude 3 when its input-channel contribution is larger, outperforming magnitude-only pruning.
- Deja Vu predicts which attention heads and MLP neurons a token needs, using earlier hidden states to overlap prediction with inference.
- Experiments found accuracy could remain nearly unchanged with about 7 times fewer active parameters per token.
- Mixture-of-Experts replaces one feed-forward network with multiple experts and routes each token to a subset, increasing total parameter count without proportionally increasing per-token computation.
- Top-K token-to-expert routing is common, but uneven token assignment can create load imbalance between experts.
- SPAtten progressively prunes low-importance tokens and attention heads using accumulated attention scores, reducing work in later layers.
- H2O retains both a recent token window and heavy hitters, the tokens with the largest accumulated attention scores during decoding.
- Eviction methods such as H2O discard KV entries, while query-aware methods such as Quest keep the full cache available and select relevant entries for each query.
- Paged, fixed-size KV blocks, commonly 16 tokens each, make sparse attention selection more practical for modern GPUs.
- Time to first token (TTFT) includes prompt prefilling and queue delay; time per output token (TPOT) measures the average interval between generated tokens.
- For an output of n tokens, end-to-end latency is approximately TTFT plus TPOT multiplied by n minus one.
- Batching can raise aggregate tokens per second while making each user's tokens arrive more slowly, so low TTFT, low TPOT, and high throughput can conflict.
- Serving benchmarks should specify input and output lengths, model precision, architecture, parallelism, and GPU configuration.
- Autoregressive decoding reuses cached keys and values for previous tokens, computing a new query, key, and value for each next token.
- For an 80-layer MHA model with 64 KV heads, head dimension 128, and FP16 cache, the KV cache is about 2.5 MB per token, or about 10 GB for 4,000 tokens.
- Reserving a maximum-length contiguous cache wastes memory through internal and external fragmentation.
- PagedAttention, used in vLLM, maps logical token blocks to a shared pool of physical blocks, limiting unused capacity mostly to the final partially filled block and enabling prefix-cache sharing.
- Static batching processes a fixed group together, while dynamic batching uses a maximum batch size or short timer to limit arrival delays.
- Both fixed-group approaches can leave GPU capacity idle when short requests finish before long ones.
- Continuous batching updates the active batch after each decoding step, immediately admitting waiting requests as completed requests leave.
- Paged KV blocks support this schedule by allowing cache memory released by finished requests to be reassigned.
- Standard prompt attention creates an n-by-n score matrix and can write and reread large intermediate tensors from high-bandwidth memory.
- FlashAttention processes tiles of Q, K, and V in fast on-chip SRAM, updating outputs and softmax normalization without storing the full score matrix in HBM.
- FlashAttention 2 changes loop organization to reduce output-memory traffic; later versions introduce producer-consumer scheduling and hardware-specific optimization.
- A small draft model proposes several tokens, and the large target model verifies the proposed sequence in one forward pass.
- The acceptance test proceeds left to right; after the first rejection, later drafts are discarded and the target model samples a replacement token.
- When draft generation is cheap and several proposals are accepted, speculative decoding reduces repeated large-model weight reads without changing the target distribution.
- Drafts can also come from parallel methods such as the diffusion-based approach mentioned as dFlash.
Transcript excerpt
0:00 All right, let's get started. And hi everyone, I'm Jamming Tong and today uh I will be giving the lecture on the large language model deployment techniques. And in this lecture we will look uh at the main bottlenecks in the large language model inference uh and explore the techniques for reducing the latency improving the throughut and using the uh GPU memory more efficiently. And today we will cover three main topics. First is the quantization and um we'll look at how using the field bits to represent the numbers that can reduce the memory usage and make the inference more efficient and we will we will cover the weight and activation quantization
0:49 uh smooth quant and the weight only quantization with AWQ and its deployment in tiny chat. We'll also look at the QER which combines the quantization of the weights activation and the KV cache. Second uh is the printing and the sparity and the idea is to reduce the amount of data or the computation in the uh inference. We will explore the uh three forms of sparity uh the sparity in the model weights, the contextual sparity and the uh sparity in attention and finally the large link model serving system. We will begin with the matrix we use to evaluate serving performance and discuss the uh page attention in VRM the flash attention speculative coding and
1:37 the continuous batching and these techniques help us to manage the KV cache and schedule multiple request more efficiently. So let's start with the quantization and the basic idea of the quantization is to use fewer bits to represent each value. So this can reduce the memory usage and the memory traffic. So it can also speed up the inference and computation. And for large language models, the challenge
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.