Lecture 7 In-Memory Computing
Overview
Shimeng Yu explains how in-memory computing reduces AI data movement by performing operations where model weights are stored, then compares analog SRAM and resistive-memory designs with newer digital approaches. Analog designs can deliver high efficiency, including a 7 nm SRAM demonstration that normalizes to about 88 TOPS/W for 8-bit MACs, but limited capacity and sensitivity to process, voltage, and temperature constrain their use; digital compute-in-memory and near-data HBM are presented as more practical paths for reliable products and large language models.
Key takeaways
- In-memory computing reduces data transfers by performing multiply-and-accumulate operations where weights reside, but it is most suitable for models small enough to fit in on-chip arrays.
- Nonvolatile RRAM can support power-gated edge inference because weights persist without standby power, while SRAM offers higher performance but continues to leak power.
- A 7 nm SRAM analog compute-in-memory chip reported 350 TOPS/W at 4-bit precision, which normalizes to about 88 TOPS/W for 8-bit MACs; its array capacity was only 4 kilobits.
- Analog compute-in-memory accuracy is vulnerable to process, voltage, and temperature variation, and measured output shifts can cause several percentage points of inference-accuracy loss.
- Digital compute-in-memory uses local SRAM-based multipliers and digital adder trees to avoid analog accuracy variation, accepting greater silicon area in exchange for reliability.
- Trillion-parameter language models exceed SRAM and RRAM capacity, so separate accelerator logic connected or bonded to DRAM-based HBM is a more plausible near-data computing path.
Chapters
- AI workloads evolved from AlexNet and ImageNet classification in 2012 to ChatGPT and large language models, increasing pressure on hardware and data movement.
- GPUs and custom silicon such as Google's TPU use systolic arrays and buffers to reuse weights and partial sums, but still move data between off-chip memory and on-chip storage.
- In-memory computing aims to reduce those transfers by computing directly where data is stored; accelerator efficiency is commonly expressed in TOPS/W.
- SRAM-based compute-in-memory suits high-performance applications and relatively small vision models, while nonvolatile memories such as RRAM, MRAM, and FeRAM can retain weights without standby leakage.
- A power-gated edge inference engine can keep roughly 10 MB of weights in nonvolatile memory, wake when an always-on sensor detects an event, then return to sleep.
- The approach does not scale to trillion-parameter language models: even DRAM struggles with that capacity, so large models require separate accelerators and memory integration such as 3D bonding.
- In an analog array, input activations are applied as voltages and weights are encoded as cell conductances, producing current proportional to the multiply operation.
- Currents from multiple active rows sum along a column, implementing weighted sums in parallel instead of using a conventional digital multiplier and adder for every operation.
- Possible cells include 8-transistor SRAM, 1T1R resistive-memory cells with multilevel conductance, and FETs whose channel conductance stores weight values.
- A TSMC 7 nm SRAM design published at ISSCC 2020 used an 8-transistor decoupled cell and supported 4-bit inputs, weights, and outputs in a 64 by 64 array.
- Its reported 350 TOPS/W figure reflects 4-bit MAC precision; normalizing to 8-bit-by-8-bit MACs requires dividing by four, yielding about 88 TOPS/W.
- Switched-capacitor circuits and a capacitor ladder reconstruct the significance of weight-bit columns, while compensation capacitors address differing RC loads and a flash ADC digitizes column sums.
- The design demonstrated strong efficiency but used only 4 kilobits of SRAM; custom SRAM cells beyond foundry-supported 6T or 8T layouts can also require costly manual design and validation.
- A 22 nm RRAM compute-in-memory example reported about 12 TOPS/W for 8-bit MACs and stored 4 megabits, much more than the 4-kilobit SRAM demonstration.
- RRAM can retain weights during power gating, making it useful for low-leakage edge inference, although its older process node limits performance compared with advanced-node SRAM.
- Analog SRAM and RRAM both face process, voltage, and temperature variation; measurements from a Georgia Tech 14 nm RRAM chip showed output shifts across temperatures and supply conditions despite compensation circuitry.
- Those chip-specific errors can reduce inference accuracy by several percentage points, a serious concern for larger datasets and safety-critical uses such as driving.
- Charge-domain computing encodes weights in capacitance and sums charge rather than current, potentially avoiding the static I²R power consumed by resistive arrays.
- Capacitors still incur dynamic switching energy, and charge-domain designs do not eliminate analog PVT variation because their computation still relies on analog summation.
- TSMC's digital compute-in-memory approach stores weights in SRAM, performs local multiplication with logic gates, and sums partial results through a digital adder tree.
- Digital summation improves reliability and supports advanced nodes: TSMC demonstrated designs at 3 nm and 2 nm, though adder trees cost more silicon and analog approaches can be more efficient.
- SRAM and RRAM cannot hold the training parameters of large language models, making DRAM-based processing-in-memory or near-data computing a more relevant direction.
- Samsung and SK hynix have introduced processing-in-memory products, but integrating compute logic directly on the DRAM die is described as a limitation.
- A proposed alternative places logic on a separate accelerator and bonds it with DRAM; Samsung's Z-HBM and Micron's V-HBM announcements reflect the push toward stacked HBM alongside GPUs or other accelerators.
Transcript excerpt
0:07 So we will briefly talk about the inmemory computing. This is more related to the research that MAB has been working on for the past 10 15 years now. So the motivation very simple right? So the AI right demand the hardware and especially the data movements is a bottleneck in many of those AI workloads. So initially the focus was the the computer vision problem right you know the annexet 2012 the image class imageet
0:57 classification started the machine learning deep learning wave and then 2023 chat GPT came out and then now we have the magic language model and so on. So in terms of hardware platforms of course GPU right was a workhorse all the time and uh we do see a lot of custom silicon these days from all kinds of companies right and one of that earlier pioneer in the custom silicon was Google TPU right tensor processing units So then we still face the memory wall problem
1:47 even in the TPU. It use a sys systoical array architecture where you you know can share the weights buffers and then just have the uh partial sound you know be moved from the labor PE which is a processing element or process engine which is a digital MAC but still then we need to load in the weights from the offd run and then the activation s and then the computed partial sound result needs to be stored back into the global buffer on the chip. So still a lot of data movements ongoing there but it's kind of near memory computing because the data movements
2:35 actually got reduced because of the sharing or caching of those uh partial sum and activations. So the metric that people are interested in is the energy efficiency of the accelerator
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Shimeng Yu.