Hot Chips 2026 started this past Sunday (8/23) and they had the memory vendors give talks about what they’re working on. I broke down their lectures into key headlines, quotes, and the associated slides that were presented.
Quick lead-in prior to the bulk of the post.
One of the first things that’s apparent going through the presentations is that memory has evolved from an accessory to becoming much of the system itself. In fact, literally ~90% of silicon in a GPU package is memory. AI’s demands such as trillion-parameter models and exploding KV cache put on a tremendous squeeze such that the memory wall is widening -- as you’ll see starting with Micron’s talk, one of their slides points out that compute is tripling every two years while HBM bandwidth only doubles.
Next is the base die itself turning into logic, which Samsung brought up, where you build the HBM base die on an advanced logic node and you pull the memory controller and compute onto it. Otherwise called custom HBM, which Jung-ho Kim had mentioned before and I had referenced in a prior setting, link below.
Samsung’s talk makes the case for custom HBM, and Kim, in his interview, made the case that custom HBM ends typical memory cycles because the memory makers effectively become system-semiconductor companies.
The importance of thermal and packaging are apparent as well, and seem like the real bottlenecks going forward. As HBM gains whether via power, density, or layers, this shows up as heat. Current thinking from the SK hynix team is hybrid bonding + dedicated hotspot heat blocks. What’s happening is that HBM keeps getting faster (pin speed) and taller (more layers) and is boxed in at a fixed height and by heat, so this is where hybrid bonding comes into play.
What we’re left with is the three primary memory makers working off the same diagnosis but with different game plans. We know memory is a bottleneck, heat is an issue, and the base die is becoming logic. They’re all doing separate things to try to get to the same place. It’s game on. Hope the post below is helpful.
Raghu Sreeramaneni · Micron
Compute scales about 3x every two years while HBM bandwidth only doubles, so the memory wall keeps widening
“Compute is scaling at roughly 3x every two years. HBM, while it has solved a lot of the bandwidth issues and scaled up bandwidth, still is lagging and is more like 2x every two years. So the memory wall is still present and, in fact, maybe getting worse.”
90% of the silicon in a GPU package is memory, so a single bad DRAM die can threaten the entire system
“So if you take a typical GPU that is maybe a reticle field dimension and has four HBM supporting it, 12-high, in this cartoon, there’s two of those GPUs together. The memory silicon is actually eight times higher than the GPU silicon in that one SIP package. So 90% of the silicon in that one GPU SIP or system and package is related to the memory.
And the pressure from that is, for DRAM, given the complex integration, if one DRAM die goes bad or is not reliable, the entire SIP has a problem with reliability. So the expectations of RAS we’ll talk about later is compounded, just given the scale and the challenge around HBM.”
