HBM

Memory Updated Sep 2026

HBM (High Bandwidth Memory) Overview

A conceptual/reference doc on High Bandwidth Memory — physical architecture, packaging, channel structure, generational evolution through HBM4, and how it compares to DDR5 and GDDR6. This rounds out the memory section alongside nand (storage-side flash); HBM is the accelerator-side working memory that sits next to GPU/AI dies rather than on a DIMM. Numbers below are industry-typical figures as of mid-2026 gathered from vendor and JEDEC material — cross-check against a specific datasheet for real analysis, since per-pin speeds and capacities are routinely revised within a generation (e.g. HBM3E, HBM4).

1. What HBM Is

High Bandwidth Memory is DRAM that is manufactured as a vertical stack of thinned dies, connected by through-silicon vias (TSVs), and mounted immediately next to (not on a DIMM across a board trace from) the processor die on a shared silicon interposer. The core idea is the opposite of DDR's approach to bandwidth: rather than pushing pin speed higher on a narrow (64-bit) bus, HBM makes the bus extremely wide (1024 bits per stack, 2048 in HBM4) and keeps pin speed comparatively modest. Wide-and-slow instead of narrow-and-fast is what makes HBM power-efficient per bit transferred, at the cost of capacity, cost per GB, and field-replaceability — all of which is why it is used in GPUs and AI accelerators rather than general-purpose CPUs or client devices.

HBM and DDR are not competitors serving the same slot; they're complementary tiers. See nand for the non-volatile storage tier underneath both.

2. Physical Architecture: The Stack

        ┌───────────────────────┐
        │   DRAM Die (top)      │  ← 8th/12th/16th layer
        ├───────────────────────┤
        │   DRAM Die            │
        ├───────────────────────┤
        │        ...            │  ← thinned DRAM dies, stacked
        ├───────────────────────┤
        │   DRAM Die            │
        ├───────────────────────┤
        │   Base Logic Die      │  ← memory controller I/F, DRFM,
        └───────────┬───────────┘     ECC, power delivery, TSV fan-out
                     │  microbumps + TSVs run vertically
                     │  through every die in the stack
        ┌────────────┴───────────────────────┐
        │         Silicon Interposer          │  ← passive routing layer
        └───┬───────────────────────────┬─────┘
            │                           │
     ┌──────┴──────┐             ┌──────┴──────┐
     │  HBM Stack   │   ...      │  GPU / XPU   │
     └──────────────┘            │     Die      │
                                  └──────────────┘
  • DRAM dies — typically 8–12 thinned dies per stack today (up to 16 in HBM4), each holding a slice of the stack's total capacity.
  • Base logic die — sits at the bottom of the stack (not counted as a DRAM layer) and hosts the memory-controller-facing interface, ECC/DRFM logic, test circuitry, and power delivery. This is the one die in the stack that can be fabricated on a different, more logic-friendly process node than the DRAM dies above it — one of the changes HBM4 leans on (see §6).
  • TSVs (through-silicon vias) — vertical copper-filled vias etched through every DRAM die, giving each die a direct electrical path to the base die and, through it, to the interposer. This is what lets a "wide" bus exist at all: routing 1024+ signal lines out to the side with board traces the way a DIMM does would be physically impossible at this pitch.
  • Microbumps — the fine-pitch solder joints stacking each die onto the one below it, distinct from (and much finer-pitch than) the C4 bumps used at board level.

This is structurally the same layered idea as 3D NAND's vertical stacking for density, but the reason is different: 3D NAND stacks to add capacity per unit area, while HBM stacks to add tens of thousands of parallel electrical connections in a footprint too small for board traces.

3. Packaging: The Interposer and 2.5D Integration

HBM stacks and the processor die sit side-by-side on a shared silicon interposer — a passive slice of silicon (no active transistors, just dense metal routing and its own TSVs) that carries the thousands of fine-pitch traces between the HBM base die and the GPU/accelerator's memory controller. Because both dies sit on top of one shared substrate rather than one being embedded inside the other, this is called 2.5D packaging (as opposed to true 3D packaging, where dies are stacked directly on top of each other).

Two dominant approaches:

Approach Description Used by
Silicon interposer (e.g. TSMC CoWoS-S) A full silicon layer under both the compute die and all HBM stacks; highest routing density, highest cost, reticle-size limited NVIDIA, most large accelerator dies
Silicon bridge (e.g. Intel EMIB) Small silicon bridge chiplets embedded only under the high-speed interfaces, within an otherwise organic substrate; cheaper, no full-interposer reticle limit Intel accelerators, some chiplet designs

TSMC's CoWoS family has itself split into variants as reticle limits became the binding constraint for multi-stack, multi-die packages: CoWoS-S (conventional interposer), CoWoS-R (organic redistribution layer, lower cost), and CoWoS-L (interposer stitching to exceed single-reticle size for the largest multi-HBM-stack accelerators). CoWoS capacity — not wafer starts — has been the practical bottleneck on AI accelerator supply for several years, because every GPU with HBM needs a matching interposer package, not just memory die.

This packaging step is also why HBM cannot be added, removed, or upgraded after manufacturing the way a DIMM can — capacity and bandwidth are fixed at the moment the package is assembled.

4. Channel and Pseudo-Channel Architecture

Where a DDR5 DIMM exposes one (or two, post-DDR5) 32/64-bit channel(s) to the memory controller, a single HBM stack exposes many independent channels simultaneously, each much narrower:

Generation Bus width/stack Channels/stack Channel width Pseudo-channels
HBM1 1024 bits 8 128 bits No
HBM2 / HBM2E 1024 bits 8 128 bits No
HBM3 / HBM3E 1024 bits 16 64 bits Yes — 2 per channel
HBM4 2048 bits 32 64 bits Yes — 2 per channel

Pseudo-channels (introduced in HBM3) split each channel into two half-width sub-channels that can execute independent commands, similar in spirit to DDR5 splitting one 64-bit channel into two 32-bit channels — both changes exist for the same reason: more independently-schedulable, smaller transactions improve efficiency for irregular access patterns instead of only benefiting large sequential transfers. See HBM Memory Explanation.pdf in this project for the equivalent DDR3→DDR5 discussion (bank groups, channel splitting).

Total addressable parallelism scales as stacks × channels × pseudo-channels, which for a GPU with 6–8 HBM stacks means hundreds of independently-schedulable memory channels feeding thousands of compute lanes — the real reason HBM exists, more than any single spec number.

5. Generational Evolution

Generation Introduced Pin speed Bus width/stack Bandwidth/stack Max stack height Typical capacity/stack Voltage (VDDQ)
HBM1 2014 ~1.0 Gb/s 1024 bits ~128 GB/s 4-high 1–4 GB 1.2 V
HBM2 2016 ~2.0–2.4 Gb/s 1024 bits 256 GB/s 4–8-high 4–8 GB 1.2 V
HBM2E 2018–2020 ~3.2–3.6 Gb/s 1024 bits ~410–460 GB/s 8–12-high up to 16–24 GB 1.2 V
HBM3 2022 ~6.4 Gb/s 1024 bits ~819 GB/s 8–12-high up to 24 GB 1.1 V
HBM3E 2023–2024 ~8–9.8 Gb/s 1024 bits ~1.0–1.2 TB/s 8–12-high up to 36 GB 1.1 V
HBM4 2026 (JESD270-4) up to 8 Gb/s 2048 bits up to 2 TB/s 4/8/12/16-high up to 64 GB 0.7–0.9 V (flexible)

The HBM3 → HBM4 jump is the first generation to widen the bus itself (1024→2048 bits, doubling channel count to 32) rather than relying solely on pushing per-pin speed higher — a direct consequence of per-pin speed scaling running into signal-integrity and power limits through the TSV stack and interposer. JEDEC's own framing of HBM4 emphasizes bandwidth and efficiency together, not bandwidth alone: flexible, lower VDDQ options (down to 0.7 V) are meant to keep power/bit from rising in step with bandwidth, since power delivery and heat removal through a stack sitting directly beside a several-hundred-watt compute die is one of the harder unsolved problems at this density.

(Figures above are industry-typical/aggregate from vendor and trade-press material, not a substitute for a specific JESD270-x datasheet — HBM3E in particular shipped at several different qualified speed grades across SK hynix, Samsung, and Micron.)

6. HBM4 in More Detail

HBM4 (JEDEC JESD270-4, finalized 2026) is the most architecturally significant revision since HBM2→HBM3:

  • 2048-bit interface, 32 independent channels (double HBM3's 16) — each with 2 pseudo-channels, for 64 total independently-schedulable channels per stack.
  • Up to 2 TB/s per stack at ~8 Gb/s pin speed, meaning the bandwidth gain comes primarily from width rather than per-pin frequency, unlike every prior HBM generation.
  • 24 Gb / 32 Gb DRAM dies, stackable 4/8/12/16-high, for a maximum single-stack capacity of 64 GB (16-high, 32 Gb dies).
  • Flexible supply voltages — VDDQ selectable at 0.7 V / 0.75 V / 0.8 V / 0.9 V, VDDC at 1.0 V / 1.05 V — letting a system trade bandwidth headroom against power/thermal budget rather than fixing one operating point for the whole product line.
  • Base die decoupled from the DRAM process — HBM4's base logic die is explicitly meant to be built on an advanced logic node (vendors have discussed foundry-partnered logic base dies, e.g. SK hynix/Samsung base dies fabbed at 12nm/10nm-class logic nodes) separate from the DRAM dies stacked on top, which is a meaningful supply-chain shift: the base die increasingly looks like a customer-specific ASIC co-designed with the GPU vendor rather than a stock DRAM-vendor part.
  • DRFM (Directed Refresh Management) — an evolution of the row-hammer mitigation scheme, letting the host direct which specific rows get extra refresh attention rather than refreshing broadly, which matters more as row counts and stack heights grow.
  • Backward-compatible command protocol with HBM3 controllers at the electrical/logical level, though a controller must still be designed for the wider bus to get HBM4's bandwidth.

7. Bandwidth Comparison: HBM vs. GDDR6 vs. DDR5

Memory type Bus width Where it lives Typical bandwidth Typical role
DDR5 (1 channel) 32 bits ×2 sub-channels DIMM, on motherboard, board-trace routed ~35–70 GB/s per DIMM CPU main memory — capacity & flexibility
GDDR6 32 bits per chip Soldered near-die on GPU board ~960 GB/s aggregate (e.g. workstation GPU, ~16–24 chips) Gaming/workstation GPU — cost-effective bandwidth
HBM3 1024 bits per stack 2.5D interposer, next to compute die ~800 GB/s–3.3+ TB/s aggregate (multi-stack) AI/HPC accelerator — maximum bandwidth per die
HBM4 2048 bits per stack 2.5D interposer up to 2 TB/s per stack (multi-TB/s aggregate) Next-gen AI accelerators

The pattern across all three: DDR trades bandwidth for capacity, board-level flexibility, and low cost; GDDR sits in the middle by soldering many narrow, fast chips near the GPU die; HBM abandons board-level flexibility entirely in exchange for the widest possible bus placed as physically close to the compute die as packaging allows. None of the three is "better" in the abstract — each is the right answer for a different capacity/bandwidth/cost/flexibility trade point.

8. Real-World Deployment: Why AI Accelerators Use HBM

Accelerator HBM generation Capacity Bandwidth
NVIDIA H100 SXM5 HBM3 80 GB ~3.35 TB/s
NVIDIA H200 SXM HBM3E 141 GB ~4.9 TB/s
NVIDIA B200 (Blackwell) HBM3E 192 GB ~8 TB/s
AMD MI300X HBM3 192 GB ~5.3 TB/s
NVIDIA Rubin (2026, HBM4 gen) HBM4 ~288 GB multi-TB/s (6.4 GT/s-class parts)

A modern large-model training or inference job is bottlenecked as often by feeding the compute array with weights/activations as by raw FLOPs — thousands of parallel compute lanes need data supplied in lockstep, and DDR5's tens-of-GB/s-per-channel ceiling simply cannot keep up. HBM's hundreds-of-GB/s-to-multi-TB/s per stack, multiplied across the 4–8 stacks typical on a large accelerator die, is what keeps those compute arrays fed. CPUs go the other direction — they need tens to hundreds of GB of flexible, expandable, cheap capacity far more than they need multi-TB/s bandwidth — which is why HBM has stayed almost entirely absent from mainstream CPU designs outside a few HPC-specific parts (e.g. some Xeon Max / Fujitsu A64FX-class chips that used HBM as a large last-level cache in front of DDR).

9. Manufacturing and Reliability Challenges

Challenge Why it's harder for HBM than for planar DRAM
Known-good-die (KGD) testing Every DRAM die and the base die must be tested and graded before stacking, because one bad die discovered after an 8–16-high stack is bonded scraps the entire (very expensive) stack. This drives heavy investment in pre-bond and mid-bond test methodology.
TSV yield & alignment Thousands of TSVs per die must align to microbump pitch across every layer; a single TSV defect or misalignment can break a signal or power path through the whole stack.
Thermal density The stack sits immediately beside a multi-hundred-watt compute die, and the DRAM dies in the middle of the stack are the hardest to cool — heat has to move laterally through silicon and TSVs, not just up through a heatsink, which is part of the motivation for HBM4's lower/flexible operating voltages.
Row hammer at scale More rows, taller stacks, and tighter cell pitch make row-hammer-style disturbance more likely; DRFM (see §6) is JEDEC's answer, directing refresh attention rather than refreshing everything on a fixed schedule.
Package-level yield coupling Because the GPU/accelerator die, interposer, and every HBM stack are assembled into one package before final test, a defect anywhere (including in a stack that tested good standalone) can fail the whole multi-thousand-dollar package — this is part of why CoWoS/interposer capacity, not raw wafer starts, has been the binding constraint on AI accelerator supply.

10. Current-Generation Snapshot (2026)

As of mid-2026, HBM production is effectively a duopoly with a smaller third player:

Vendor Approx. HBM share Status
SK hynix ~50–60% HBM4 development completed first (2025); HBM4 production at 12nm-class base-die process; HBM3E still the higher-volume product through most of 2026
Samsung Remainder of the ~90% combined with SK hynix HBM4 production beginning 2026 at 10nm-class base-die process; reported rapid yield improvement on HBM4 through 2026
Micron Minor but growing share Shipping HBM4 to key customers; HBM capacity reported sold out through 2026–2027

HBM3E remains the volume product through most of 2026, with HBM4 ramping through the year to match next-generation accelerators (e.g. NVIDIA's Rubin platform) that require it. Because every HBM stack has to be matched with interposer/CoWoS packaging capacity and a compatible compute die, the ramp of any HBM generation is gated as much by packaging and co-design schedules as by DRAM fab output.

11. Where HBM Fits in the Broader Memory/Storage Picture

   Compute die (GPU/XPU)
        │  (multi-TB/s, ns-scale latency)
   HBM stacks  ← this doc
        │
   DDR5 system memory  ← HBM Memory Explanation.pdf (DDR3/4/5 evolution)
        │  (tens of GB/s, tens-of-ns latency, large/flexible capacity)
   NAND-based SSD (persistent)  ← [nand](https://globaltrendanalysis.com/nand/), [ssd-controller-architecture](https://globaltrendanalysis.com/ssd-controller-architecture/)
        │  (GB/s-class, µs-to-ms latency, non-volatile)

HBM, DDR, and NAND flash sit at three different points on the same capacity/bandwidth/latency/volatility trade curve, each optimized for a different job: HBM feeds a compute array that would otherwise starve; DDR gives a CPU large, flexible, reasonably fast working memory; NAND gives the system large, cheap, non-volatile capacity behind both. None of the three is on a path to replace either of the others — the interesting engineering (and the reason all three show up together in a modern AI system) is in how the tiers are managed together, which is really a controller/software problem as much as a memory-technology one.

Sources

Contact

Have an Idea?
Let's Build It.

Whether you're a startup with a hardware concept, an established company needing deep design help, or an enthusiast with a wild project — we want to hear about it.

Emailaglucas@globaltrendanalysis.com
Phone(303) 921-4492
LocationBoulder, Colorado