TodayToday's edition
Researchers at ETH Zürich and NYU tested 36 real memory chips of the kind used in AI processors. They found the industry's standard worst-case test misses how much power the chips can really draw.
2 min read
Stacked memory beside a processor die.
Illustration (AI-generated)
HBM, or high bandwidth memory, is fast memory stacked right next to AI processors. Researchers at ETH Zürich and New York University tested real HBM2 chips, an older generation of it. They found the chips can draw 23.6% more power on average, and up to 35.4% more, than the most demanding test in the JEDEC standard. JEDEC is the industry body that sets memory specifications.
The team calls this the first detailed power study of real HBM2 parts. It covered 36 chips on 18 AMD Alveo U55C FPGA boards, which are reprogrammable chip boards used for testing. Semiconductor Engineering flagged the paper.
HBM stacks memory dies right next to the processor.
VoltGridNews
Source · Original VoltGridNews explanatory diagram based on the cited source
The standard tests ignore some details that change power use. Reading data costs far more than anything else the team tested. It drew about 3.1 times the current of refresh, the routine work memory does to keep data from fading. In older flat DDR3L memory, it was the other way round.
Where the data sits matters too. Memory is split into banks, grouped into bank groups. Power varied by 8.1% between bank groups and by 5.3% between banks.
The data itself also matters. Inside the chip, signals on internal wires called bank group buses flip between 0 and 1. Data patterns that made those signals flip a lot raised power by 29.4%, compared with keeping them steady. Combining the hungriest banks with the worst pattern produced the peak result.
Measured HBM2 power effects from the ETH Zürich and NYU study.
VoltGridNews
Source · Original VoltGridNews diagram; data from cited source
This is about power budgets. The authors measured newer HBM3E memory using about 248 W on average across three NVIDIA H200 GPUs. That is roughly 35% of the card's 700 W budget.
System designers plan power supply and cooling around expected peak use. If they rely only on JEDEC or vendor tests, the paper warns, memory-heavy AI training or inference jobs could cause problems. Those include thermal throttling, where chips slow down to avoid overheating, as well as shutdowns or power delivery failures.
The group also released a free, open source model for predicting HBM power, built on an existing tool called DRAMPower. Its average error against measured chips was 7.6% to 8.1%. That included a test based on running the Llama 3.1 8B AI model. Existing models were off by 20% to 42%. Checked against the H200 cards, it matched memory power during heavy reading to within 2.4% to 5.3%.
Some questions remain. The tested chips are likely made by Samsung, but the maker could not be confirmed. The tests ran at 1,200 MT/s, or million transfers per second, below the board's top speed. The figures for the coming HBM4 generation are estimates from the model, not measurements. The authors call for tests on real HBM4 chips.
