Samsung Electronics: "zHBM Will Open the Door to 1,000 Tokens per Second...The AI-Agent Race Will Be Won on Memory"

[Financial News] Samsung Electronics has unveiled a vision to increase response speeds tenfold in the era of AI agents by using zHBM, which stacks memory in three dimensions (3D) on top of an AI accelerator.
Kim In-dong, an executive responsible for memory product planning at Samsung Electronics North America (DSA), presented the goal of increasing AI system response speeds tenfold at the AI Infrastructure Summit held on the 16th (local time) at the Santa Clara Convention Center in Santa Clara, California.
In his presentation, Kim said, "Interactive AI systems currently respond at about 100 tokens per second per user," adding, "For agent-based AI, we aim to achieve a quantum leap by increasing that figure tenfold to 1,000 tokens per second."
He emphasized that achieving this advance requires moving beyond the 2.5D approach, which places high-bandwidth memory (HBM) alongside an AI accelerator. Instead, he described vertically stacked zHBM, which places HBM directly on top of the accelerator, as the only breakthrough solution.
Kim compared the structure to "having a dedicated elevator in a hotel room that goes directly to the first-floor lobby." He explained that it can dramatically reduce data-movement distances and latency.
As a result, zHBM could deliver up to eight times the performance and more than three times the power efficiency of HBM5.
To address heat generated by the accelerator and transferred to the memory, Samsung Electronics plans to establish a co-design framework with AI accelerator customers from the early design stage and develop customized solutions.
The company also unveiled a roadmap for zNAND-O, a 3D storage solution based on NAND flash memory designed for the on-device AI market.
Kim said, "By 2030, it will become common for AI models with around 1 trillion parameters to run independently in environments such as AI workstations, rather than on large-scale servers." He added, "Building memory for a 1-trillion-parameter model using only DRAM would be extremely costly, but applying zNAND-O would make it possible at one-sixth the cost."
He said Samsung Electronics plans to begin supplying zNAND-O samples in 2028, adding, "We will provide customers with unparalleled value in speed to market, which is central to competition in AI semiconductors."


[email protected] Lee Seok-woo, International Affairs Specialist Reporter