Bernstein Comments on Seven Memory Types: After HBM, DRAM and NAND Compete for the Next Trillion-Dollar Market

TL;DR · The demand for storage in AI goes beyond HBM. Training requires a complete memory system from HBM, system DRAM to SSD and shared storage. · The real bottleneck in inference occurs during the decoding phase. KV Cache will grow with the context length and the number of concurrent users, and memory capacity may limit commercialization sooner than model weights. · Agents further amplify storage pressure. Multi-step calls repeatedly generate context, invoke external tools, and increase the demand for CPU, DRAM, and KV Cache. · Multiple new tiers are emerging between HBM and traditional SSDs, including CXL, "Storage Next," and CMX, all aiming to handle the expanding inference data at a lower cost. · Not all new technology routes can be realized. HBF, zHBM, NVHBM, ZAM, and PIM each face issues such as heat dissipation, yield, ecological compatibility, or redistribution of supply chain interests. · Bernstein remains optimistic about Samsung Electronics, SK Hynix, Micron, SanDisk, Seagate, and Western Digital, maintaining a "underperform" rating for Kioxia.

For the past two years, the market narrative around AI storage has largely revolved around HBM. However, as large models transition from training to large-scale inference, simply increasing HBM is no longer sufficient to solve all problems.

In its latest global storage report, Bernstein points out that different AI workloads have significantly different memory requirements: training emphasizes computing power and bandwidth, while the decoding phase in inference relies more on capacity, RAG requires large-scale databases, and Agent workflows simultaneously increase the burden on both traditional and AI servers.

This means that the changes brought by AI are being transmitted from HBM down to system DRAM, SSD, HDD, and even tape storage. Around the "memory wall," the industry chain is beginning to insert new products between existing tiers, hoping to find a new balance between performance, capacity, and cost.

The upper limit of inference scale may depend on KV Cache

During the training phase of large models, GPUs and HBM remain at the core.

Training requires frequent reading of model parameters and intermediate data, with high demands for computing power and memory bandwidth. However, the training of a large model does not solely rely on HBM: the raw dataset needs to be stored in lower-cost storage media; before data enters the GPU, it usually needs to be cached and preprocessed by system DRAM and local SSD; training that lasts for weeks or even months also requires regular checkpoint saves to avoid starting over due to hardware or software failures.

Therefore, a large training session actually calls upon a complete system from HBM, system DRAM to local SSD and network storage.

Once entering the inference phase, memory demands further diversify.

Inference can typically be split into two stages: prefill and decode. Prefill is responsible for processing user input and generating the first token, mainly performing large-scale matrix operations, leaning more towards "compute-bound." In this phase, GPU utilization and HBM bandwidth are more critical, with common metrics being first token latency.

The decoding phase is different. The model needs to generate tokens one by one and call upon previously generated information when generating new tokens. To avoid redundant calculations, the system typically stores this data in KV Cache.

KV Cache has two important characteristics: its capacity increases linearly with context length, and each user requires an independent cache. Therefore, as context lengthens and the number of concurrent users increases, both factors will jointly drive up memory usage.

Bernstein believes that in large-scale AI deployments, the memory occupied by KV Cache may exceed model weights, becoming the main factor limiting the number of concurrent users and context windows. How many users the model can serve and how long the context can be maintained will ultimately directly impact revenue scale.

From this perspective, the competitive focus in the inference era is not just on how much computation chips can perform, but also on how low the system can cost-effectively store and read the ever-expanding context.

RAG and Agent push demand towards traditional memory

The popularity of RAG and Agents has further spread AI storage demand beyond HBM.

RAG mainly includes two stages: database construction and database retrieval. During database construction, the system needs to process a large amount of unstructured data such as PDFs, web pages, and code, and then convert it into searchable vectors and indexes. This process relies more on large-capacity SSDs and system DRAM, with HBM's role being relatively limited.

Once the database is established, user queries are first converted into vectors, which are then matched with the content in the database. Vector generation can be quickly completed in GPUs and HBM, but the actual search typically relies more on system DRAM. The retrieval results are then merged with user questions, entering the normal prefill and decoding process.

The burden brought by Agent workflows is even heavier.

Traditional dialogue is usually a single call of "input---model---output," while Agents need to break down the target into multiple steps, invoking other models or external tools, saving intermediate results, and replanning based on feedback. The results generated from each call may also become the input for the next model call.

This simultaneously increases two types of demand: on one hand, tool calls and non-AI tasks require more CPU and system memory; on the other hand, the continuous transfer of context between multiple models will rapidly expand the burdens of prefill, decoding, and KV Cache.

Therefore, the development of Agent applications does not only benefit GPUs and HBM, but may also drive demand for server DRAM, enterprise-grade SSDs, and lower-cost storage media.

New memory tiers are emerging between HBM and SSD

The memory system of traditional servers can generally be divided into processor internal caches, system DRAM, local SSDs, and shared storage. AI servers have added HBM to this structure, but HBM has limited capacity and high costs, making it difficult to handle all data.

The current solution in the industry chain is to introduce new products between different tiers.

CXL attempts to integrate physically dispersed memory into a shared resource pool, allowing CPUs, GPUs, and expansion devices to more flexibly access DRAM. Some products also use DRAM or SRAM as cache, combined with NAND, to reduce costs while shortening access latency.

"Storage Next," driven by NVIDIA, aims to shift some storage management from CPU to GPU, allowing NAND to achieve latency, IOPS, and data access granularity closer to DRAM. Kioxia's GP series SSD, based on XL-FLASH, is a representative of this direction.

CMX mainly targets KV Cache. It deploys SSDs in independent data nodes, connecting to computing nodes via DPU, Ethernet, and switching chips. Its goal is to share inference context between different GPUs, reduce redundant storage, and break through the memory capacity limits of a single server.

These solutions point to the same trend: AI systems cannot keep all active data in HBM for long periods; they need to distribute it across different tiers according to data access frequency and latency requirements.

Hot data stays in HBM, some context shifts to system DRAM or high-performance SSDs, while colder data continues to sink to ordinary SSDs, HDDs, or even tapes. The finer the memory tier, the more likely the system is to achieve a balance between performance and cost.

New technologies are emerging densely, but commercialization still has uncertainties

Around the "memory wall," the industry chain has proposed multiple new routes.

Samsung's zHBM plan stacks HBM above the processor to further shorten data transmission distances. However, this design needs to handle the heat generated by GPUs while also raising higher requirements for the yield and cost of wafer-level hybrid bonding.

NVIDIA's NVHBM hands over the basic bare chips to NVIDIA for design, which may be manufactured by TSMC. This solution is expected to reduce power consumption and increase bandwidth, but it may also weaken the design and manufacturing value of storage manufacturers for HBM bare chips. As product standardization occurs, some added value may shift from storage manufacturers to NVIDIA and wafer foundries.

SanDisk and SK Hynix's HBF aims to leverage NAND to provide bandwidth close to HBM while achieving greater capacity and lower unit costs. However, there are still significant gaps in latency and performance between NAND and DRAM, and HBF needs to overcome multiple technical levels, making its implementation challenging.

Intel's ZAM attempts to rotate DRAM bare chips 90 degrees to improve heat dissipation, aiming for practical application by fiscal year 2029; Qualcomm's HBC uses LPDDR and traditional packaging to bypass CoWoS costs at the expense of some performance.

Additionally, PIM attempts to directly integrate computing capabilities into storage chips to reduce data movement between processors and memory. However, this would change the existing computing architecture, requiring processors, software, and networks to adapt together, and would impact the already highly mature division of labor between logic chips and storage chips. Bernstein believes its industry adoption remains limited.

From this perspective, the rapid increase in the number of new solutions does not mean that all routes can form a large-scale market. Whether they can be compatible with existing software and hardware ecosystems, whether they have cost advantages, and whether all parties in the supply chain can reach a balance of interests will determine the final commercialization outcome.

AI storage beneficiaries will not only be HBM manufacturers

From an investment perspective, Bernstein's judgment is quite clear: AI's pull on the storage industry is spreading from a few high-end products to more tiers.

HBM remains the core for training and high-performance inference, and Samsung Electronics, SK Hynix, and Micron will continue to benefit from high-bandwidth storage demand. However, as the scale of inference expands, the importance of system DRAM and NAND will rise. KV Cache overflow, RAG databases, and the large amounts of intermediate data generated by Agents will also increase demand for SSDs and shared storage.

Colder data will continue to sink. Bernstein states that the data growth brought by AI has begun to benefit HDDs; in some scenarios, due to insufficient capacity of NAND and HDD, the demand for tapes, traditionally used mainly for archiving, is also increasing.

The report continues to give "outperform" ratings to Samsung Electronics, SK Hynix, Micron, SanDisk, Seagate, and Western Digital. Among them, Samsung Electronics, SK Hynix, and Micron correspond to DRAM and HBM, SanDisk benefits from NAND and HBF, while Seagate and Western Digital correspond to lower-cost, high-capacity storage. Kioxia is rated as "underperform."

However, the core value of this report does not lie in listing a batch of new technology abbreviations, but in redefining the boundaries of the AI storage market.

The bottleneck of the training era mainly focuses on GPUs and HBM; in the inference and Agent era, the bottleneck begins to spread along the entire memory system. The future competition for AI infrastructure will depend not only on how fast chips can compute but also on whether data can flow efficiently at sufficiently low costs between HBM, DRAM, NAND, and shared storage.