This article continues (and concludes) the discussion on the proceedings of DPC-4, covering the remaining four contestants and a summary of the trends observed in all eight prefetchers presented in the championship. Similar to Part I, we focus on how each prefetch algorithm functions, and why it is effective. Finer implementation details can be obtained from the workshop papers or the source code.

BertiGO (Simranjit Singh, University of Murcia; Agustín Navarro Torres, University of Zaragoza; Alberto Ros (University of Murcia

Motivation

When evaluating the baseline prefetcher configuration, the authors noted that Berti frequently issues redundant prefetch requests for lines already prefetched or present in the cache. Also, using only the PC provides very limited context for pattern recognition, limiting the prediction capabilities of Berti. Furthermore, Pythia is found to generate a lot of useless prefetches for some workloads, which pollutes the L2 cache and wastes memory bandwidth.

Idea

1. A Region-Based Bit-Map Filter is added, which is a fully associative structure storing the prefetched and accessed cache lines per region, in the form of a bit-vector. For regions tracked by the filter, having the M-th bit set in the bitmap implies that we drop all prefetch requests for the M-th cache line inside the region.
2. In addition to using PC, the authors propose using a hash (shifted XOR) of the last 4 PCs with the current PC, to index the Berti tables with additional context.
3. Set-Dueling is added to Pythia: instead of using the default policy to issue prefetches, 5 different policies are introduced, including a No-Prefetch policy that disables Pythia. All 5 policies are enabled for a 10M-instruction tournament, at the end of which the policy with the lowest miss rate is chosen for the rest of execution.
4. An Adaptive Next Line (ANeLin) is added to the LLC, which uses a sampling cache to track the demand misses and insert next-line prefetches. A heuristic mechanism is used to track useful and useless prefetches globally and per-PC. ANeLin can be disabled if the ratio of useful to useless prefetches drops below a threshold.

Why It Works

Adding a Bit-Map Filter eliminates redundant and useless prefetches. Using PC history adds context from the program flow while learning memory accesses with minimal overhead. Disabling Pythia and Next Line prefetching when they do not generate enough useful prefetches solves the problem of cache pollution due to wasteful prefetching. This is especially useful in the constrained bandwidth and multicore scenarios where data and memory need to be shared judiciously for optimal performance.

Entangling Data Prefetcher (Agustín Navarro Torres, Universidad de Zaragoza;  Simranjit Singh, University of Murcia; Biswabandan Panda, IIT Bombay; Alberto Ros, University of Murcia)

Motivation

Comparing Berti with other state-of-art prefetchers, the authors identify a SPEC2017 workload where Berti achieves negligible performance gain over no-prefetch baseline. Profiling this trace reveals that it consists of long-reuse strides (stride accesses separated by 2K-cycle interval) and zero-strides (consecutive accesses to the same cache line). Berti cannot issue zero-delta prefetches, and even though prefetches are correctly issued for long-reuse deltas, they get evicted before the cache line gets accessed. T-SKID, a Time Skipping Prefetcher is built on top of a standard PC-Stride prefetcher, but decouples the PC that triggers a prefetch (TriggerPC) from the PC that trains the predictor(TargetPC). This allows it to prefetch long-reuse and zero stride patterns. However, the underlying stride prefetcher limits its scope to constant stride instead of complex delta patterns predicted easily by Berti.

Idea

EDP is proposed as a VA-based L1D prefetcher. It gets trained and triggered on cache misses or prefetch hits (cache hit on a prefetched line). For every TargetPC, it records the fill latency of the demand access or prefetch request. It then searches the global PC history for the most recent PC that was observed more than (current cycle – fill latency) cycles ago – this is the TriggerPC which could have triggered a timely prefetch for TargetPC. This ‘Entangling Pair’ of PCs is added to the Entangling Table, that stores the set of TargetPCs for a given TriggerPC. EDP also looks at the address history of each TargetPC to calculate the list of timely deltas (similar to Berti) and stores them with the current address in a Delta Table indexed by TargetPC. To issue prefetches, the TriggerPC is used to obtain one or more TargetPC, which are used to obtain address and deltas for timely prefetch. The prefetches calculated in this way are passed through a Bloom Filter to drop redundant requests, and then placed in a Proxy Prefetch Queue (PPQ) where the prefetch request waits till slots open up in the demand read queue. If there is no space in the latter, prefetch requests are not issued. Pythia is implemented at L2, with a throttling mechanism at LLC that tracks each core’s requests and sets the EDP aggressiveness.

Why It Works

Using a different PC to trigger prefetches allows EDP to successfully prefetch zero and long reuse delta patterns for its target PC. Filtering out redundant prefetches reduces contention for resources. Using a dedicated PPQ for prefetch requests prevents prefetches from competing with critical loads for resources. The LLC throttling mechanism helps evenly distribute resources in the multi-core scenario.

Composite Prefetching with Bandits (Charles Block, Pedro Palacios, Abraham Farrell, Gerasimos Gerogiannis, Josep Torrellas, University of Illinois at Urbana-Champaign)

Motivation

The authors point out that the current state-of-the-art prefetchers try to optimize low-level metrics such as accuracy, timeliness and coverage. The system performance (IPC) depends on these factors, but can have variable sensitivity to each of them depending on the workload and program phase. Furthermore, a single prefetcher is generally insufficient to deliver the best performance for a diverse set of workloads – industrial processors generally deploy a composite prefetcher consisting of multiple prefetch engines.

Idea

A Multi-Armed Bandit is a Reinforcement Learning agent that chooses the best action (arm) to maximize the reward function value. Inspired by this, a Micro-Armed Bandit (MAB) is used to prefetch at L2C. Each ‘arm’ consists of different configurations for 5 state-of-the-art prefetchers-

• Next Line, Spatial Memory Streaming, Best Offset Prefetcher: Can be turned ON or OFF
• Stride, Stream prefetchers: Degree can be tuned to control aggressiveness

A bloom filter is implemented to prevent issuing redundant prefetches. Each arm is used for a fixed time period (bandit step) after which the reward generated by it is evaluated by the agent. This is evaluated against the rewards generated previously to calculate which arm to use next. The total IPC of the core is used as a reward function for the MAB.

To optimize multi-core performance, another agent called ‘µMama’ is added at the system level, using the geometric mean of IPCs across all cores as a reward function. At each timestep, it decides whether to allow the cores to pursue their independent actions, or to force them into joint actions which have a record of increasing the µMama reward.

Why It Works

Using Reinforcement Learning to directly maximize the system performance ensures that the prefetcher dynamically re-configures itself with execution to improve IPC. The caveat is that this now becomes a search space problem – the arms of the bandit need to be diverse enough to support different kinds of workloads, in order to deliver the best performance.

Global Berti (Gilead Posluns, Mark Jeffrey; University of Toronto)

Motivation

Berti is a state-of-the-art prefetcher that detects Streaming patterns, i.e., consistent delta values between accesses by the same PC. Practical workloads however, often exhibit Spatial patterns identified by consistent delta values between accesses by different PCs. In the absence of streaming patterns, prefetching based on spatial patterns could alleviate the efficacy of Berti.

Idea

Global Berti detects spatial patterns using Berti’s existing structures – the History Table conventionally stores within a row, the addresses of all the lines accessed by a particular PC, in FIFO order. When a streaming pattern cannot be detected, local training is useless and Global Berti looks at the most recent address for all PCs to detect spatial patterns (global training). Berti’s Delta Table holds the row delta values for the same PC; Global Berti stores the global deltas (across PCs) in the same table, adding a local bit to differentiate between streaming and spatial training.

Why It Works

By itself, Berti is quite effective at detecting and covering streaming patterns. Adding the capability to detect spatial patterns in the absence of streaming patterns increases Global Berti’s coverage and therefore, the overall performance. As expected, the highest speedup over Berti is obtained in SPEC2017 and Graph workloads that are dominated by irregular accesses which require spatial prefetching. On the other hand, AI workloads containing mostly streaming patterns see a much lesser speedup.

General Trends

Although the major focus of almost all DPC-4 submissions is to overcome the limitations of the high-performing Berti/Pythia baseline, they highlight several key trends in data prefetching research:

• Prefetching across Physical Page Boundaries: Issuing page-crossing prefetches is extremely useful for AI workloads since they are dominated by streaming accesses. This is leveraged by most submissions to gain an edge over the baseline prefetcher configuration.
• Preventing Redundant Prefetches: Quite a few papers also combat excessive prefetching and resource contention through advanced throttling, priority, and filtering mechanisms.
• Increased System-Level and Multi-Core Awareness: There is a growing emphasis on system-aware solutions to judiciously manage shared resources like memory bandwidth, which is constrained in high-core-count datacenters. This includes core-level fairness throttling (Emender, EDP) and global coordination agents (µMama) to dynamically adjust prefetcher configurations for optimal multi-core performance.
• Expanding Pattern Coverage for Diverse Workloads: Submissions seek to improve coverage beyond simple streaming patterns. This includes detecting spatial patterns across different PCs (Global Berti), and targeting complex patterns like long-reuse and zero-strides (EDP). The adoption of PC history (BertiGO) also provides better context for pattern recognition.
• Shift Towards Adaptive and Composite Designs: Recognizing that a  single prefetcher is insufficient for diverse workloads, the trend moves toward composite prefetchers. This is accompanied by dynamic re-configuration to select the best prefetcher setting at runtime, and adaptive heuristics to tune aggressiveness.

About the Author

Digvijay Singh obtained his Bachelor’s degree from BITS Pilani and his Master’s degree from Texas A&M University where he worked on data prefetching as part of the CAMSIN research group. He currently works as a Silicon Architect in Google’s mobile CPU team.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

This article is the first in a two-part series that summarizes the key contributions of 4th Data Prefetching Championship (DPC-4), held in conjunction with the 32nd iteration of HPCA in 2026. While discussing innovative data prefetching techniques presented in this contest, we focus on the functionality of proposed algorithms and also explain why they are effective. Finer implementation details can be found from the papers or the source code.

Implementation Constraints

All prefetchers are evaluated against a baseline configuration that employs: Berti prefetcher (DPC3 winner) at L1D (Level-1 Data cache) and Pythia prefetcher at L2 (Level-2 cache). While there were no constraint on design complexity, upper limits were defined on the storage budget of the prefetchers to ensure the design was practically feasible for implementation. These limits were defined as follows: L1D Prefetcher: 32KB, L2 Prefetcher: 128KB, LLC (Last Level Cache) Prefetcher: 256KB.

Keynotes

The event included two keynote talks. The first keynote, titled “Is Prefetcher Research Still Alive?”, was given by Leeor Peled from Huawei. Leeor discussed the modern relevance of prefetching research, offering a pragmatic philosophy for academic researchers. He argued that the primary objective should not necessarily be to surpass “best-in-class” models – which are often the result of years of ‘engineered’ fine-tuning – but rather to introduce novel, high-potential concepts that invite further optimization. He emphasized that while an individual effort might not immediately surpass the state-of-the-art, a sufficiently “interesting” technique can evolve into a transformative solution through subsequent community-driven iteration.

He suggested two optimizations that can be explored:

1. Building a Semantic Prefetcher that correlates memory accesses with address generating code, i.e., a high-precision version of the Runahead Prefetcher that selectively runs only the code responsible for generating a future address.
2. Training neural networks to identify deep correlations between memory accesses, potentially unlocking the ability to predict complex, non-linear patterns that remain invisible to current heuristic-based logic.

The following issues can (and should) be addressed to build better prefetchers:

• Generalizing complex patterns, e.g. pointer chasing loads
• Accurately choosing memory access with high correlation for better training
• Prefetching to the appropriate cache level to optimize for timeliness
• Throttling prefetches for fairness amongst multiple cores
• Using LLMs to process memory traces instead of text sequences

The second keynote, titled “Data Prefetching: A Datacenter Perspective”, was given by Akanksha J. from Google.  Addressing the memory bottleneck problem in modern datacenters (40% of the CPU cycles are spent idling for memory responses) Akanksha highlighted that cloud environments are characterized by massive multi-threading and incessant context switching. In these scenarios, a single thread may migrate across multiple cores, while each core rotates through a vast “plethora” of applications. The Google workloads utilized in DPC-4 are a better representation of this reality, and are primarily frontend-bound. Without a sophisticated instruction prefetcher to streamline code delivery, the underlying bottlenecks in data prefetching remain obscured and impossible to solve. She also analyzed structural failures of current prefetching solutions, identifying these primary aspects:

1. Current design philosophy focuses on “tuning for the common case,” resulting in hard-coded heuristic values—such as fixed confidence thresholds and prefetch degrees—that are taped out into non-programmable silicon. While these “black boxes” are meticulously engineered to squeeze every drop of performance from SPEC workloads, they lack the flexibility required for the high heterogeneity of datacenter tasks. Consequently, these resource-hungry techniques often penalize cloud performance rather than enhancing it.
2. If we disable hardware prefetchers entirely and rely on software to insert prefetches, we miss out on critical opportunities to utilize valuable information about system states (coherence, timeliness, cache hits/misses) that improves prefetching. Akanksha proposed a shift towards “Software-Defined Prefetching,” a paradigm that transcends current ISA limitations. In this model, the software layer dynamically selects which code segments to target and determines the optimal hardware prefetcher to activate for peak accuracy. Simultaneously, the hardware leverages real-time system state data to maximize coverage.

Furthermore, Akanksha advocated for evaluating all prefetching techniques within constrained-bandwidth environments, arguing that such stress tests better reflect the realities of modern compute environments.

Now, on to prefetcher designs themselves.

Virtual Inter-Page Prefetcher (Ho Je Lee, Won Woo Ro; Yonsei University)

Motivation

• Analyzing the baseline prefetecher configuration, the authors observed that the L2 Prefetcher (Pythia) is more effective than the L1 Prefetcher (Berti) in reducing Misses Per Kilo Instructions (MPKI) for the Last Level Cache (LLC).
• Since Pythia operates in the Physical Address (PA) space, it is not feasible to let it issue prefetches across page boundaries, as incorrect physical page access poses a security risk.
• A roofline study shows that there is significant performance to be gained when Pythia is allowed to issue page-cross prefetches in the PA space. This advantage amplifies when it is granted visibility of the Virtual Address (VA) space, preventing incorrect page accesses.

Idea

VIP is implemented at L1 level,  but issues prefetches to the L2. It gets trained on L1 Misses by reading the {PC, VA} information off the packets sent to L1 MSHR. These are written to the VIP Stride Table that calculates the observed stride for a particular PC and stores it. If a stride value is repeated, the confidence gets incremented. Otherwise it gets reset. The confidence value determines the prefetch degree.

Why It Works

The implemented VIP configuration is a simple yet elegant solution to gain performance over the baseline by supplementing the existing Berti and Pythia prefetchers with cross-page prefetches (note that the DPC-3 version of Berti operates in the PA space and cannot issue prefetches across page boundaries). As expected, the stride prefetcher boosts AI workloads with sequential accesses of large data structures that span across pages. The typical CPU workloads such as SPEC see a moderate gain; the control-flow dominated Google workloads have a marginal slowdown since they rarely have uninterrupted streams.

Signature Pattern Prediction and Access-Map Prefetcher (Maccoy Merrell, Lei Wang, Paul Gratz, Stavros Kalafatis; Texas A&M University)

Motivation

Access Map Pattern Matching (AMPM) and Signature Path Prefetching (SPP) are both considered state-of-the-art prefetching techniques; while SPP is sensitive to the order of memory accesses, AMPM is resistant to OoO execution. However, AMPM relies heavily on stored patterns for each region and  is unable to issue prefetches for new regions or when the observed accesses deviate from expectations. SPP excels at this and can even make predictions from its issued prefetches.

Idea

Implemented at L2 level, a Region Table (RT) tracks all access maps (as bit-vectors) on a per-region basis. Upon a memory access, an N-bit portion from the respective access map is used to index a Pattern Table (PT). The PT outputs the most frequently occurring N-bit pattern as a prefetch candidate, which can be used to speculatively index the PT. Similar to SPP, speculative prefetching continues till the overall confidence drops below a threshold. The RT access map indicates the recently accessed cache lines and filters out redundant prefetches.

Why It Works

The authors have identified the complementary nature of SPP and AMPM, and have combined them effectively to utilize the OoO resistance of AMPM with the Speculative mechanism of SPP. Additionally, numerous throttling mechanisms are implemented which consider pattern usefulness as well as global metrics such as DRAM bandwidth and overall usefulness to drop prefetches and set prefetch degree. SPPAM is implemented at L2C with Berti (the MICRO version which operates in the VA space) at L1D and Bingo at LLC. Similar to the previous paper, the cross-page stream information is passed to SPPAM from L1D.

Emender (Jiajie Chen, Tingji Zhang, Xiaoyi Liu, Xuefeng Zhang, Peng Qu, Youhui Zhang; Tsinghua University)

Motivation

An evaluation of different combinations of state-of-the-art prefetchers shows that VBerti (L1D) and Pythia (L2) is the highest performing combination. Here, VBerti refers to the MICRO version of Berti that operates in the VA space, allowing it to issue page-crossing prefetches. It is observed that this optimal prefetcher combination issues too many prefetch requests that fill the prefetch queue quickly, which leads to useful prefetches getting dropped. A second-order effect of a full prefetch queue is the excessive usage of L1D to Memory bandwidth that can delay critical loads.

Idea

Four key features are added to tackle the problem of over-prefetching in the VBerti+Pythia configuration:

1. Pending Target Buffer is added to sort all issued prefetches by confidence, which helps prioritize useful prefetches between different PCs.
2. Cuckoo Filter is added which tracks the VAs already present in the cache to prevent redundant prefetches. This structure is chosen due to its O(1) query time, high accuracy and zero false negatives.
3. Dynamic Confidence Threshold is added which increases with the cache miss rate, throttling low-confidence prefetches.
4. A Fairness-based Throttling scheme is implemented across cores, which tracks the useless prefetches per-core at L3 and stops the core with the most useless prefetches from prefetching.

Why It Works

The authors identify problematic areas in the baseline Berti+Pythia system and propose features to effectively address them. The best performance improvement comes from the Cuckoo Filter for single-core and Fairness Throttling for multi-core configuration. Since Emender provides the least gain for limited bandwidth configuration, it would be interesting to look at the accuracy data.

sBerti (Jiapeng Zhou, Ben Chen, Kunlin Li, Yun Chen; HKUST, Guangzhou)

Motivation

When profiling the DPC4 workloads on the given baseline prefetcher configuration (Berti + Pythia), the authors observed a high L1D miss rate in the AI-ML and Google workloads. A deeper analysis of the traces indicated that most of these misses occurred when the access stream moved across the 4KB physical page boundary, which happens frequently in these workloads. The version of Berti used in the baseline does not issue prefetches across page boundaries, and thus, a stride prefetcher can help.

Idea

A decoupled Smart Stride Prefetcher is added at L1D, which operates on the VA space and can  therefore track memory access streams across page boundaries. It is trained using a Smart Stride Table (SST), which is indexed by a hash of the PC, and subtracts the lastVA from the current VA to calculate the delta value. If the absolute value of delta is a multiple of the stored stride, the confidence is updated; this also provides resistance to out-of-order execution. Prefetches are issued if this confidence is greater than a static threshold. The lookahead is tuned via a heuristic which is incremented upon observing late prefetches and decremented by timely prefetches. A Recent Prefetch Table stores the recently issued prefetches to track their timeliness and filter duplicate prefetches between Berti and Smart Stride engines.

Why It Works

The addition of a decoupled stride prefetcher gives sBerti the ability to issue prefetches across physical page boundaries, reducing the “Cold-start Penalty” of Berti. The heuristic based dynamic distance adjustment helps tune the aggressiveness at runtime, allowing longer lookahead for AI-ML workloads dominated by streaming accesses. The final sBerti configuration (Stride + Berti at L1D, Pythia at L2) delivers the best performance in a full bandwidth scenario, where the stride engine can prefetch further ahead.

We will overview the rest of the prefetchers in part 2 of this post.

About the Author

Digvijay Singh received his Bachelor’s degree from BITS Pilani and his Master’s degree from Texas A&M University where he worked on data prefetching as part of the CAMSIN research group. He currently works as a Silicon Architect in Google’s mobile CPU team.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Hybrid continuous-discrete-variable (CV-DV) quantum computing combines oscillators and qubits to tackle problems that are difficult for either model alone, from bosonic simulation to quantum error correction. At ASPLOS 2026, our tutorial introduced the foundations, compilation stack, benchmarking methods, and programming tools behind this emerging architecture model. In this blog post, we overview the key elements of our tutorial. 

Tutorial website: https://cvdv.ncsu.edu/resources/asplos-tutorial/

Foundations

We began with the foundations of hybrid CV-DV quantum computing, introducing the physical model, mathematical language, and programming abstractions behind qubit-oscillator systems. Many leading quantum platforms naturally combine qubits with oscillator modes, such as cavities, vibrational modes, or photonic fields. Rather than treating oscillators as auxiliary hardware, hybrid CV-DV computing views their large Hilbert spaces as a computational resource.

The tutorial covered core representations of CV states in both Fock space and phase space, along with the key operators and gate families that support universal CV-DV computation. A central message was that hybrid systems are not simply “qubits plus extra hardware,” but a distinct computational model with their own instruction sets, abstractions, and compilation challenges. We showed how familiar qubit concepts such as Pauli and Clifford structure extend into the oscillator setting through displacement operations, squeezing, quadratic Hamiltonians, beamsplitters, and controlled hybrid interactions.

We also discussed why this matters from a computer architecture perspective. Hybrid CV-DV systems introduce new instruction set architectures (ISAs), abstract machine models (AMMs), and compilation choices that help separate hardware details from software design. Depending on the platform and compiler stack, the same computation may be expressed in phase-space language, Fock-space language, or a mixed qubit-oscillator representation.

To ground these ideas, we highlighted emerging algorithmic primitives and applications where hybrid systems may offer advantages, including oscillator-mediated entangling gates, state-transfer protocols, Hamiltonian simulation, bosonic quantum error correction, vibronic dynamics, and sensing. We closed the session by surveying two leading implementation pathways, superconducting circuit QED and trapped-ion systems, and discussing the distinct control and connectivity tradeoffs they expose. A comprehensive tutorial on the foundations of hybrid CV-DV quantum processors is available here.

Compilation

We also presented Strategies and Tools to Compile CV-DV Quantum Circuits. We began by emphasizing why Hamiltonian simulation is a central application and one of the most promising directions for hybrid continuous-variable and discrete-variable (CV-DV) quantum systems. CV systems can naturally represent continuous degrees of freedom, while DV systems provide strong control and interaction structures. Together, they enable important applications in areas such as quantum chemistry and materials science. However, a key challenge lies in decomposing the time-evolution operator e^{-iHt} into a sequence of executable quantum gates. This transformation is fundamentally a compilation problem, bridging high-level quantum algorithms and low-level hardware. As such, compilers play a critical role in hybrid quantum systems.

We then focused on the dominant approach today: symbolic compilation. In particular, we discussed two early CV-DV Hamiltonian simulation compilers from Chen et al., ISCA’25 and Decker et al., QCE’25. The core idea is to avoid direct matrix-based computation and instead leverage the algebraic structure of operators for rule-based decomposition. Techniques such as Trotter-Suzuki product formulas, the Baker–Campbell–Hausdorff (BCH) expansion, and bosonic commutation relations are used to gradually break down complex Hamiltonians into hardware-executable primitive gates. This process is typically implemented through rule matching and recursive rewriting, where expressions are repeatedly transformed until only supported base gates remain. While this approach avoids the exponential blowup of high-dimensional matrices, it introduces tradeoffs between approximation error and resource overhead.

Finally, we analyzed the limitations of current compilers and outlined future research directions. Key challenges include limited gate sets and decomposition rules, the tradeoff between accuracy and resource cost, hardware connectivity constraints, and insufficient optimization flexibility. To address these issues, we highlighted the need for improved programmability, richer native gate support, more accurate cost models, and optimizations that exploit algebraic properties such as commutativity. We also presented the Genesis compiler from Chen et al., ISCA’25 as an end-to-end solution example, including typical use cases and code snippets. Genesis employs a multi-level intermediate representation (IR) and a full compilation pipeline to automatically translate Hamiltonians into limited hardware connectivity physical circuits, demonstrating a systematic and extensible compilation framework for hybrid CV-DV quantum computing.

Benchmark and Circuit Simulator

We also presented HyQBench by Mohapatra et al., an open-source benchmark suite implemented in Bosonic Qiskit and QuTiP. HyQBench covers eight representative hybrid circuits spanning three abstraction levels: primitives, algorithms, and applications. These include cat state generation, GKP state preparation, CV-to-DV state transfer, CV-DV QFT, CV-DV VQE, CV-QAOA, Jaynes-Cummings-Hubbard (JCH) Hamiltonian simulation, and Shor’s algorithm.

One key takeaway is that hybrid architectures can reduce hardware resources dramatically for some workloads. For example, simulating a 3-site JCH model in a DV-only encoding requires 9 qubits and 393 CNOT gates, whereas a hybrid implementation uses only 3 qumodes, 3 qubits, and 8 gates. This kind of reduction highlights why benchmarking hybrid systems requires more than simply counting qubits.

To support this, we introduced a feature map tailored to hybrid systems. In addition to standard structural metrics such as gate counts, circuit depth, and qubit/qumode counts, we proposed three CV-DV-specific metrics: Wigner negativity as a proxy for non-classicality and classical simulation hardness, truncation cost to quantify population near the Fock cutoff, and maximum energy. These metrics help separate workloads with very different simulation and execution behavior. For example, JCH simulation remains relatively close to Gaussian behavior, while CV-QAOA and Shor’s algorithm exhibit higher Wigner negativity and are harder to simulate classically.

We also discussed early hardware validation. A cat-state preparation benchmark was executed on Sandia National Laboratories’ QSCOUT trapped-ion platform and achieved a fidelity of 0.71. HyQBench was further used to calibrate conditional displacement gates on the same platform, reinforcing the need for standardized benchmark suites that support both evaluation and device calibration. The full paper is available at https://arxiv.org/abs/2603.04398.

To lower the barrier to entry for this area, we also developed HyQSim, a browser-based hybrid CV-DV circuit simulator that requires no installation. HyQSim supports drag-and-drop circuit construction, arbitrary Fock cutoffs, and built-in visualization through Wigner plots, Fock-state amplitudes, and Bloch sphere views. It is available at https://cvdv.ncsu.edu/resources/simulator/, and the code is hosted at https://github.com/shubdeepmohapatra01/HyQSim/.

Programming

Finally, we discussed programming support for hybrid CV-DV systems. Quantum programming languages and frameworks have developed many important ideas over the years, including linear quantum types for enforcing the no-cloning theorem, automatic uncomputation of ancilla qubits, and dynamic lifting of classical variables for mid-circuit measurement. Hybrid quantum computing introduces an additional requirement: heterogeneous quantum registers containing both qubits and qumodes.

To address this challenge, we developed Hybridlane, a CV-DV quantum programming framework built on PennyLane. By extending PennyLane, Hybridlane inherits a broad library of qubit algorithms, gates, and compilation routines while remaining familiar to existing users. Hybridlane tracks wire types automatically through symbolic circuit analysis and type inference, enabling scalable circuit construction, platform independence, and integration with downstream compilation flows.

The tutorial concluded with example workflows using Hybridlane. In one example, we reused an existing PennyLane quantum phase estimation template for a CV-DV Hamiltonian simulation and then lowered it through symbolic compilation to a gate sequence executable on the Bosonic Qiskit backend. In another, we demonstrated a cross-platform workflow in which a conditional displacement gate was calibrated in simulation and then compiled for execution on Sandia’s QSCOUT trapped-ion platform. Together, these examples showed how hybrid quantum software can begin to support the same define-simulate-execute workflow that has become standard in mature qubit SDKs.

We hope Hybridlane helps enable a broader ecosystem of reusable software and research for hybrid quantum computing. It is available at https://github.com/pnnl/hybridlane.

Closing

Hybrid CV-DV computing sits at the intersection of quantum hardware, computer architecture, compilation, and programming systems. We hope this tutorial helps make the area more accessible to researchers across architecture, systems, programming languages, and quantum information, and we invite readers to explore the tutorial materials, benchmarks, and tools linked above.

About the Authors

Yuan Liu is an Assistant Professor of Electrical & Computer Engineering and Computer Science at North Carolina State University. Prior to joining the NC State faculty, he was a postdoctoral researcher at the Massachusetts Institute of Technology. His research interests lie at the intersection of quantum computing, quantum engineering, quantum algorithms/architectures and applications.

Zihan Chen is a Ph.D. student in computer systems at Rutgers University, advised by Prof. Eddy Z. Zhang. His research focuses on compiler and system-level techniques, as well as parallel computing, to enhance the efficiency, programmability, scalability, and fault tolerance of emerging quantum computing systems.

Shubdeep Mohapatra is a Ph.D. candidate in Computer Engineering at NC State University, advised by Prof. Huiyang Zhou and Prof. Yuan Liu. His research focuses on quantum error characterization, mitigation, and benchmarking, aimed at improving the reliability and fault tolerance of near-term quantum computing systems.

Jim Furches is a post-masters research associate at Pacific Northwest National Laboratory. His current research interests are in quantum benchmarking, algorithms, and quantum programming and compilation.

Zheng (Eddy) Zhang is a Professor in the Department of Computer Science at Rutgers University. Her research focuses on full-stack compiler and programming systems for quantum computing. She studies how to better coordinate quantum applications, programming languages, intermediate representations, compilation, pulse-level control, and hardware architecture to improve the performance, usability, and scalability of quantum systems.

Huiyang Zhou is a Professor of Electrical and Computer Engineering at North Carolina State University. His current research interests include GPU architecture, processor security, and quantum computing.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Years ago, I came across three pioneering works (CSI-NN, Cache Telepathy, and DeepSniffer) in the field of reverse engineering neural networks that inspired my journey into side-channel attacks to uncover the secrets of modern Deep Neural Networks (DNNs). Fast forward to today, and there has been significant exploitation of side-channel attacks to discover the secrets of neural networks. It’s a good time to provide an overview of where we stand, the outlook for the future, and the challenges ahead.

Motivation: Let’s take a step back and first try to understand why we care about secrets in deep learning models. It basically boils down to two fundamental challenges associated with deep learning: i) Financial, ii) Security and Privacy challenges. In general, DNNs are intellectual property (IP), as they are products developed over years of research, implementation, and investment in computing units, and they entail significant training costs (time, energy, and labor), making them a valuable asset for their owners. Just to give a rough estimate, OpenAI’s GPT-4 costs more than $ 100 million, and its GPT-5 model is expected to be more than 5x as expensive (Cost of Training GPT). I do not know about you, but if I spent 100 million on something, I would care about protecting it. The next challenge is knowing that a model secret gives an adversary white-box knowledge, which is extremely powerful in security and privacy settings. Any adversary with knowledge of a target victim’s model architecture (e.g., model type, layer sequence, and number) and weight information, formally defined as “white-box,” can launch powerful security (adversarial attacks) and privacy threats (model inversion attacks/membership inference attacks). As highlighted in Figure 1, the attacker’s final objective in the DNN reverse-engineering attack is to gain white-box privileges either to steal IP for financial gain or to launch subsequent attacks.

In summary, in security and privacy research, defining the threat model is the first step towards any exploitation, and the underlying assumption is often that a reverse-engineering attack has successfully uncovered the model architecture, weights, and other hyperparameters.

Attack Objectives: By now, we have established that an attacker’s goal is to uncover two key properties of a victim DNN: its architecture and its parameters. However, this is an oversimplified goal and can often be misleading. To understand this, let’s consider a deep neural network as a function of x, denoted f(x).  If an attacker wants to recover the exact victim model, their objective is for the stolen model to be identical to the original f(x), which is practically impossible for large-scale DNNs, whether using existing side-channel attacks or the exact victim dataset. As a result, a more practical and plausible goal for an attacker would be to achieve functional equivalence. If the stolen function is different, such as g(x), then, for incentive purposes, all an attacker cares about is that these two functions produce identical output,  i.e., f(x)= g(x), for inputs x that are of the attacker’s interest. As a result, achieving functional equivalence means recovering the DNN model architecture, often as close as possible to the victim architecture’s topology. On the weight side, even if an attacker cannot extract the exact weights, they must aim for a weight-space solution that captures the victim model’s functionality.

In summary, to steal a copy of the victim model/function, an attacker must identify the victim model architecture. In modern deep learning, where most practical applications use some version of a DNN model from an existing pool (e.g., GPT, Llama), recovering the architecture often boils down to detecting the model’s topology. Once the architecture is revealed, the attacker must recover the model parameters/weights, which is often a challenging part of the attack. Then again, as we discussed earlier, exact model recovery can be challenging, but achieving functional equivalence is a modest objective. Most importantly, to achieve functional equivalence, the attacker may not need to reveal the exact numerical weights; rather, gradually recovering coarse-grained information (e.g., weight sparsity, quantization pattern, weight distribution) is often sufficient.

Figure 1: Spectrum of attack threats characterized by attacker’s knowledge: Black-Box (No Knowledge), Grey-Box (Partial Knowledge, e.g., architecture), and White-box (Complete knowledge of model architecture and weights), the ultimate goal of reverse-engineering (AI-generated).

Attack Techniques and Capabilities. Among the popular types of side-channel attacks, i.e., physical and microarchitectural, both can be utilized in two different threat model settings. In edge or embedded devices, the physical side channel is the dominant threat, and several works (CSI-NN, BarraCUDA) have shown that it is possible to recover the model architecture and weights of simple neural networks. On the other hand, micro-architectural side channels are a popular choice for resource-sharing cloud environments where users can upload and run their code in a colocated environment (e.g., Amazon SageMaker and Google ML Engine). Microarchitectural attacks have been successful in recovering model architecture across the board using cache timing channels, memory access patterns, and GPU context switching. I acknowledge that there are many ways to recover DNN model weights, including learning-based approaches and mathematical recovery techniques. In this blog post, I focus on side-channel attacks. At the same time, learning-based approaches can work as a complementary approach with side-channel attacks once the architecture information has already been leaked. 

In summary, while side-channel attacks have been successful in leaking model architecture information, as the scale of modern DNNs, e.g., LLM weights, continues to reach new heights of billions, none of the existing side channels can scalably and predictably recover model parameter information. A common workaround would be to support these methods with a learning approach, assuming an attacker has a partial training set, which may not be practical, even in a resource-sharing environment where data remains private.

Future Challenges and Opportunities: 

What is the future of architecture-recovery attacks, given the success of existing side channels?

As the next wave of vision and language domain architectures emerges, they present new challenges and opportunities for the microarchitectural side-channel attack community. These models require modern compute support, which can accelerate their inference (e.g., tensor cores), as GPUs become more modern and newer generations may leave new traces of side-channel information. Hence, these newer compute platforms (e.g., new GPUs) and their associated architectural support demand new innovation in side-channel capabilities to recover the model architecture. We must remember that architecture recovery is essential; without it, model parameter recovery is no longer useful. Moreover, as LLMs emerge as the dominant model, the question is not just about recovering weights or architecture; leaking other components, such as KV cache in a multi-tenant setting, can lead to privacy leakage. 

Can a microarchitectural side channel alone ever be sufficient to recover model weight information? 

The sheer scale of the modern model poses an even greater challenge for recovering weights, making direct recovery an ambitious, and even impossible, goal; instead, we should focus on functional equivalence. To achieve functional equivalence, weight recovery methods can set tiny stepping stones to augment learning-based recovery. 

Complete weight recovery using a side channel at the scale of LLMs or even a smaller vision model may be too ambitious. Instead, the attacks should focus on coarse-grained information about weights, such as model sparsity levels, quantization mechanisms, weight sign recovery, and other optimization techniques. The key idea is to achieve functional equivalence by first recovering coarse-grained information, which is sufficient to support other learning-based recovery. It is time to work towards an achievable target: recovering this statistical weight-level knowledge and studying how critical their role is in improving subsequent attacks. As models and their computation units are increasingly optimized, leaking information such as sparsity levels or bit-widths will become more feasible by detecting optimized paths through side-channel leakage.

Finally, an attack is never the end goal. We probe attacks from every angle so we can study them before any attacker ever thinks about them. The endgame is always to develop subsequent defenses, which I leave for another discussion.

About the author:

Adnan Siraj Rakin is an Assistant Professor at the School of Computing at Binghamton University. He received his Master’s (2021) and PhD (2022) from Arizona State University. He works on emerging security and privacy challenges in modern AI systems and algorithms. His paper on DNN model weight recovery has been crowned as Top Picks in Hardware and Embedded Security in 2024. 

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

The debate of sparsity versus quantization has made its rounds in the ML optimization community for many years. Now, with the Generative AI revolution, the debate is intensifying. While these might both seem like simple mathematical approximations to an AI researcher, for a hardware architect, they present fundamentally different sets of challenges. Many architects in the AI hardware space are deeply familiar with watching the scale tip from one side to the other, constantly searching for a pragmatic balance. Let’s look at both techniques, unpack the architectural challenges they introduce, and explore whether a “best of both worlds” scenario is truly possible (Spoiler: It depends).

Note: We will only be looking at compute-bound workloads, which traditionally rely on dense compute units such as tensor cores or MXUs. We will set aside memory-bound workloads for now, as they introduce their own distinct set of tradeoffs for sparsity and quantization.

Sparsity

The core idea of sparsity is beautifully simple: if a neural network weight is zero (or close enough to it), just don’t do the math. Theoretically, pruning can save massive amounts of compute and memory bandwidth.

The Architecture Challenge: The Chaos of Unstructured Data

The golden goose of this approach is fine-grained, unstructured sparsity. It offers a high level of achievable compression through pruning, but results in a completely random distribution of zero elements. Traditional dense hardware hates this. Randomness leads to irregular memory accesses, unpredictable load balancing across cores, and terrible cache utilization. High-performance SIMD units end up starving while the memory controller plays hopscotch trying to fetch the next non-zero value. To architect around this, pioneering unstructured sparse accelerators—such as EIE and SCNN—had to rely heavily on complex routing logic, specialized crossbars, and deep queues just to keep the compute units fed, often trading compute area for routing overhead.

The Compromise: Structured and Coarse-Grained Sparsity

To tame this chaos, the industry shifted toward structured compromises. The universally embraced N:M sparsity (popularized by NVIDIA’s Ampere architecture) forces exactly N non-zero elements in every block of M. This provides a predictable load-balancing mechanism where the hardware can perfectly schedule memory fetches and compute.

More recently, to tackle the quadratic memory bottleneck of long-context LLMs, we’ve seen a surge in modern sparse attention mechanisms that leverage block sparsity. Techniques like Block-Sparse Attention and Routing Attention enforce sparsity at the chunk or tile level. Instead of picking individual tokens, they route computation to contiguous blocks of tokens, allowing standard dense matrix multiplication engines to skip entire chunks while maintaining high MXU utilizations and contiguous memory access. Other approaches, like StreamingLLM, evict older tokens entirely, retaining only local context and specific “heavy hitter” sink tokens.

The trade-off across these methods is clear: we exchange theoretical maximum efficiency for hardware-friendly predictability, paying a “tax” in metadata storage (index matrices), specialized multiplexing logic, and the persistent algorithmic risk of dropping contextually vital information.

Quantization

While sparsity aims to compute less, quantization aims to compute smaller. Shrinking datatypes from 32-bit floats (FP32) to INT8, or embracing emerging standards like the OCP Microscaling Formats (MX) Specification (such as MXFP8 E4M3 and E5M2), acts as an immediate multiplier for memory bandwidth and capacity. But the frontier has pushed much further than 8-bit. Recent advancements in extreme quantization, such as BitNet b1.58 (1-bit LLMs using ternary weights of {-1, 0, 1}) and 2-bit quantization schemes (like GPTQ or Quip), demonstrate that large language models can maintain remarkable accuracy even when weights are squeezed to their absolute theoretical limits.

The Architecture Challenge: The Tyranny of Metadata and Scaling Factors

From an architecture perspective, the challenge of extreme quantization isn’t just the math—it’s the metadata. To maintain accuracy at 4-bit, 2-bit, or sub-integer levels, algorithms demand fine-grained control, requiring per-channel, per-group, or even per-token dynamic scaling factors. Every time we shrink the primary datapath, the relative hardware overhead of managing these scaling factors skyrockets. Along with that, the quantization algorithm also becomes more fine grained, dynamic and complex. We are forced to add additional logic and even high-precision accumulators (often FP16 or FP32) just to handle the on-the-fly de-quantization and accumulation. We aggressively optimize the MAC (Multiply-Accumulate) units, only to trade that for the overhead of adding scaling factor handling and supporting a potentially new dynamic quantization scheme, which can outweigh the gains.

The Compromise: Algorithmic Offloading

To fix this without blowing up the complexity and area budget, the community relies on algorithmic co-design. Techniques like SmoothQuant effectively migrate the quantization difficulty offline, mathematically shifting the dynamic range from spiky, hard-to-predict activations into the statically known weights. Similarly, AWQ (Activation-aware Weight Quantization) identifies and protects a small fraction of “salient” weights to maintain accuracy without requiring complex, dynamic mixed-precision hardware pipelines. By absorbing the complexity into offline mathematics, these techniques allow the hardware to run mostly uniform, low-precision datatypes.

However, much like the routing tax in sparsity, this algorithmic offloading comes with some compromises. These methods heavily rely on static, offline calibration datasets. If a model encounters out-of-distribution data in production (a different language, an unusual coding syntax, or an unexpected prompt structure), the statically determined scaling factors can fail, leading to outlier clipping and catastrophic accuracy collapse. Furthermore, relying on offline preprocessing creates a rigid deployment pipeline that prevents the model from adapting to extreme activation spikes on the fly.

Is there a “best of both worlds”?

So, knowing these trade-offs, do we sparsify or do we quantize? Many years ago, the Deep Compression paper proved we could do both. But today, pulling this off at the scale of a 70-billion parameter LLM is incredibly difficult. It suffers from the classic hardware optimization catch-22 (see All in on Matmul?) : No one uses a new piece of hardware because it’s not supported by software, and it’s not supported by software because no one’s using it.

So what’s the path forward for hardware architects? In my opinion, the following:

• Deep Hardware-Software Co-design: The days of throwing a generic matrix-multiplication engine at a model are over. We need to work directly with AI researchers so that when they design a new pruning threshold or a novel sub-byte data type, the hardware already has a streamlined, fast path for the metadata.
• Generalized Compression Abstractions: Historically, we have designed accelerators that are either “good at sparsity” (with complex routing networks) or “good at quantization” (with mixed-precision MACs). Moving forward, we need to view these not as orthogonal features, but as a unified spectrum of compression. Architectures must be designed to dynamically adapt—perhaps fluidly dropping structurally sparse blocks during a memory-bound decode phase, while leaning on extreme sub-byte quantization during a compute-heavy prefill phase—potentially even sharing the same underlying logic.
• Balance Efficiency and Programmability: As explored in the “All in on MatMul?” post, we need to keep our hardware flexible. Over-fitting to today’s specific sparsity pattern or quantization trick risks building being trapped in the local minimum. We must maintain enough programmability to enable future algorithm discovery and break free from the catch-22.

Some notable research going along this path include Effective interplay between sparsity and quantization, which proves the non-orthogonality of the two techniques and explores the interplay between them and also the Compression Trinity work which takes a look at multiple techniques across sparsity, quantization and low rank approximation and tries to take a holistic view of the optimization space across the stack.

Ultimately, as alluded to before, there is no single silver bullet, and like all open architecture problems, the answer is always “it depends”.  But in the era of Generative AI, it depends on whether we view sparsity and quantization as competing alternatives or as pieces of the same puzzle. Perhaps it’s time we stop asking which one is better, and start designing architectures flexible enough to embrace the realities of both.

About the Author:

Sai Srivatsa Bhamidipati is a Senior Silicon Architect at Google working on the Google Tensor TPU in the Pixel phones. His primary focus is on efficient and scalable compute for Generative AI on the Tensor TPU.

Authors’ Disclaimer:

Portions of this post were edited with the assistance of AI models. Some references, notes and images were also compiled using AI tools. The content represents the opinions of the authors and does not necessarily represent the views, policies, or positions of Google or its affiliates.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

As we close the book on 2025, Computer Architecture Today has seen another successful year of community engagement. We published 29 posts covering a wide spectrum of topics—from datacenter energy-efficiency to the evolving debate on LLMs in peer review, alongside trip reports from our major conferences. I want to thank all our authors for their insights, with special appreciation for those who contributed multiple times.

Over the last year, we shifted our editorial model, moving from a roster of set contributors to a more flexible, open-submission approach. We also re-established our conference trip reports, highlighting top architecture venues.

The blog thrives on new voices, and our door is always open. We are actively looking for:

• New Ideas: If you have a topic in mind, please propose it using this link or email me directly.

• Trip Reports: Planning to attend a conference? Volunteer to share your experience.

• Event Summaries: Organizers of workshops or tutorials are welcome to publicize their events through summary posts.

• Industry Perspectives: We would like to hear from our industry colleagues about their take on the future landscape of computer architecture.

Finally, as AI tools proliferate, the conversation around their role in our paper reviewing process is far from over. I look forward to seeing more of that debate here.

Here’s to the new advances in Computer Architecture in 2026!

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Large language model (LLM) agents are quickly moving from “single agent” to *multi-agent systems*: tool-using agents, planner-orchestrator, debate teams, specialized sub-agents that collaborate to solve tasks. At the same time, the *context* these agents must operate within is becoming more complex: longer histories, multiple modalities, structured traces, and customized environments. This combination creates a bottleneck that looks surprisingly familiar to computer architects: memory.

In computer systems, performance and scalability are often limited not by compute, but by memory hierarchy, bandwidth, and consistency. Multi-agent systems are heading toward the same wall — except their “memory” is not raw bytes, but semantic context used for reasoning. After dipping our heads building various LLM multi-agent frameworks over the past two years (e.g., OrcaLoca for software issue localization, MAGE for RTL design, Pro-V for RTL verification, and PettingLLMs enabling RL training on multiple LLM agents), we would like to share our insights learned from our experience through the lens of a computer architect. This blog frames multi-agent memory as a computer architecture problem, proposes a simple architecture-inspired model, and highlights the key challenges and protocol gaps that define the road ahead.

While our perspectives are still preliminary and evolving, we hope they serve as a starting point to ignite a broader conversation.

Multi-Agent Memory Systems in Growing Complex Contexts

Why memory matters: Context is changing

• Longer context windows: Long-context evaluation suites like RULER and LongBench show that “real” long-context ability involves more than simple retrieval — it includes multi-hop tracing, aggregation, and sustained reasoning as length scales.
• Multi-modal inputs: Benchmarks such as MMMU (static images: charts, diagrams, tables) and VideoMME(videos with audio and subtitles) demonstrate that models must handle diverse visual modalities alongside text, extending beyond single-modality processing.
• Structured data & traces: Text-to-SQL (e.g., Spider, BIRD) highlight that agents increasingly operate over structured, executable data — database schemas and generated SQL queries — rather than only raw chat history.
• Customized environments: In SWE-bench and Multi-SWE-bench, models are evaluated by applying patches to real repositories and running tests in containerized (Docker) environments, making “environment state + execution” part of the memory problem. Similarly, WebArena and OSWorld provide realistic, reproducible interactive environments that stress long-horizon state tracking and grounded actions.

Bottom line: Context is no longer a static prompt — it’s a dynamic, multi-format, partially persistent memory system.

Basic Prototypes: Shared vs. Distributed Agent Memory

Before we talk about “hierarchies,” it helps to name the two simplest prototypes, which mirror classical memory systems.

1) Shared Memory

All agents access a shared memory pool (e.g., a shared vector store, shared document database).

• Pros: Easy to share knowledge; fast reuse.
• Cons: Requires coherence support. Without coordination, agents overwrite each other, read stale info, or rely on inconsistent versions of shared facts.

2) Distributed Memory

Each agent owns local memory (local scratchpad, local cache, local long-term store) and shares via synchronization.

• Pros: Isolation by default; more scalable; fewer contention issues.
• Cons: Needs explicit synchronization; state divergence becomes common unless carefully managed.

Most real systems sit somewhere in between: local working memory plus selectively shared artifacts.

An Agent Memory Architecture Inspired by Modern Computer Architecture Design

Computer architecture teaches a practical lesson: you don’t build “one memory.” You build a memory hierarchy with different layers optimized for latency, bandwidth, capacity, and persistence.

A useful mapping for agents will be:

Agent I/O Layer

What it is: Interfaces that ingest and emit information.

• Audio/speech
• Text documents
• Images
• Network calls/web data

Analogy: Devices and I/O subsystems feeding the CPU.

Agent Cache Layer

What it is: Fast, limited-capacity memory optimized for immediate reasoning.

• Compressed context
• Recent trajectories and tool calls
• Short-term latent storage (e.g., KV cache, embeddings of recent steps)

Analogy: CPU caches (L1/L2/L3): small, fast, and constantly refreshed.

Agent Memory Layer

What it is: Large-capacity, slower memory optimized for retrieval and persistence.

• Full dialogue history
• External knowledge databases (vector DBs, graph DBs, document stores)
• Long-term latent storage

Analogy: Main memory + storage hierarchy.

This framing emphasizes a key principle: Agent performance is an end-to-end data movement problem. Even if the model is powerful, if relevant information is stuck in the wrong layer (or never loaded), reasoning accuracy and efficiency degrade.

And just like in hardware, caching is not optional. Similar to computer memory hierarchies, agent memory benefits from I/O and caching layers to improve efficiency and scalability.

Protocol Extensions for Multi-Agent Scenarios

Architecture layers need protocols. In multi-agent settings, protocols determine what can be shared, how fast, and under what rules.

Today, many agent frameworks rely on MCP (Model Context Protocol) as a connectivity layer. Agents registered via MCP can connect and communicate, but inter-agent bandwidth remains limited by message-passing. MCP largely uses JSON-RPC, so it’s best viewed as a protocol for agent context I/O: request/response, tool invocation, and structured messages.

That’s necessary — but not sufficient.

Missing Piece 1: Agent Cache Sharing Protocol

Many recent studies, such as DriodSpeak and Cache to cache, explored KV cache sharing between LLM. However, we still lack a principled and unified protocol for sharing cached artifacts across agents.

Goal: Enable one agent’s cached results to be transformed and reused by other agents.

In architecture terms, this is like enabling cache transfers or shared cache behavior — except the payload is semantic and may require transformation before reuse.

Missing Piece 2: Agent Memory Access Protocol

Although frameworks like Letta and Mem0 support shared state within agent memory, the protocol defines how agents read/write each other’s memory is missing.

Goal: Define memory access semantics: permissions, scope, and granularity.

Key questions:

• Can Agent B read Agent A’s long-term memory, or only shared memory?
• Is access read-only, append-only, or read-write?
• What is the unit of access: a document, a chunk, a key-value record, a “thought,” a trace segment?
• Can we support “agent RDMA”-like patterns: low-latency direct access to remote memory without expensive message-level serialization?

Without a memory access protocol, inter-agent collaboration is forced into slow, high-level message passing, which wastes bandwidth and loses structure.

The Next Frontier: Multi-Agent Memory Consistency

The largest conceptual gap is consistency. The goal of memory consistency in computer architecture and systems design is to define constraints on the order of reads and writes to memory addresses. Consistency models (e.g., sequential consistency, TSO, and release consistency) clarify what behaviors programmers can rely on.

For agent memory, the goal shifts: It’s not about bytes at an address, but about maintaining a coherent semantic context that supports correct reasoning and coordination.

Why Agent Consistency Is Harder

• The “state” is not a scalar value; it’s a plan, a summary, a retrieval result, a tool trace.
• Writes are not deterministic; they may be speculative or wrong.
• Conflicts aren’t simple write-write conflicts — they’re semantic contradictions.
• Freshness depends on the environment state (repo version, API results, and permissions).

What a Multi-Agent Memory Consistency Layer Might Need

A practical direction is to define consistency around the artifacts agents actually share — cached evidence, tool traces, plans, and long-term records — across both shared and distributed memory setups (often a hybrid: local caches + shared store). The layer should expose a consistency model (e.g., session, causal, eventual semantic, and stronger guarantees for “committed” outputs), provide richer communication primitives than plain message passing, and include conflict-resolution policies (source ranking, timestamps, consensus, and optional human intervention for high-stakes conflicts).

Research on this is still rare, but it is likely to become foundational — much like coherence and consistency were for multiprocessors.

Conclusion

Many agent memory systems today resemble human memory — informal, redundant, and hard to control — leaving a large opportunity for computer architecture researchers to rethink what “memory” should mean for agents at scale. To move from ad-hoc prompting to reliable multi-agent systems, we need better memory hierarchies, explicit protocols for cache sharing and memory access, and principled consistency models that keep shared context coherent.

Acknowledgement

We sincerely thank Wentao Ni, Hejia Zhang, Mingrui Yin, Jiaying Yang, and Yujie Zhao for their invaluable contributions through brainstorming, discussions, data collection, and survey work over the past few months. This article would not have been possible without their dedicated efforts.

About the authors:

Zhongming Yu is a PhD student in the Computer Science and Engineering Department at University of California, San Diego. His research interests are in combining machine learning and computer systems, with a special focus on LLM agent system for machine learning systems, evolving ML and systems, and autonomous software engineering. 

Jishen Zhao is a Professor in the Computer Science and Engineering Department at University of California, San Diego. Her research spans and stretches the boundary across computer architecture, system software, and machine learning, with an emphasis on memory systems, machine learning and systems codesign, and system support for smart applications.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

TL;DR: Latency-tolerant architectures, e.g., GPUs, increasingly use memory/storage hierarchies, e.g., for KV Caches to speed Large-Language Model AI inference. To aid codesign of such workloads and architectures, we develop the simple PipeOrgan analytic model for bandwidth-bound workloads running on memory/storage hierarchies. 

Background

For three reasons, memory bandwidth, more than latency, limits AI inference performance. First, AI inference uses latency-tolerant compute engines, such as GPUs. Second, it principally uses hardware memory hierarchies to store a data structure called a Key-Value (KV) Cache that holds information from recent queries to reduce redundant computation. With PagedAttention, each KV Cache fetch obtains one or more multi-megabyte blocks (often called pages) that require substantial bandwidth to complete. Third, inference’s “decode” phase is memory-bound due to low arithmetic intensity, putting great pressure on memory bandwidth.

Traditional CPU memory/storage hierarchies are shaped by increasing latency, but designing hierarchies for AI workloads requires focusing on decreasing bandwidth. Since AI software is flexible, codesigning software and hardware is essential. 

To provide intuition and first answer to the above questions, we next contribute the simple PipeOrgan analytic model for optimizing bandwidth-bound workloads running on a memory hierarchy with many parallel pipes from memories to compute. The PipeOrgan model shows that husbanding and providing bandwidth is important for AI software and hardware. Analytic models have long provided computing intuition, e.g., Amdahl’s Law, Iron Law, and Roofline.

Example System with Two Parallel Memories

Let’s start simple. Consider the hardware depicted in Figure 1 with High Bandwidth Memory (HBM) with bandwidth 16 TB/s in parallel with an LPDDR memory with bandwidth 0.5 TB/s. Assume for now that there are no transfers between memories, e.g., to cache. 

Using the PipeOrgan math from the next section, Figure 2’s blue line shows how system performance changes depending on what percentage of data comes from LPDDR memory. (The orange line comes later when we add caching.) Performance is highest when LPDDR provides exactly 3% of the data (arrow 1), which matches its 3% bandwidth (0.5/(16.0+0.5)). At this point, both LPDDR and HBM memories finish transferring data at the same time, so they act as co-bottlenecks and the system runs at peak efficiency.

When less than 3% of data is from LPDDR (left of the peak), HBM finishes last and limits performance. When LPDDR sources more than 3% (right of the peak), it is the bottleneck. LPDDR might have to source more data, because HBM’s limited capacity, currently 48-64GB per stack, may preclude it from being able to source its share (97%). If so, performance drops quickly: 4% from LPDDR gives 76% of peak (arrow 2), and 20% yields just 15% (arrow 3).

However, future AI systems will feature multiple memory and storage levels, using HBM, LPDDR, host DDR, pooled DDR, and attached or pooled FLASH storage.

PipeOrgan Model of Systems with N Parallel Memories

The above result generalizes to an N-level memory/storage hierarchy with each level feeding compute in parallel. Optimal performance is achieved when all parallel memories complete a workload phase simultaneously, leading to this PipeOrgan principle:

Memory-bandwidth-bound workloads perform best when data is sourced from each memory level in proportion to its bandwidth.

Proof: 

1. Let each memory provide bandwidth b_i TB/s in parallel for total bandwidth B = b_1 + … + b_N.
2. For a workload, let each source d_i bytes in parallel for total data transferred D = d_1 + … + d_N.
3. By assumption, the workload is limited by data transfer time with compute hidden.
4. Time for each memory to finish its data transfer is d_i/b_i  = TB/(TB/s) = seconds.
5. Workload Time is the maximum of all memories finishing: MAX [d_1/b_1, …, d_N/b_N].
6. Workload Performance = 1/ Time = MIN[b_1/d_1, …, b_N/d_N].
7. Set each d_i = (D/B)*b_i = proportional to its bandwidth b_i.
8. Performance = MIN[b_1/((D/B)*b_1), …, b_N/((D/B)*b_N)].
9. Performance = MIN[(B/D), …, (B/D)] = B/D and Time = 1/Performance = D/B. 

This makes sense: PipeOrgan shows that best performance occurs when one moves all the data using all the bandwidth with no bandwidth idling.

But Caching Is Critical

The PipeOrgan version above assumes all data goes directly to compute, without transfers among memories. In reality, systems move data from lower- to higher-bandwidth memories, caching it for reuse. For a two-level system (see Figure 4), assume the entire fraction of the workload’s data used from LPDDR is first transferred to HBM for caching (orange arrow). Let the data used from LPDDR be f*D where f ranges from 0 to 1.

• Performance with caching = MIN[(b_1/D)/(f+1), b_2/(f*D)] = MIN[limited by HBM BW, limited by LPDDR BW].

Figure 2 shows an orange curve for caching that is hidden under the original blue curve when more than 3% of data is sourced from LPDDR. At more than 3% from LPDDR, performance–without and with caching–is limited by the time to transfer needed data with the same limited LPDDR bandwidth.

While it might look like caching doesn’t matter, caching is actually important. This is because caching can greatly shift a workload’s x-axis operating point. For example, sourcing 20% of data from LPDDR yields 15% of peak performance (arrow 3). If LPDDR data is cached in HBM and reused five times, then–as the orange dashed arrow shows–only 4% comes from LPDDR and performance gets boosted to 76% of peak—a ~5x improvement (arrow 2).

Consequently, caching remains critical. Moreover, PipeOrgan and its N parallel memory principle also applies bandwidth-bound workloads once caching’s more complex information flows are accounted for.

Implications, Limitations and Future Work

Statistician George Box famously said, “Essentially, all models are wrong, but some are useful.” 

We conjecture that the PipeOrgan model is useful for AI codesign, especially in the early stages and with software people having less hardware understanding. Its key implication is that bandwidth-bound workloads must carefully manage bandwidth from larger, slower memories and storage. While vast data can be stored statically, dynamic use from low-bandwidth memories should remain modest.

Three PipeOrgan limitations motivate future work. First, most workloads aren’t bandwidth bound throughout, and PipeOrgan doesn’t address other phases. Modeling these requires more parameters, increasing accuracy but also complexity.

Second, the caching model variant only covers two memory levels and always transfers data first to the higher-bandwidth level before use. Future work should extend this to N memory levels and more advanced caching policies. Modeling the many options for caching may be challenging.

Third, PipeOrgan may need to be extended for systems that do some processing in or near the memories themselves rather than moving all data to a segregated compute unit.

Burks, Goldstine, & von Neumann, 1946: We are therefore forced to recognize the possibility of constructing a hierarchy of memories, each of which has greater capacity than the preceding but which is less quickly accessible.

In sum, after eight decades of memory hierarchies focused mostly on latency, we are now at the exciting early stages of codesigning bandwidth-focused memory/storage hierarchies for more flexible AI software.

About the Author: Mark D. Hill is John P. Morgridge Professor and Gene M. Amdahl Professor Emeritus of Computer Sciences at the University of Wisconsin-Madison and consultant to industry. He initiated the PipeOrgan model consulting for Microsoft and was given permission to release it. He is a fellow of AAAS, ACM, and IEEE, as well as recipient of the 2019 Eckert-Mauchly Award.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Michael J. Flynn is a widely respected contributor—indeed a giant—in the field of Computer Architecture.  He made highly significant and impactful contributions throughout his career, both in industry and in academia.  Sadly, he passed away peacefully December 24, 2025, having lived a long and full life.

Born May 20, 1934, in New York, NY, Flynn earned his Bachelor’s, Master’s, and Ph.D. degrees in Electrical Engineering from Manhattan College (1955), Syracuse University (1960), and Purdue University (1961), respectively, and he received an honorary Doctor of Science degree from the University of Dublin (1998).  After ten years as a design engineer and project manager at IBM (1955-65, in Endicott and Poughkeepsie, NY), he became a member of the faculty at the University of Illinois at Chicago (1965-1966), Northwestern University (1966-1970), and Johns Hopkins University (1970-1975) before joining Stanford University in 1975 as Professor of Electrical Engineering.  He taught internationally, in Ireland, other places in Europe, Singapore, and Japan.

As a young project manager at IBM, Flynn was responsible for the design of the well-known IBM System 360 (Models 91/92/95 series), the first computer to implement the sophisticated Tomasulo algorithm, along with many other groundbreaking high-performance architectural techniques.  As the first family of general-purpose computer mainframes that featured architectural compatibility for both commercial and scientific applications, the System 360 is widely recognized as revolutionizing computing during that time—and in many ways persisting even today.  Indeed, many of the high-performance computing techniques developed by Flynn and his IBM colleagues are used throughout the industry today, having migrated from barn-sized mainframes to finger-nail sized microprocessor chips.  Flynn also was the first to shed light on the performance potential and limitations of parallel computers with what’s become known as Flynn’s classification (or Flynn’s taxonomy), a pioneering framework for categorizing parallelism in computer architectures based on the number of simultaneous instruction streams and data streams they handle, e.g., SISD, SIMD, MISD, and MIMD.  His original taxonomy is still used widely today, with various extensions derived from it, to distinguish between different kinds of parallel processor computer systems.

In 1972, together with some colleagues from IBM, Flynn co-founded Palyn Associates which provided consulting services in the field of high-performance computer architecture and design.  For more than 30 years, he and his colleagues advised nearly every major computer company in Japan, Europe and the United States, including IBM, CDC, Fujitsu, Hitachi, Honeywell Bull, and ICL.  Later, he played a prominent role in Maxeler, products of which made use of advanced dataflow techniques to provide high performance processing for specific applications, such as automated trading.  As a renowned professor at Stanford until his retirement in 1999 and transition to emeritus status, Flynn made seminal contributions to instruction set architecture (ISA), computer arithmetic, advanced floating-point design, multimedia, parallel processors and interconnects, emulation, and performance evaluation, to name a few.  He (co-)authored several textbooks, including Introduction to Arithmetic for Digital Systems Designers, Computer Architecture: Pipelined and Parallel Processor Design, and Advanced Computer Arithmetic Design. An IEEE Fellow, ACM Fellow, and Fellow of the Institution of Engineers of Ireland, Flynn received numerous other honors and awards for his impactful technical contributions, including the ACM/IEEE Eckert-Mauchly Award (1992), IEEE Computer Society’s (CS) Harry Goode Memorial Award and Medal (1995), the Tesla Award and Medal from the International Tesla Society in Belgrade (1998), IEEE CS Charles Babbage Award, IEEE CS Computer Pioneer Award (2015, his acceptance speech video is here), and many others.

Notably, when the field of computer architecture was still in its infancy more than fifty years ago, Flynn founded the IEEE CS Technical Committee on Computer Architecture (TCCA) and ACM’s Special Interest Group on Computer Architecture (SIGARCH); he also started the ACM/IEEE International Symposium on Computer Architecture (ISCA), co-sponsored by both, which is among the most prestigious flagship computer architecture conferences in the world.  At ISCA’s 50^th anniversary conference at FCRC 2023, Flynn was invited to give a “50 Year Retrospective Lecture” and was given an honorary plaque with these words inscribed: “In recognition, with tremendous gratitude, of your lifetime dedication and leadership to the computer architecture community on this the 50^th anniversary of your founding of ISCA, SIGARCH, and TCCA.”

Even more than his impressive technical contributions, which are many, Flynn is remembered fondly by the many dozens of doctoral graduate students he advised—for his unending kindness, wealth of wisdom, caring tutelage, gentle encouragement, constant motivation, and enduring support, especially when most needed.  He treated each and every student as if they were a member of his own family, and he was viewed by them not only as their academic “father,” but referred to affectionately as “the Great Man.”  Many of his former mentees returned to Stanford several times each year for luncheons to enjoy his company and reminisce about exciting times working with him in tackling some of the most compelling technical issues of the day.

Flynn was an equally generous mentor to his junior faculty colleagues, helping them establish their careers and providing sage advice as they made their way.  Kunle Olukotun attests to this: “Meeting Mike Flynn near the end of my Ph.D. at the University of Michigan changed the trajectory of my career.  At the time, I was firmly on a path toward industry, but Mike believed that I could be a strong academic, and he encouraged me to apply to Stanford.  Mike saw something in me that I did not yet see in myself, and that confidence made an enduring difference. Once I arrived at Stanford, Mike served as my mentor. He helped me navigate the academic waters with thoughtful and wise advice, provided opportunities to showcase my research, and supported me through nominations for awards and professional recognition.  I am deeply grateful to Mike for all he did to help establish my career, and for the role he played in the success of so many other junior colleagues whom he mentored with the same generosity and vision.  I am deeply saddened by his passing.”  Similar sentiments are echoed by Bill Dally, who shares the following: “I first met Mike as a graduate student at Stanford in 1980.  I was awed by his accomplishments and his understanding of parallel computing. He kindled my interest in parallel computing which launched me on a very successful career.  Later, when I came to Stanford as a faculty member in 1997, I found Mike to be a great source of advice about Stanford, being a faculty member, research strategy, and many other topics.  I am deeply saddened to hear of Mike’s passing.  He will be greatly missed.”  Another of his faculty colleagues at Stanford, Christos Kozyrakis, recalls the following: “One of the most memorable moments of my early teaching years was hosting him in class to discuss the Flynn taxonomy of computer architecture—a special experience for both the students and myself and a vivid reminder of the lasting impact of his work.”  Indeed, Mike Flynn was highly respected and revered by fellow colleagues all throughout his professional career.  Solemnly noted by John L. Hennessy, “Mike was the person who hired me at Stanford, gave me some of my first research funding, jointly published an early paper with me, and gave me my first consulting opportunity.  Sadly, his passing marks the end of an important era in computing: Mike was the last of the great System 360 pioneers—Gene Amdahl, Bob Evans, Fred Brooks, Eric Bloch, Gerry Blaauw, and Robert Tomasulo—all are now gone.”

He was a wonderful human being.

Professor Michael J. Flynn will be sorely missed by his loving family as well as by his extended academic family and all those whose lives he has indelibly touched over his blessed ninety-one plus years.  May he rest blissfully in peace, and may his venerable legacy be inspirational and long lasting.  Fittingly, through Mike Flynn’s final public words to all of us in the computer architecture community in his ISCA 50^th Anniversary Lecture, he exhorted us all by saying: “Now it’s your turn!”

About the Authors:

Ruby B. Lee is the Forest G. Hamrick Professor Emeritus in the ECE department at Princeton University, and chief architect at Hewlett-Packard in Silicon valley before that. She is a Fellow of the IEEE, ACM and the American Academy of Arts and Sciences, and recipient of awards such as the most Influential Paper award in 20 years at ISCA 2025 and the Test of Time award at the ACSAC 2024 security conference. Her research combines cyber security, computer architecture and deep learning, including secure processor and cache architectures, attacks and defenses, low-cost AI and multimedia.

Charlie Neuhauser is now retired after more than 50 years in the field of computer design and analysis.  During the latter half of his career, he provided technical insight to attorneys and companies in the area of intellectual property.  He is currently the registration chair for the IEEE Hot Chips Symposium.

Timothy M. Pinkston is the George Pfleger Chaired Professor of Electrical and Computer Engineering at the University of Southern California and also is a Vice Dean in USC’s Viterbi School of Engineering.  A Fellow of AAAS, ACM, and IEEE, and recipient of the ACM SIGARCH Alan D. Berenbaum Distinguished Service Award, Timothy’s research contributions mainly are in the area of interconnection networks and efficient data movement in parallel computing systems.

All three authors are former  Ph.D. students of Mike Flynn at Stanford (Lee and Pinkston) and Johns Hopkins (Neuhauser).

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.