This is the second (and final) post in the series covering the Next Generation Branch Prediction Champtionship (CBP-NG). Part-I details the provided infrastructure and the winning submissions.

Distilled Branch Predictors (Simha Sethumadhavan – Columbia University)

The author proposes an ‘Ensemble Predictor’ composed of a fast, one-cycle predictor P1 and a slower (but more accurate) two-cycle predictor P2. However, instead of training them both independently on the actual branch outcomes, P1 is trained on the predictions made by P2 (which continues to be trained on branch outcomes), turning it into a faster, distilled version of the more accurate predictor. This student-teacher model improves the VFS score by lowering the misprediction recovery latency. An additional PC-tagged structure is added to identify one-sided branches and prevent the usage of P2 to predict them, further lowering the energy consumption. In the final submission, a bimodal predictor is used as P1, and P2 is an 8-table TAGE predictor.

Run-Ahead Block TAGE (Prakhar Gupta, Yuwei Sun, Rishav Sen, Reet Sinha, Swetha Karthikeyan – University of Illinois Urbana-Champaign)

The authors use a single (15-table, geometric history length) TAGE predictor. Ahead Pipelining is implemented to hide the table access latency, so that the effective prediction latency is one cycle. To correctly use ahead-pipelining, a secondary tag is added to all table entries; an entry can be used only if the tag calculated from the last branch PC of the previous block matches with the stored tag. The resulting increase in throughput outweighs the increase in TAGE MPKI, boosting the VFS score. A maximum of 7 branches can be predicted at once, over a 256-instruction predict block. An 8k-entry bimodal predictor is used as a fallback when TAGE is unable to provide a prediction. Independent Resolution Chains allocate a tag, usefulness and hysteresis field to each slot in the predict block, allowing each slot to use a different TAGE table. Using independent history length per branch improves the MPKI for most cases, but is detrimental when there is intra-block correlation.

Coding Agents as Design Searchers: An Autonomous TAGE Tuning Campaign
(Matt Pallan – University of Maryland)

This submission stands out because its primary contribution is the methodology, not the predictor. The author proposes a systematic approach to leverage AI for design space exploration – the tasks are split between two independent agents Ag1 (Claude Code) and Ag2 (OpenAI Codex). Ag1 is responsible for coming up with structural (table size, hashing, indexing, etc.) or behavioral (update policy, initial value, etc.) changes and their implementation. Ag2 acts as a critic – questioning the intent behind Ag1’s changes and coming up with out-of-the-box architectural ideas. The agents stay in sync through shared collateral – a journal and a machine-readable ledger, which are updated after every experiment. The only human inputs for this study are the objective (maximizing the VFS score), the record-keeping mechanism and some hard constraints to reduce the search space (such as capping the predictor latency at 2 cycles). Each configuration proposed by Ag1 is first tested on a reduced instruction count on the provided traces, only those that perform well (31 out of 2300+) are run on the full instruction count. Out of these, only six changes proved beneficial and survived cross-inspection by Ag2. The final TAGE model is tuned to prioritize throughput over accuracy and ditches the Statistical Corrector mechanism as it is deemed infeasible for HARCOM implementation.

Offset-Free TAGE-SC
(Yashwant Kumar Balivada, Sairam Viswanathan Susarla – Texas A&M University)

A two-level predictor is implemented – the first level is a single-cycle gShare that gives fast predictions. The second level consists of a TAGE-SC predictor which is more accurate. The key distinction of the latter is its usage of Branch Number instead of Branch Offset within a predict block. This leverages the championship-specific assumption that we can magically identify conditional branches, to reduce storage and energy expenditure. The VFS score is maximized when predicting upto 8 branches in a 64-instruction window. The Statistical Corrector uses 3 components –  branch-PC bias, taken branch history and global path history.

Key Takeaways

The usage of  TAGE was a common trend in the championship, with all the submissions implementing some form of tagged history tables in their predictor to reduce MPKI. Ahead Pipelining was another popular technique that many of the contestants used to hide the table-access latency for large predictors. This was also the first championship to have a submission developed by AI agents; the easily reproducible methodology holds great potential for future computer architecture research.

This was the first Branch Prediction Championship to make energy consumption a part of the final score. Naturally, the submissions saw a shift from the highly accurate and complex predictors that dominated the past championships to simpler, implementation-friendly models. This, along with the usage of HARCOM to model the predictors in a hardware description language (or something close to it) ensured that the industry can easily adopt the novel ideas that were proposed.

However, despite this imposition of practicality, the provided infrastructure made some simplifying assumptions that were criticized by some of the popular names in Branch Prediction. The first issue was the magical detection of conditional branches by the simulator, which is quite unrealistic. Allowing the contestants to implement a BTB (Branch Target Buffer) and RAS (Return Address Stack) could help paint a more complete picture. Secondly, the usage of high throughput techniques like Ahead Pipelining requires some consideration of the misprediction recovery mechanism, which is a key stage in any high performance branch predictor.

About the Author

Digvijay Singh obtained his Bachelor’s degree from BITS Pilani, India and his Master’s degree from Texas A&M University where he worked on data prefetching as part of the CAMSIN research group. He currently works as a Silicon Architect in Google’s mobile CPU team.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Background

The Next Generation Branch Prediction Championship (CBP-NG) aimed to address practical implementation challenges in branch prediction, moving beyond metrics used in earlier iterations. By emphasizing energy-adjusted performance—incorporating throughput, timing, and dynamic energy alongside traditional Misprediction Per Kilo-Instruction (MPKI) metrics—the championship sought to align research with industry design constraints. Utilizing an extensive set of 1579 traces derived from ARM and Ampere architectures, the event evaluated predictors through a two-level model: a first-level (P1) predictor for immediate output and a second-level (P2) predictor that verifies and corrects potential mis-predictions. All submissions were required to be implemented using the HARCOM C++ library that imposes restrictions on the behaviour of all variables and structures, treating them as hardware components such as wires, registers and SRAMs arrays. The energy consumption was approximated based on the ‘hardware’ used, assuming 5nm FinFET technology. The final scoring was done using a mathematical formula that combines:

• IPC: Number of instructions predicted per cycle
• CPI: Number of mispredictions (by the L2 predictor) per correct-path instruction times misprediction penalty
• EPI: Energy consumed per correct-path instruction

into a Voltage Frequency-scaled Speedup (VFS) score. The objective is to maximize this VFS score.

In this blog post, we first describe the gSHare-N-ahead predictor, which delivers the highest VFS score among all example predictors provided with the infrastructure. It implements a mechanism called ahead-pipelining, which inspired several submissions in this championship. This is followed by an overview of the championship winners – starting from first place to third. The remaining submissions will be discussed in Part 2 of this blog post.

GShare N-Ahead Predictor

GShare N-ahead is one of the predictors provided as part of the simulation infrastructure. It supplements the conventional N-wide (N predictions per cycle) gShare predictor with ‘Ahead Pipelining’. Instead of using the current block B1’s address (PC_B1), it uses the prior block B0’s address (PC_B0) to index the prediction tables. Since PC_B0 is available one cycle earlier than PC_B1, predictions can be made earlier, effectively reducing prediction latency. The only catch is that in a predict block with N branches, there can be N+1 paths leading out of it, so until PC_B1 is known, we have to consider all possibilities and make (N+1)*N predictions in BP0. Once PC_B1 is known in BP1, it is combined with B0’s last conditional branch direction and the number of conditional branches in B0 to multiplex the BP0 predictions to obtain the final N predictions.
The particular implementation used in the championship predicts N = 7 branches per cycle as a P1 predictor; P2 simply reuses the P1 prediction for super-aggressive prediction throughput.

MORSL: Minimal-Overhead Rank-Based Predictor with Summation-Free Correction and Lazy Access (Toru Koizumi, Masanari Mizuno, Soma Nishida – Nagoya Institute of Technology; Kanata Abe – The University of Tokyo; Tomoaki Tsumura – Nagoya Institute of Technology; Ryota Shioya – The University of Tokyo). First place

The authors started with a TAGE predictor for high prediction accuracy, and implemented the following features: 

• Ahead-pipelining is enabled for a higher throughput.
• While deciding the TAGE history lengths, the shortest history length is increased to compensate for the reduced context due to ahead pipelining. Furthermore, the larger history lengths are spread farther apart than the smaller ones. This allows for pseudo-skewed associativity within smaller tables, since each TAGE table has a single tag per entry. 
• A Tagged Corrector (TC) mechanism consisting of two tagged predictors.  TC-Bias is indexed using the current block address and can override the TAGE prediction  in case of a high confidence tag match. TC-History uses the previous block’s address, BrIMLI and Path History to override the TAGE + TC-Bias prediction in case of a tag match. Using overrides instead of summation reduces energy consumption by avoiding multiple table reads on the adder datapath.
• An ‘allocation guided’ access filter is implemented (as two 1-bit tables indexed by PC and PC XOR shortest history), which records whether a branch PC had any allocations in a longer TAGE table in the past. The intuition is that if the PC did not trigger allocations to a longer table, the shorter tables were able to accurately make predictions for it. Unless both the access filter table reads return 1, the ‘mini-TAGE’ mode is activated and only the two shortest TAGE tables are read to save energy.

Ahead-Pipelined N-branch GShare with Tagged Tables (Jun Fan). Second Place

This proposal improves the accuracy of the ahead-pipelined gShare predictor by using 18-bit history and 2 hysteresis bits. Furthermore, the gShare RAM size is halved to make room for two ahead-pipelined tagged tables (20 and 80-bit histories, respectively). These three tables are looked up in parallel during prediction, and the longest matching entry is used to make the first stage prediction P1. The final prediction P2 simply reuses P1 to maximize throughput. Upon a branch misprediction, the table that produced the prediction is updated, and new entries are allocated in tables with longer histories.  If a wrong prediction came from the 20-bit table, a new entry will be allocated only in the 80-bit table, but if it came from the 18-bit table, the 20 and 80-bit tables will both have new allocations. Finally, a direct-mapped ‘bias’ table is added which stores a PC along with  a 3-bit saturating counter in each entry. When gShare is used to make the final prediction and the corresponding PC is found in the table with a strong bias (000 or 111), the predictor is updated with the stored strongly biased value.  

Energy-Efficient Ahead-Pipelined TAGE
(Nhat Dang, Eric Rotenberg – North Carolina State University). Third Place 

The authors implement a single branch predictor to reduce the energy consumption due to multiple table reads and updates in a dual-predictor scenario. Ahead Pipelined TAGE is implemented, consisting of a baseline bimodal predictor and 10 tagged tables with geometric history lengths. It uses the previous block’s address to predict upto 4 branches per cycle in a 256-instruction region. A 6-bit secondary tag (generated from the lower bits of the last branch PC) is added to the TAGE entries to allow disambiguation of the missing history for the previous block. Additionally, the access to  6 largest TAGE tables is gated. Specifically, if there have been no mispredictions in the last 512 blocks, and the total number of mispredictions is less than the (number of retired branches / 1024), these tables are not accessed, to save dynamic energy. Further energy optimizations include placing the tag comparison logic near the tag SRAMs, and placing the SRAMs accessed during prediction together.

Stay tuned for Part-2 of this article to learn more about other submissions and the industry/academia’s perception of this contest.

About the Author

Digvijay Singh obtained his Bachelor’s degree from BITS Pilani, India and his Master’s degree from Texas A&M University where he worked on data prefetching as part of the CAMSIN research group. He currently works as a Silicon Architect in Google’s mobile CPU team.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Zero-Knowledge Proofs

As conversations about LLMs continue to dominate computer science, the cryptographic community has been abuzz around a different three letter acronym: ZKP. Zero-Knowledge Proofs (ZKPs) allow a prover to convince a verifier that a statement is true without revealing the secret information that makes it true. In other words, ZKPs make it possible to verify computation without exposing private data behind it. 

In a cloud computing context, ZKPs could allow a service provider to convince a client that they executed a service faithfully, without having to reveal their sensitive trade secrets, IP, or (for AI services) model weights. On the other hand, ZKPs could allow an edge user concerned about security or anonymity to prove identification, financial eligibility, or authenticity of image transformation to untrusting services. 

Two flavors of ZKPs have emerged with particularly intriguing tradeoffs: zkSNARKs  (zero-knowledge Succinct Non-Interactive ARguments of Knowledge) and zkSTARKs (zero-knowledge Scalable Transparent ARguments of Knowledge). zkSNARKs are attractive because they generate compact proofs (as small as 192 bytes!) which are fast to transmit and verify. This made them especially appealing for early blockchain adopters of ZKPs. zkSTARKs, on the other hand, offer both faster proving times and post-quantum security, but generate much larger proofs (upwards of several megabytes). This increases communication costs and requires more processing by the verifier. As such, zkSTARKs may be more advantageous in applications where proof size constraints are more relaxed or where the number of verifiers is small.

Both zkSNARKs and zkSTARKs suffer from the computational cost of proof generation. The proving time to generate a zkSNARK for a target application is roughly 5-6 orders of magnitude more than the time to run the original application itself. zkSTARK provers are similarly expensive. Further, as the problem complexity grows (e.g., running an AES is much simpler than running a GPT inference), the corresponding ZKP’s computational cost also grows. Consequently, many otherwise compelling and exciting ZKP applications remain out of reach today. To address this, researchers have begun accelerating both zkSNARKs and zkSTARKs with GPUs, FPGAs, and ASICs. 

Dissecting a ZKP

Modern ZKPs are diverse and defined by combining a polynomial interactive oracle proof (PIOP), which reduces program correctness to polynomial checks, with a polynomial commitment scheme (PCS), which binds the prover to the relevant input polynomials and lets the verifier check claimed polynomial values at specific points. PIOPs and PCSs can be mixed and matched to define many ZKP protocols with different tradeoffs in proof latency, proof size, verifier latency, setup assumptions, and post-quantum security.

The diversity between PIOP and PCS schemes typically results in ZKPs built upon some kernels with extremely high arithmetic intensity on wide bitwidths and others with low arithmetic intensity and complex data movement patterns. Any hardware seeking to accelerate ZKPs must account for both flavors of computational kernel. 

The high data volume follows from how PIOPs represent the execution of a function. Many PIOPs encode a program’s execution trace as one or more polynomials (represented as long vectors of polynomial evaluations). Verifying program correctness then reduces to checking that these polynomials are consistent with the program’s constraints and satisfy a number of algebraic identities. Polynomials are especially useful because, rather than checking these constraints individually across the entire execution trace, they can be encoded as polynomial identities that the verifier tests at a small number of randomly chosen points. These long vectors can range from 2¹⁴–2³⁰ elements, amounting to 0.5 MB – 32 GB for a single such vector. These vectors may be repeatedly transformed, permuted, hashed, or combined, creating substantial pressure on memory capacity, bandwidth, and data movement. 

While PIOPs often rely on more data movement with lower arithmetic intensity, PCSs often behave quite differently. The PCSs found in zkSNARKs primarily rely on modular arithmetic over large prime fields and elliptic curve (EC) groups. Generally, field elements are on the order of 256 bits wide, with some ECs’ field elements being upwards of 384 bits. EC computations are even more computationally expensive than large-bitwidth finite field computation, with the addition of two EC group elements requiring 16 (or more) modular multiplications. zkSNARK PCSs are dominated by the latency to calculate many EC group additions. 

zkSTARK PCSs differ significantly from zkSNARKs. zkSTARK PCSs are built upon repeated hashing as opposed to EC arithmetic. Some zkSTARK PCSs use smaller, hardware-friendly finite fields, which can reduce arithmetic bit widths; others still operate over wide fields on the order of 256 bits. In either case, zkSTARK provers must perform extensive hashing and repeatedly process, reorganize, and extend large vectors of field elements, creating substantial memory-bandwidth and data-movement demands.

Due to differences between PCS and PIOP kernels, accelerating proof generation requires more than support for wide datatypes and long vectors. A practical architecture must efficiently handle both compute-intensive kernels, such as elliptic-curve arithmetic and hashing, and bandwidth-intensive kernels, such as polynomial transform and large-scale data rearrangement.

Accelerating ZKP Kernels

Seeing the promise and current computational limitations of ZKPs, many architects have begun work accelerating ZKPs on GPUs, FPGAs, and ASICs. This largely amounts to accelerating individual ZKP kernels.

The Multi-Scalar Multiplication (MSM) kernel serves as the bedrock of zkSNARK PCSs. MSMs compute the sum of many EC point multiplications, which can be thought of as dot products between vectors of scalar integers (derived from the polynomials) and vectors of two-dimensional EC points (derived from the protocol setup phase). However, scalar multiplication by an EC point is actually achieved via repeated EC point addition (or PADDs) described above. Naively, assuming 256-bit scalars, a single scalar multiplication requires 2²⁵⁶ PADDs. This cost can be tempered by employing a binary representation of scalars and performing a “shift-and-add” algorithm for multiplication, resulting in at most 255 PADDs and 255 PDBLs (point doubles) per scalar that must be computed serially. To improve upon this, most MSM implementations today use Pippenger’s algorithm, a technique that parallelizes computations across independent slices of the scalars, to further reduce the number of PADDs. However, Pippenger’s algorithm is data-dependent upon the scalars being committed. Many MSM workloads are sparse, meaning they contain a disproportionate number of zeros, ones, and other small scalars, while others are dense, consisting of randomly distributed full-width field elements. These differences can significantly change the amount of useful EC work performed, the load balance across parallel workers, and the memory-access behavior. Recent ASIC approaches have proposed dynamically scheduled microarchitectures, contention-free multi-PE datapaths, priority-based bucket scheduling, and reconfigurable modular arithmetic arrays. Recent GPU approaches have proposed fine-grained, load-balanced task mapping, sparse-matrix-based bucket accumulation, adaptive preprocessing for improved time–space trade-offs, and tensor-core-assisted arithmetic and multi-GPU orchestration.

Merkle Trees are the zkSTARK analog to MSMs, enabling commitments to a vector via hashing as opposed to EC cryptography. Constructing a Merkle Tree is equivalent to constructing a binary tree where the vector values being committed are the leaves, and parents are constructed by performing a hash seeded by its two children. The root node of the resulting Merkle Tree is a commitment to the polynomial. Constructing a Merkle tree requires a large number of hash operations: for a tree with N leaves, the prover must compute approximately N-1 internal-node hashes. However, performance is shaped not only by hash throughput, but also by how the tree is traversed. A conventional breadth-first construction materializes each tree level before computing the next, creating substantial intermediate-memory traffic. Recent ASIC approaches like the MTU use a hybrid traversal that preserves parallel execution while retaining partial reductions locally, reducing the need to repeatedly write and reread intermediate tree nodes from memory. Other approaches tile Merkle trees into scratchpad-resident subtrees and process each subtree entirely on-chip using spatial arrays, while GPU approaches pipeline batched Merkle trees across layer-specific GPU kernels to sustain thread utilization while overlapping data transfers with hashing. Recent work also explores FPGA acceleration of ZK-friendly hash functions to increase hash throughput. 

Number Theoretic Transforms (NTTs) are analogs of the FFT but over large, ~256 bit finite field elements. NTTs are primarily used to reduce the cost of polynomial multiplication from O(N²) operations to O(N log N). The key challenge for NTTs is handling variable memory access patterns in between NTT stages and handling off-chip memory transfers when computing NTTs whose working set doesn’t fit into on-chip caches and scratchpads. Recent approaches have focused on recursive decompositions, pipelined MDC based architectures, and constant geometry approaches.

SumChecks are another major prover kernel for many PIOPs. At a high level, a SumCheck prover repeatedly streams large tables of polynomial evaluations, performs elementwise finite-field computations determined by the polynomial being evaluated, and reduces the results into progressively smaller tables. Although each round is built from additions, multiplications, and linear combinations, these operations are performed over wide field elements and must be applied across large vectors, creating substantial arithmetic and memory-bandwidth demand. While this seems relatively simple, unlike NTTs, SumCheck’s dataflow changes with the polynomial. More input polynomials or higher-degree expressions require more elementwise evaluation products, increasing both the compute load and memory pressure for storing  intermediates during folding. Initial work addressed this with vector architectures and fixed-function ASICs to exploit short-term data reuse within streaming frameworks. Recent work like zkPHIRE has proposed programmable ASICs to efficiently handle variable dataflow, decomposing SumCheck into common primitives—table updates, extensions, and evaluation products—and mapping them onto programmable compute pipelines, allowing the support of arbitrary polynomial structures and gate types.

The above kernels typically account for the vast majority of ZKP compute time, but all ZKP protocols also rely on a variety of smaller kernels to stitch these major components together. In order to avoid Amdahl’s Law bottlenecks, it is essential ZKP hardware accelerates these kernels as well.   

The Future of ZKPs

ASIC acceleration of ZKPs has evolved from fixed-function modules towards reconfigurable and programmable datapaths. Most existing accelerators remain specialized to one or a small number of protocols, while recent work has begun to support multiple protocols. Recent characterization studies on CPUs and GPUs show that execution bottlenecks, memory behavior, and scalability vary substantially across protocols and implementations. As ZKP protocols continue to evolve, there is growing interest in more general architectures that can accommodate its heterogeneous kernels, arithmetic requirements, and execution patterns. Understanding how different protocols stress the hardware stack remains an active area of research. 

To learn more and join the conversation with architects working to make real-time, deployable ZKPs a reality, join us at the ZKARCH workshop at MICRO 2026!

About the Authors:

Alhad Daftardar is a Ph.D. candidate in Electrical and Computer Engineering at NYU. He received his BSEE from Georgia Tech and his MSECE in VLSI from the University of Michigan. His research interests are in computer architecture for Zero-Knowledge Proofs and privacy-preserving computation.

Brendan Sweezy is a Ph.D. student at New York University studying Electrical and Computer Engineering. He received his BS from Duke University. His research focuses on hardware acceleration for Zero-Knowledge Proofs, particularly targeting programmable and edge-scale devices.

Brandon Reagen is an Assistant Professor of Electrical and Computer Engineering at New York University with a focus on computer architecture. His work has been recognized with the ASPLOS’25 Best Paper Award, multiple best paper nominations (DAC, PACT, HASP), and multiple Top Pick/honorable mention awards. He has been a performer (and PI) on the DARPA DPRIVE, PROWESS, and COOP programs, was recognized as a DARPA Riser, and won the NSF CAREER award. He is a lead PI on the NSF CIRC Grand Cryptolets project and has generously received support from Google, DTCC, and AMD.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

LLM execution requires moving gigabytes of data, which includes static model weights, the generated KV cache, and other intermediate representations. Depending on the deployment, this may involve communication across the network, PCIe links, accelerator links (e.g., NVLink), and memory interconnects, often leaving LLM execution bottlenecked by the data movement feeding it.

An obvious mitigation for this overhead is to simply reduce the amount of data transferred. Quantization is a field of research that adopts this principle, where tensors are shrunk to a smaller data format, reducing the space and interconnect traffic needed for inference. There’s a rich body of work examining quantization’s usefulness on different LLM components like weights, activations, attention, and the KV cache.

While quantization has earned its place in the field, it is difficult to universally deploy. Model accuracy degrades due to loss of information, requiring quantization to be tuned for specific models and use cases of LLMs. It has been shown that even when model accuracy appears to be unaffected, side effects manifest in unpredictable ways, such as increased rambling from quantized LLMs. The impact of quantization varies considerably across model architectures and workloads, making it difficult to guarantee consistency after deployment.

Lossless compression: Reducing data without losing information

Is it possible to reduce data volume without compromising on accuracy? Lossless LLM compression is an emerging line of work that demonstrates this. All information is retained, preserving model accuracy, while data volume is also reduced. While necessarily providing less compression than quantization, the reduction in volume is significant (1.4x average data reduction).

Lossless compression is not without challenges. Tensors have to be reconstructed, leading decompression to lie along the critical path, potentially overriding the benefits of reduced data transfers. The floating point values contained in tensors can also be difficult to compress. For example, outlier values must be preserved, requiring additional tracking metadata.

Fortunately, structure within LLM tensors provides compression opportunities. Models typically use BF16 data type, containing 1 sign bit, 8 exponent bits, and 7 mantissa bits (Figure). The values of the mantissa bits follow a fairly random distribution, but the exponent bits exhibit structure. While 8 bits are provided, usually only ~3 bits carry useful information. The other 5 bits are low entropy. 

Figure. Exponent bits for BF16 LLMs are highly compressible. Figure reproduced from DFloat11 paper with authors’ permission.

A few lossless compression schemes have emerged that take advantage of this structure, demonstrating improvements for LLM inference. These schemes are developed for GPUs, as they are typically the processor of choice for LLM execution due to their high throughput. The mechanisms described here partition the compressed stream into independent units so that thousands of GPU threads decompress in parallel, generating significant decompression throughput. The schemes minimize metadata to promote GPU cacheability and avoid extraneous overhead of metadata accesses during decompression. 

For example, DFloat11 targets cases where LLM weights exceed GPU memory capacity. Excess weight tensors are kept in CPU memory and transferred across PCIe as needed, this is known as offloading. The key idea is to use offline Huffman coding to compress the exponents ofBF16 weights, replacing these 8 bits with fewer bits that act as lookup keys. Compact lookup tables are generated that fit within the GPU’s shared memory. Leveraging DFloat11 lossless compression, model sizes are reduced by ~30%, allowing a 48 GB model to fit in a 40 GB GPU. This translates to 2 – 40x faster inference over HuggingFace Transformers, as data is no longer moved across PCIe.

Going further, ZipServ investigates data movement that occurs within the GPU, specifically, when bringing model weights from the GPU memory into tensor cores before performing matrix multiplication. Similar to DFloat11, ZipServ replaces the 8 exponent bits of BF16 LLM weights with 3 bits, compressing the top 7 most frequent exponent values offline. The remaining exponents map to a special value, which indicates the exponents are stored in full form in a separate location. Fused matrix multiplication kernels fetch and decompress directly into tensor cores, avoiding the memory round trip typically required for decompression. This increases throughput by 22% on average over vLLM.

Our work, Invariant Bit Packing (IBP), generalizes compression and makes it data-type agnostic. It enables not only runtime decompression, but also compression, extending the use case to include the runtime-generated KV cache. IBP discovers invariant bits, i.e., repeated low-entropy bit values across tensors, and stores these bits as metadata in the GPU, small enough to fit in GPU shared memory. Identifying invariant bits across a small subset of tensors is sufficient to generalize to the full set. When compressing, these bits are removed from the tensors, while decompression adds the invariant bits back using cheap bit shift operations. IBP achieves an average of 24% faster LLM inference over FlexGen and InfiniGen, by decompressing tensors on-the-fly while transferring from the CPU. This improves both weight and KV cache transfers, demonstrating the generality of lossless compression.

This topic is making its way to deployment. Cloudflare revealed Unweight for NVIDIA H100 lossless decompression, building on the ideas previously discussed. IBM Research demonstrated ZipNN boasting similar lossless compressibility for LLMs, targeting storage and network transfers. Doubleword.ai has shown that newer FP8 models are also compressible due to the same exponent redundancy.

The next frontier: Compressing the full data path

Lossless compression is having its moment for GPU data movement optimization, but the GPU is not the only place where compression can help. In scale-out LLM deployments, model and computations are split across multiple axes (data, tensor, sequence, pipeline, and expert) referred to as 5D parallelism. Each axis can add new data traffic. For example, tensor parallelism adds partial activation movement, incurring overheads of 20% on NVLink transfers in multi-GPU deployments. Deploying across multiple machines exacerbates this problem, as these rely on lower bandwidth network connections. Further, expert parallelism, where specific LLM layers are split across GPUs, requires all-to-all communication to move intermediate tokens, incurring 40% overhead. All of this movement could be compressed.

A fundamental question is where the decompression should happen. Using the GPU for decompression can steal cycles from other concurrent compute tasks. Conversely, if we decompress early, e.g., at a network smart switch or SmartNIC, the data grows before crossing PCIe or NVLink, losing benefits for downstream links. Depending on the deployment, either approach may have acceptable tradeoffs. NVIDIA B200 GPUs demonstrate decompression along the transfer path by including a new hardware decompression engine that transfers compressed data across PCIe and then decompresses without utilizing GPU multiprocessors. It currently supports a limited range of algorithms (LZ4/Snappy/Deflate). How to extend this range and make the engine flexible, is an open question.

Hardware architects have other potential substrates for decompression present within the memory system: processing in or near memory can store data in compressed form in memory, then decompress data as it leaves memory, increasing effective GPU memory capacity without reserving GPU cores for decompression. This allows more data to be kept within GPU memory, conversely reducing the volume of data needed to be transferred across low bandwidth links like PCIe.

Adapting lossless compression for dynamically changing data is another significant open problem. All the schemes we discussed target static data. This works well for inference, where data values (e.g., KV cache entries or weight tensors) are reused across iterations, amortizing compression overhead. Training breaks this assumption. The entire working set changes every iteration, needing repeated loops of compression and decompression. Mitigating this overhead requires lossless compression that can quickly recompress data. More ambitiously, a scheme that supports tensor updates (e.g., gradient descent) on compressed data could eschew the need for recompression entirely.

Conclusion

Lossless compression for LLMs is still in its infancy, but already showing promise. Existing work has demonstrated that LLM tensors can be compressed by ~30%. The overhead of communication in multi-GPU deployments is significant (20 – 40%), providing a prime future use case. However, it requires tailoring compression to the intricacies of the intervening links, and involves identifying the best candidate to perform decompression. This is an open research field ripe for investigation.

About the Authors:

Aditya K Kamath was a Ph.D. student in Computer Science and Engineering at the University of Washington (UW). His work revolved around analyzing data movement and mitigating its performance impact in memory-intensive applications.

Simon Peter is an associate professor in Computer Science and Engineering at UW. His work is in low latency, scalable, and energy-efficient data center systems and cloud application design, across the hardware/software continuum. 

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

AI is starting to shape architectural mechanisms, workloads, and evaluation. To make sense of it, we need a compact, shared way to preserve enough of that process for other groups to evaluate and build on AI-assisted claims.

At the 53rd ISCA in Raleigh, AI for architecture stopped feeling like a side conversation. In the hallways, the talk kept coming back to one thing. AI is starting to enter the architecture design loop, the repeated process of framing a problem, proposing or editing a design, measuring it, rejecting weak candidates, and deciding what to try next.

In particular, there were two deep-dive workshops coupled with other activities. The MLArchSys workshop added A³, a segment on agentic approaches to architecture, and the full-day Architecture 2.0 workshop focused entirely on agentic design. Both drew well over a hundred people and were standing-room-only by the end. The same shift was visible in the main program, in a plenary panel on research and education in the GenAI era.

The Architecture 2.0 workshops at ISCA 2026.

The MLArchSys workshops at ISCA 2026.

Full rooms on a particular subject matter are undoubtedly a sign of community momentum. The opportunity now is to turn that momentum into a durable engineering practice. That will take shared evidence, reusable tools, and enough agreement for a claim to leave the room where it was born and still be checked, compared, taught, improved, or rejected by someone else. AI is already producing architectural ideas. The question is what must travel with those ideas for them to become engineering knowledge.

Suppose a paper reports an AI-generated memory prefetcher with a 15 percent speedup. The code runs, and the speedup reproduces under the reported setup. But the agent saw some workloads and not others, adapted to simulator feedback, tried many candidates, and picked this one. What, exactly, is the contribution here? The final prefetcher? The speedup? The prompt? The agent? Or the process that connected them?

Once AI chooses workloads, responds to feedback, and selects which candidate to report, the same uncertainty about what counts as the contribution reappears for every such result. As AI gains more influence, we have to decide which evidence should accompany a result, so that another group can tell which part actually holds. This blog post is about the evidence that should accompany them if they are to become engineering knowledge.

The Scale of the Shift

The workshops reflect a broader rise in AI-mediated systems research. A recent cross-stack survey of more than 7,800 arXiv papers found that the annual AI-for-systems publication volume grew roughly 23× from 2017 to 2025, and even faster in hardware and chip design (GenAI for Systems). These categories reach beyond architecture, and not every paper in them runs an adaptive design loop. But where AI adapts to workloads, simulator feedback, or selection criteria, the final artifact can obscure how the result emerged. As this body of work grows, leaving that process implicit makes it harder to compare results or carry a finding from one group to the next.

Figure 1: Annual AI-for-systems publication volume grew about 23× from 2017 to 2025 (a). The hardware and chip-design categories grew roughly 43× and 60×, respectively, compared with 21× for software (b).

Recent SIGARCH posts show the field working this out in public from different angles. Karu Sankaralingam asks whether architecture has reached an AlphaZero moment, with evaluation, not idea generation, as the real bottleneck. Dimitrios Skarlatos argues that agentic co-design is already reshaping the architect’s role and the hardware-software contract. Jeff Dean and David Patterson project a 10,000× future built on compounding gains, one of them AI automating hardware design itself. Together, their arguments point toward a common question about what should count as evidence when AI helps produce a design. Answering it requires being precise about what changes when AI moves from a bounded tool to an actor in the design process.

What Changes With Agentic Design

AI for architecture means using learned or agentic systems to help shape architectural designs and the evidence used to evaluate them, rather than building hardware optimized to run AI workloads. This direction, framed in recent work on the foundations of AI agents for modern computer system design, overlaps with software generation and electronic design automation (EDA), but it is distinct from both. An open-ended architecture agent can influence mechanisms, abstractions, workloads, simulator configurations, and interfaces whose effects propagate through many downstream programs and tools. That reach is what makes both its designs and its decisions worth scrutinizing.

Machine learning has entered architecture before. Perceptron branch predictors, reinforcement-learning memory controllers, learned prefetchers such as Pythia, surrogate models that approximate expensive simulations, and autotuners that automatically search configuration choices all used statistical learning to sharpen a mechanism or search a design space. In much of that work, ML was part of the artifact or a bounded optimizer, while the workloads, evaluator, and rules governing the search were set outside the model. The agentic shift is not a clean break from autotuning. It expands the scope and authority of the adaptive process. When a system can propose or edit mechanisms, call tools, choose workloads, adapt to simulator feedback, and influence which candidate survives, ML is no longer only inside the design. It starts to shape the claim we make about the design.

This concern predates AI. Human researchers explore design spaces, tune systems, and discard candidates, too, and research has always run on authors disclosing what others need to judge the work, backed by a degree of trust. What changes with an agent is how we scale. An adaptive system can make and revise these choices at machine speed across mechanism code, simulator configurations, workloads, tool calls, and selection criteria, often in response to the same evaluator that later supports the claim. The issue is not that a choice made by an AI system is inherently less trustworthy. It is that a large, tool-mediated search collapses into a final mechanism and a score, and the path that produced them disappears unless someone deliberately records it. The end goal is not to eliminate trust, but to keep that part of the methodology visible enough for others to assess the claim.

Gupta and colleagues’ ArchAgent makes this concrete by designing and implementing cache-replacement policies, not just their parameters. Starting from Mockingjay, a prior state-of-the-art policy, ArchAgent generated Policy31 for the single-core SPEC CPU 2006 suite, with a usage-intensity mechanism that its authors could inspect and test feature by feature. It also generated Policy12, which appeared to beat Mockingjay through what the paper calls a simulator escape, a higher score won by exploiting the simulator rather than the architecture. In ChampSim, unsupported bypassing of last-level cache writes was protected only by an assertion that optimized builds removed, so Policy12 looked faster because the bypassed writes vanished rather than being handled correctly.

The same agentic process produced both a genuine mechanism and a broken measurement, and the reported scores alone would not tell a reviewer which was which. The authors, to their full credit, caught the escape through manual inspection and reported it, exactly the kind of evidence future studies should preserve. A fuller record would not have found the bug automatically, but preserving the build configuration, the rejected policy, and the check that disqualified it would let others see why Policy12 failed and reuse that check in the next study.

Figure 2: (Left) Artifact-only review sees a generated design and a reported number. (Right) Design loop-aware review keeps the artifact in view while adding the declared bounds, the evidence and failures, and a record of who could accept or reject the candidate.

ArchAgent also helps show where Karu Sankaralingam’s AlphaZero comparison holds and where architecture departs from it. AlphaZero discovered powerful Go strategies through self-play, but the board and the rules stayed fixed. Only the strategy could change. Depending on its permissions, an architecture agent can influence the strategy, the board, the rules, and the score used to judge it. Benchmarks can become data the agent adapts to, simulators can become environments it acts on through tool calls, metrics can become optimization targets, and interfaces define what actions it can take. That is why the claim must carry a record of the “design loop,” not just the artifact that emerged from it.

What Evidence to Preserve

Architecture papers already describe mechanisms, baselines, workloads, simulators, and evaluation procedures, so this is not a call for longer methods sections. What they rarely preserve is how the search reached the reported design. A compact record would make a few things visible:

• Bounds: what the agent could see and change, and what stayed fixed
• Feedback: how much simulator feedback it drew on, and how it steered the search
• Evidence and failures: what supported the reported result, and which candidates were rejected and why
• The decision: who could reject a candidate, who made the final call, and what would overturn the result

None of it is exotic. It is the part of the process that an adaptive search tends to erase.

Machine learning has already faced a version of this gap. A released model or dataset often lacked sufficient context to assess its intended use, evaluation, or provenance. In response, model cards and datasheets for datasets provided the field with compact records that accompany the artifact, short enough to read yet specific enough to state what the work does and does not support. Architecture needs the same kind of record, extended from a finished artifact to the search that produced it.

The reporting burden should scale with how much authority the agent had. If AI only helped implement a mechanism specified by a human, ordinary artifact disclosure is probably enough. If it chose workloads, edited the design, adapted to simulator feedback, or determined which candidate was reported, some account of that process should accompany the result. A simple test is whether AI materially shaped the mechanism, workload, evaluator, stopping rule, rejection rule, or reported result. That record need not become a universal checklist, expose the model’s private reasoning, or promise an exact replay of a randomized search. Its format should emerge through use and revision rather than being settled in advance. What matters is how much of an AI-shaped process must stay visible for another group to see why a result survived and whether it holds under different assumptions.

Existing practice offers only partial precedents. Declaring the setup before a search resembles preregistration, keeping the evidence trail resembles artifact evaluation, and preserving failed alternatives resembles ablation studies, which test the effect of changing one part of a design, as well as negative result reporting. The individual practices are not new. The change is that a single adaptive system can operate continuously across the design, workload, evaluator, and stopping rule, which are usually documented separately.

A record like this is a good-faith disclosure, not proof. An author can omit an inconveniently rejected candidate, and a reviewer cannot rerun an adaptive search to catch the omission, especially when the agent relies on a proprietary model that shifts over time and never repeats a run exactly. The record cannot stand on its own. What keeps it honest is disclosure scaled to the agent’s authority, read by reviewers rather than filed as a badge, and confirmed against evidence the search did not produce.

A result selected through adaptive evaluation should face at least one confirmation check outside the search, using held-out workloads, a second simulator, or a targeted test of the claimed mechanism. If the same agent tunes against the simulator that scores it, selects its evaluation workloads, and stops once the metric looks good, the evaluator has become part of the optimization loop, the architecture equivalent of training on the test set, or evaluation leakage. The check must also use measurements appropriate to the claim, and as a recent SIGARCH post on full-system timing simulation argues, simulation speed and fidelity are already in tension before an agent begins optimizing. Agent feedback makes the measurement window and metric part of the search surface, so authors should explain why the confirmation is credible and what evidence would overturn the result.

The same shift that put agents into the design loop is now putting them into the review loop. In systems research, agents already drive the design loop end-to-end, and elsewhere they draft and review their own papers, with a language model serving as the judge. An automated judge can share the blind spots of the system it reviews, so it does not replace the independent check. But it does raise the value of a record built to be machine-readable as well as human-readable, one that the next agent can use to rebuild the setup, rerun the disqualifying check, and test the claim rather than take a summary on faith.

A Shared Layer for the Loop

A record inside a single paper is a good start. It becomes a shared convention when authors use common fields and present supporting evidence in a form others can inspect. A reviewer can then challenge the record, a student can learn why the reported design survived, and another group can revisit a rejected candidate under the same conditions. Some variation in this scaffolding is healthy, but shared infrastructure gives groups a common base without requiring them to pursue the same research questions.

The computer architecture community has previously built shared responses to analogous coordination problems. SPEC provided common workloads, while simulators such as SimpleScalar and gem5 provided researchers with reusable experimental platforms. MLPerf and the long line of prediction and prefetching championships showed that we can agree on workloads, rules, and scoreboards. These shared objects did not settle every question, but they gave the field durable things to run, dispute, teach, and improve. Benchmarks do not capture the path through an adaptive search, but they show how common boundaries make comparisons meaningful. Agentic design now needs a similar layer for search state, allowed actions, failures, and independent checks.

When Amir Yazdanbakhsh and I first articulated the Architecture 2.0 vision in a SIGARCH blog post in 2023, it was conceived as a data-centric AI gymnasium, a shared ecosystem of data, benchmarks, and tools for ML-assisted architecture research. Three years later, many of those building blocks have emerged, including knowledge benchmarks such as QuArch, assembled with the help of more than 140 contributors across 40 institutions, capability evaluations such as ArchEval, and design-space infrastructure such as ArchGym. Building them has taught us something more important. The remaining challenge is not simply another benchmark, evaluation, or piece of infrastructure. A benchmark tests what a model knows, an evaluation tests what an agent can do, and infrastructure runs the search. Some of these systems log a run in detail, but that record stays inside the tool. What no published result yet carries with it is a portable account of how a study bounded its search, rejected candidates, and chose what to report, and that is the part we cannot supply by building one more tool.

Making It Routine

Making this kind of evidence part of everyday research practice will take deliberate community effort. Machine-learning communities have shown one path. Benchmarks and competitions provide participants with shared tasks and rules, while model cards and datasheets establish shared reporting expectations. NeurIPS requires submissions to include a paper checklist addressing reproducibility, transparency, limitations, and experimental details, a step it introduced to help authors document the completeness and limits of their work. ICML has likewise published paper guidelines, based on the NeurIPS checklist, that ask authors to document claims, limitations, code, data, and experimental details. A complementary perspective appears in the 2022 FAccT paper by Smith and colleagues on REAL ML, which argues that responsible machine learning depends not only on models and metrics, but also on documenting the broader research process.

Architecture conferences and workshops could experiment with a few concrete practices. Artifact-evaluation tracks can request versioned configurations, failures that affected the result, and at least one confirmation check that was not used to select the reported result. Competitions can specify workloads, allowed actions, limits on evaluator queries, stopping rules, and held-out tests to ensure scores remain comparable. We do not need to harden these practices into permanent rules at once, and venues can learn what helps reviewers and discard what does not. For proprietary work, the detailed record may remain internal, but a public claim still requires sufficient disclosure for outside groups to assess it. The simplest shared form for that disclosure is a single page. Authors could include or link to a one-page design-loop card summarizing the process in a consistent format.

Figure 3: One possible one-page record makes the bounds, actions, feedback, evidence, failures, and final decision visible.

These conventions also need a public home alongside tools, benchmarks, failure cases, and examples. The Architecture 2.0 hub is one possible starting point. Its value will depend on whether multiple groups use, challenge, revise, and help govern its contents.

No single convention will make AI for architecture an engineering discipline. The value of a shared record is that it lets a result move beyond the group that produced it so someone who was not there can check it, build on it, or challenge it when the evidence does not hold. The full rooms at ISCA were the momentum. Making the evidence travel with the design is what turns momentum into a discipline.

About the Author

Vijay Janapa Reddi is the Gordon McKay Professor of Electrical and Computer Engineering at Harvard University and a visiting professor at ETH Zurich. His work spans computer architecture, machine learning systems, and autonomous agents. He is Vice President and a board member of MLCommons and the author of the open-source Machine Learning Systems book.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

The conference

The 53rd International Symposium on Computer Architecture (ISCA) was held at the Raleigh Convention Center in Raleigh, North Carolina, from June 27 to July 1, 2026. Raleigh sits at one corner of the Research Triangle, anchored by North Carolina State University, the University of North Carolina at Chapel Hill, and Duke University. General Chairs Huiyang Zhou and James Tuck, both of NC State, led the organizing effort.

The most notable structural change this year was that ISCA offered remote attendance, making it a hybrid conference. The organizers provided deeply discounted remote registration to broaden access for students and researchers who could not travel, broadcast the main and keynote sessions on Zoom, and made recordings available to registrants for offline viewing. This was ISCA’s first hybrid offering and an experiment intended to lay groundwork for remote attendance at future architecture conferences.

Workshops and tutorials

Preceding the main symposium, ISCA 2026 opened with two full days of workshops and tutorials on Saturday, June 27 and Sunday, June 28, organized by Workshops and Tutorials Co-Chairs Lisa Wu Wills (Duke) and Brandon Reagen (NYU). The program totaled 15 workshops and 16 tutorials, spanning the full breadth of the field, from open-source infrastructure and DRAM to quantum computing, encrypted AI, and agentic design.

Saturday’s tutorials leaned on open-source and simulation infrastructure, such as ASTRA-sim, the Ramulator and DRAM Bender memory tools, and FAVA on formal hardware verification, while the workshops included gem5, SAFE AI on encrypted AI, and Architecture 2.0 on agentic AI for computing-systems design, which marked the launch of the book Architecture 2.0: Agentic Design Loops for Computing System Synthesis. Sunday leaned into mentoring, open-source hardware, and quantum: the uArch Mentoring Workshop and YArch’26 for students; XiangShan and the SODA Synthesizer on the open-source side; and FHE & Cheddar and Janus 4.0 for quantum, alongside the 6th DRAMSec, a carbon-accounting tutorial, and the Championship in Branch Prediction.

Panel on the Impact of AI on Higher Education & Computer Architecture @uArch 2026

The main program

This was the largest ISCA program ever. Program Co-Chair Carole-Jean Wu (FAIR, Meta) and Kevin Skadron (University of Virginia) reported 850 regular-track submissions, a 49% increase over the previous year, of which 161 were accepted, for an 18.9% acceptance rate (down from 23% the year before). To accommodate the volume, ISCA ran a fourth parallel track for the first time. 

The reviewing operation scaled to match. It involved 22 Area Chairs, 211 full PC members, and 192 lightweight PC members, by far the largest committee in the conference’s history. Reviewing ran in two rounds, with most papers reaching six reviews. Discussion followed the “Identify the Champion” model; 301 papers reached a clear online consensus, while the remaining 59 were resolved in a series of real-time Zoom PC meetings held over five days. In the end, 116 papers were accepted outright and another 45 were conditionally accepted with shepherding, all of which were eventually accepted.

Keynotes

ISCA 2026 featured three keynotes.

Debbie Marr (CEO and Co-Founder of AheadComputing) opened with “Computing at the Crossroads: Architecture, Economics, and the Next Era.” She reflected on the trajectories that shaped the field: Moore’s Law, Dennard scaling, increasing abstraction, and the long expansion of general-purpose computing. Many of those assumptions, she observed, are now being questioned simultaneously. She tied the technical inflection point to shifting economics, ecosystem dynamics, and leadership transitions, and suggested that the architecture community’s choices today will define the next era of computing.

The second keynote piloted a new “dialogue” format on quantum computing, pairing Fred Chong (University of Chicago; Chief Scientist for Quantum Software at Infleqtion) and Jay Gambetta (IBM Fellow and Director of Research) for “Architecting Hybrid Quantum-Classical Computing for Scale and Fault Tolerance.” Their shared theme: with fault-tolerant machines on the horizon and near-term machines increasingly integrated with classical HPC, computing will be heterogeneous and accelerator-based, and architects are needed to bridge theory and physical technology across applications, software, error correction, workflow management, and machine organization.

Babak Falsafi (EPFL) closed the keynote lineup with “Beyond the AI Energy Wall: Optimal Server Design and Operation”. He described how AI is pushing cloud infrastructure toward an energy wall, with compute demand growing faster than power, cooling, and datacenter capacity can be sustainably provisioned, and suggested that clearing it requires full-stack optimization rather than simply scaling accelerators or building larger facilities. He questioned the long-standing assumption that single-thread performance should dominate server design and operation.

Keynote by Debbie Marr (Computing at the Crossroads)

Awards

A number of the community’s honors were presented during the conference.

The ACM/IEEE-CS Eckert-Mauchly Award went to Srinivas Devadas (MIT) for pioneering contributions to secure architectures with broad industrial and academic impact. The ACM SIGARCH Maurice Wilkes Award was presented to Tushar Krishna (Georgia Tech) for outstanding contributions to architectures and modeling tools for large-scale AI systems. The TCCA Young Architect Award went to Akshitha Sriraman (Carnegie Mellon University) for contributions to the design and management of efficient and sustainable cloud datacenters. The ACM SIGARCH/IEEE CS TCCA Outstanding Dissertation Award went to Olivia Hsu (Stanford University), with an honorable mention to Jovan Stojkovic (University of Illinois Urbana-Champaign). The SIGARCH Alan D. Berenbaum Distinguished Service Award was presented to Sarita Adve (University of Illinois Urbana-Champaign) for sustained and transformative contributions to ACM SIGARCH, the broader architecture community, and via CARES, the ACM SIG ecosystem. The ISCA Influential Paper Award recognized “Adaptive Insertion Policies for High Performance Caching” (ISCA 2007) by Moinuddin K. Qureshi, Aamer Jaleel, Yale N. Patt, Simon C. Steely, and Joel Emer, for its commercial impact and for reinvigorating research on cache management with an elegant set-dueling framework that can be broadly applied to cache-policy selection.

Two ISCA Best Paper Awards were selected from a field of five nominations: “Cerberus: Cross-Layer ECC Co-Design for Robust and Efficient Memory Protection” — Junhwan Kim, Seunghyun Kim, Yesin Ryu, Saeid Gorgin, and Jungrae Kim. “Patterns Behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference” — Zhongkai Yu, Yue Guan, Zihao Yu, Chenyang Zhou, Zhengding Hu, Shuyi Pei, Yangwook Kang, Yufei Ding, and Po-An Tsai.

Two ISCA Distinguished Artifact Awards were also recognized: “Transpiler-Architecture Co-Design to Curb Clifford Costs in Fault-Tolerant Quantum Computing” — Meng Wang, Chenxu Liu, Samuel Stein, Yufei Ding, Poulami Das, Prashant Nair, and Ang Li. “Towards Practical Interrupt Side-Channel Attacks on macOS for Apple Silicon” — Xin Zhang, Chang Liu, Jiajun Zou, Yi Yang, Qingni Shen, Zhi Zhang, and Trevor E. Carlson.

ISCA Influential Paper Award recipients

Industry track and artifact evaluation

The Industry Track, chaired by Brad Beckmann (AMD), accepted 11 papers out of 27, reviewed by a 32-member committee drawn entirely from industry across a diverse set of startups and established companies. The accepted set ranged from silicon to software. Two additional papers were recommended for an IEEE Micro Special Issue on Commercial Products.

Artifact Evaluation, in its fourth year at ISCA, received 49 submissions. 42 papers earned all three badges (Available, Functional, and Reproduced), 3 earned Available and Functional, and 4 earned Available. The co-chairs Hyeran Jeon (UC Merced), Linghao Song (Yale), and Mark Zhao (University of Colorado Boulder) flagged a growing challenge: the increasing heterogeneity of hardware and software platforms, which reviewers do not always have access to.

A snapshot of the Industry Track session

Excursion

ISCA’s excursion was an evening dinner and social at Raleigh’s historic City Market. Built in 1914 and known for its cobblestone streets and early-twentieth-century lamplight, the district hosted a relaxed, open-air affair, with food stations of North Carolina–inspired dishes, beer, and wine spread across the historic Market Hall, The Grove, and the outdoor spaces between them. With attendees spilling across the market, it made for an excellent networking opportunity and a welcome chance to unwind and catch up with people after the intensity of the technical program.

Excursion venue: City Market

About the author: Bingyao Li is an Assistant Professor in the Computer Science and Engineering Department at the University of California, Riverside. Her research focuses on designing architecture and system features for next-generation GPU platforms and building high-performance LLM infrastructure and systems.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Accurate timing simulation remains one of the most important tools in computer architecture, but modern systems have made cycle-level simulation increasingly impractical. Today’s platforms combine many-core CPUs, deep memory hierarchies, accelerators, complex I/O, and large software stacks, making detailed simulation extremely slow—often requiring months to simulate seconds of execution. This “timing simulation wall” has pushed researchers toward approximations such as application-only simulation, fixed instruction windows, or instruction windows representing only the workload. While these reduce runtime, they often sacrifice rigorous end-to-end measurement of real microarchitectural behavior.

This blog argues for a return to rigorous full-system timing simulation—not by simulating everything in detail at all times, but by measuring the right execution intervals, using the right performance metrics, and applying statistically sound methods to make accurate simulation practical again.

Why Full-System Simulation?

Full-system simulation emulates an entire computer system: CPU, memory, devices, operating system, and applications. Unlike user-level simulation, it captures interactions across the full software and hardware stack. Full-system simulation matters because many critical behaviors emerge from OS activity, interrupts, I/O, memory management, synchronization, and device interactions—not from application code alone. Ignoring these layers can misrepresent real system bottlenecks and performance.

Full-system simulation dates back to the 1990s with systems like SimOS, later influencing platforms such as Simics (now Intel Simics Simulator), M5 (integrated into gem5) and QEMU (used in MARSS and QFlex).

Today, full-system simulation is becoming essential again for four reasons:

1. Modern workloads are service-oriented and multi-tenant, relying on microservices, RPCs, storage stacks, and OS-mediated interactions.
2. Many server and mobile workloads spend significant time in the OS, making kernel behavior central to performance analysis.
3. Heterogeneous systems increasingly combine CPUs with GPUs, accelerators, and smart NICs, with the CPU and OS orchestrating coordination, memory, and synchronization.
4. Agentic AI workloads depend heavily on tool invocation, scheduling, APIs, databases, and system integration, making CPU and OS behavior critical to end-to-end performance.

As a result, full-system simulation is no longer just a legacy methodology—it is increasingly necessary because the entire system stack has become the target of architectural innovation.

The Timing Simulation Wall

Simulators span a broad spectrum of abstraction, functionality, and performance. At the fastest end are execution-driven full-system simulators that use JIT translation to dynamically map target ISA instructions into the host ISA at runtime. Since early systems such as SimOS, these simulators have typically operated within roughly an order of magnitude of native hardware speed.

Modern ISA emulators such as QEMU can additionally generate detailed execution traces for functional simulation, enabling analysis of cache and TLB miss rates, branch predictor behavior, and prefetcher accuracy. This tracing introduces another order-of-magnitude slowdown relative to native execution.

Timing simulators go further by modeling cycle-level interactions among microarchitectural components in the CPU, accelerator, memory and I/O devices resulting in substantially lower simulation throughput. The table below compares simulation speeds for a single ARM Neoverse N1 target core with its cache hierarchy running server workloads on an AMD Zen 3 host.  The first row presents QEMU’s raw ISA emulation speed. The second row shows the slowdown due to instrumentation for user-level functional simulation. The third row demonstrates the impact on speed when functionally simulating the microarchitectural components, including the cache hierarchy and TLBs, front-end tables, and data prefetcher, for all user-level instructions. The fourth row shows the impact of functional simulation of all instructions, including the OS. Finally, the fifth row shows the timing simulation speed.

Modern workloads are not steady streams of similar instructions. Their performance fluctuates over time due to network activity, resource contention, background OS activity, synchronization effects, software hiccups, DVFS throttling, UI and graphics activity, and other asynchronous events. Alameldeen et al. presented a statistically rigorous methodology to determine the minimum measurement window needed to capture workload performance variability within a specified error bound and confidence level. 

Unlike conventional database workloads (e.g., TPC benchmarks) which have prescribed measurement windows, typical benchmarks and workloads used in research do not. Applying Alameldeen’s methodology, we find that capturing performance variability for a single ARM Neoverse N1 core and its cache hierarchy requires five to 120 seconds of target execution time across server workloads from CloudSuite, DCPerf, and DeathStarBench. Simulating even a few seconds of a single core with today’s fastest cycle-accurate simulator, gem5, at 250 KIPS requires months of simulation time.

What Should We Measure?

The second question is which performance metric to use. Timing simulators count cycles, so architects often report IPC, or instructions per cycle. IPC is reasonable for single-core workloads when most executed instructions correspond to program progress.

For multicore workloads, however, IPC can be misleading. Threads may spin, poll, block, wait on locks, synchronize, or execute OS code that does not advance useful work. A system can therefore sustain high IPC while making little forward progress; in effect, total IPC can reward busy waiting. 

This is why user-level IPC, or U-IPC, is often a better proxy. U-IPC counts user-level instructions over time, assuming that user instructions per request remain roughly stable and that most spinning occurs in the OS. Under that assumption, U-IPC tracks useful throughput more directly than total IPC.

But U-IPC must be validated for each workload. If spinning occurs in user space, as in systems with user-level network stacks, raw U-IPC still counts non-productive work and must be corrected to exclude spinning. The broader requirement is therefore metric validation: a rigorous simulation methodology must show that the chosen metric—IPC, U-IPC, throughput, or latency—actually captures forward progress for the workload under study.

How Should We Measure?

Due to the timing simulation wall, researchers often use abbreviated measurements. The most common technique is to measure a single unit of 100 million to one billion instructions. Unfortunately, depending on where in the execution the fixed measurement is taken from, this technique may lead to inconclusive results or worse, incorrect conclusions. 

Instead, designers often use sampling to capture variability in performance estimates. Phase-based sampling, such as SimPoint, is a popular technique that uses clustering of basic-block vectors (BBVs) to select representative application “phases.” Such sampling properly captures the representing repetitive instruction streams that account for most of the execution. 

While simple and practical, phase-based sampling may ignore OS effects, interrupts and I/O interactions, communication among cores, and software hiccups. Moreover, Wunderlich argues in his thesis that phase-based sampling: (1) misses the microarchitectural footprint of less common instruction streams and their impact on performance, and (2) forgoes any error bounds with confidence in estimates.

A rigorous sampling technique is statistical sampling, such as SMARTS, taking a large sample (e.g., hundreds) of small (e.g., 200k cycles), uniformly distanced measurement units that is representative of execution, not phases in the workload. This technique enables bounding the error in estimates and delivers quantifiable confidence. It also opens an entire plethora of statistical sampling tools to trade off confidence in estimates for measurement in speed and quantify sample divergence to detect bias in estimates.

The figure below compares error magnitude in performance estimates among three abbreviated measurement techniques from full-timing simulation runs of tens of target seconds on a two-core socket with 2.0 GHz ARM Neoverse N1 cores running single-tier, multi-tier and consolidated server workloads (CloudSuite, DCPerf and DeathStarBench). The figure compares the error against the full-timing baselines for: (1) single units of one billion instructions per core starting from three equally distanced positions in the minimum measurement window (i.e., beginning, 1/3 and 2/3 into the population), (2) units of 100 target microseconds (i.e., 200k cycles for a 2.0 GHz clock) including basic-block vectors (BBV) derived from K-means clustering, and (3) a uniform sample (of hundreds) of 100 target microseconds drawn with an error bound of 5% with 95% confidence with statistical sampling. 

Both one-billion instruction units and BBV result in high error estimates with the former not being representative of execution and the latter representing only frequently executed instructions. In contrast, statistical sampling results in a desired error bound with confidence because it represents not just frequently executed instructions but also instructions that have a high impact on performance due to their microarchitectural footprint.

A SOTA Sampling Framework

The figure below presents a state-of-the-art sampling framework using QFlex 3.0 (derived from SimFlex) for full-system timing simulation of ARM ISA. For each workload, the software stack together with the OS is first loaded and warmed on a real platform, then tested to identify the minimum window—such as five to 120 target machine’s seconds—called a “population”, using Alameldeen et al.’s technique. The workload is then loaded again, this time with QEMU and run through a functional simulator running on average at 6 MIPS for the entire duration of population. 

The functional simulator simulates all microarchitectural components with long-term state (e.g., cache hierarchy and TLBs, branch tables, data prefetcher) and periodically dumps checkpoints with architectural and microarchitectural state into a checkpoint library. Because the functional simulator is not cycle-accurate, it requires an approximation for time. The most common approximation is IPC=1 or IPC derived from neighboring units.

The checkpoints in the library are then run using a timing simulator for 100 us independently and embarrassingly parallel. Each checkpoint is first run for a bounded window of time (e.g., 200 us) to make sure microarchitectural components with short-term state (e.g., buffers in the pipeline, cache hierarchy and NoC) are warm, followed by a measurement. The timing results for the sample are then aggregated to determine whether the sample (i.e., number of checkpoints) is large enough to bound the error for a desired level of confidence (e.g., 5% with 95% confidence). If not, the sampling framework creates a new checkpoint library with a shorter interval between checkpoints.

Challenges and Open Problems 

Even with accurate measurement techniques, there are fundamental challenges with sampling (for both phase-based and statistical sampling).

1. Accurate state generation. Timing-induced activity during functional simulation and its impact on the microarchitectural footprint may result in a significant bias because time is approximated. This challenge is more pronounced with variable performance among target threads in multi-tier and consolidated workloads where the speed bias may significantly impact the resulting shared microarchitectural footprint.
2. The functional simulation wall. Sampling minimizes the required measurement using timing simulators but shifts the bottleneck to the functional simulator (which at 6 MIPS is only 24x faster than a 250 KIPs timing simulator). Parallelizing functional simulation may be a promising approach to enable scalability with multicore hosts. Parallel simulation is fundamentally limited by the granularity at which target threads communicate.
3. Support for checkpointing. Generating and restoring an entire checkpoint for every measurement is impractical in both storage capacity and runtime overhead. Practical sampling therefore requires incremental checkpoint storage and restoration.
4. Sampling non-average metrics. Statistical sampling works well for average-like metrics such as IPC or U-IPC, but it is harder to apply to extreme or rare-event metrics such as maximum temperature, worst-case power, or rare latency spikes.
5. Capturing service-level metrics. Metrics such as request latency or p99.9 latency are much coarser-grained than sampling units needed for IPC or U-IPC. Capturing service-level metrics and tail latency may require an order of magnitude larger population and sampling units which poses a challenge for both functional and timing simulation.
6. Multi-node full-system simulation. Many modern workloads are distributed across multiple machines. Single-node simulation is often insufficient for datacenter-scale behavior, but despite progress, rigorous  multi-node full-system timing simulation remains an open challenge.
7. Interoperability across simulators. A practical ecosystem should allow one tool to generate a checkpoint library and another to perform timing simulation. This interoperability requires an interface definition language allowing interoperable architectural and microarchitectural state among simulators.

About the Authors

Shanqing Lin is a final-year PhD student at the School of Computer and Communication Sciences at EPFL and the principal developer of QFlex v3.0.

Mohammad Alian is an Assistant Professor in the Electrical and Computer Engineering Department at Cornell University.

Babak Falsafi is a Professor in the School of Computer and Communication Sciences at EPFL (epfl.ch) and the founding President of Swiss Datacenter Efficiency Association (sdea.ch).

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

What does speculative execution in a processor — and the predictor that drives it, such as a branch predictor — have to do with AI agents? They may seem very different, yet, at a high level of abstraction there are similarities.

Both speculate: A processor predicts which way a branch will go and begins executing instructions along the predicted path before the branch has resolved. An AI agent infers a user’s intent, reads/writes files, executes programs, makes network calls etc., before it knows whether its interpretation of the user’s intent is right.

Because prediction can fail, both systems require roll back mechanisms. In a processor, once a misprediction is detected, the wrong path work disappears (from the programmer’s point of view). When the agent is told it is wrong, or figures that out itself, it may revise its plan and redo the work after rolling back to a good checkpoint.

Both systems litter and leave residues: While a processor can recover from a misprediction without any programmer visible effects, under the covers, microarchitecturally, wrong path execution perturbs on chip structures like caches. The Spectre attack (2018) showed that this residue can be observed through covert channels. AI agents have a similar problem. When an agent, or its human user, notices a mistake and corrects it, the failed attempt can leave at least two types of residues: a) residue that is easily observable like bad outputs, stale files or processes, or b) harder to know/track/undo residue like timing and volume of network requests, model context summaries shipped to third party servers to name a few.

Also both systems can be tricked and steered through adversarial inputs: In Spectre, the attacker influences the on chip predictor state by executing a pattern, then supplies an adversarial input that causes the victim to transiently execute along the trained path that it should not take architecturally. While that transient execution is later squashed the microarchitectural residue of the execution can still be measured. Malicious prompts can play a similar role in AI agents: they can steer the system toward actions that are later corrected or denied, but in the process may leave litter data that attackers can use.

Given these similarities, we can ask two questions.

1) Can AI agents completely eliminate easily observable “architectural” residues on mispredictions?

2) What are the dangers/risks of hidden “microarchitectural” residue left behind by AI agents?

Regarding architectural residue, processors can hide speculative wrong path work cleanly because the ISA defines what counts as visible committed state. Currently there is no equivalent for AI agents: the absence of an interface that can precisely define operations, state, life time of state, and triggers for misprediction recovery, makes these systems hard to reason about and a fertile ground for leakage.

While observable residue is a serious problem it is also a solvable problem to some degree: if one is satisfied with imprecise, best effort work, one simple thing to do is to just prompt the agent to clean up after itself. A really smart agent, in theory, should be able to use mechanisms like transactions, two phase commit, distributed undo protocols, disposable containers and VMs, sandboxes, access controls, versioning and information flow tracking to minimize overt residues. However, if we wanted to do better than prompting we probably will need an ISA-like layer.

The second, and harder, question is about what happens to hidden/microarchitectural residues. In general, clean up of this type of residue is hard because it is often left in places no one thinks to inspect, or in places users cannot practically inspect because the those parts are proprietary or distributed across organizational boundaries. Also with AI agents, the problem is broader in scope than in a processor because it spans a larger number of tech layers from model context to hardware, local and remote. Further, an agent’s speculation window may last seconds or minutes, compared with nanoseconds in a processor. That longer window creates more opportunity for residue to diffuse. It is highly unlikely that we can simply prompt the agent to clean up hidden/microarchitectural residue because, by definition, there isn’t an architectural interface to observe or control microarchitectural state/work.

How likely are we to solve agentic littering? Who needs this problem solved? And, who should solve this problem?

In addition to technical aspects, economics and incentives often determine whether solutions are adopted. Here too we can look at processor misprediction recovery and compare them to AI agents.

Overt architectural and hidden microarchitectural residues have different economics and incentives at play.

Overt residues are easier to price. If an agent leaves behind a directory full of junk, consumes too many resources, or corrupts a file, that failure is visible to users. Users will complain, and because there are complaints, product teams can justify spending resources to fix them.

Hidden residues are harder. These residues may not produce an obvious effect like a crash. They may also require complex conditions to manifest. That makes it harder to attribute with accuracy and consequently easier to dismiss. It also makes it harder for users to demand fixes, because users often cannot see the thing they are supposed to complain about.

Spectre, an issue due to adversarial steering and microarchitectural residue, was disclosed roughly eight years ago, and the broader class of this leakage has still not been completely fixed. This is not because principled technical solutions do not exist. It is because these solutions increase design complexity, impact performance, change the hardware and software interface in ways that is not easy to adopt, or require coordination across vendors and different layers of the computing stack all of which add recurring or non-recurring costs. Also, each layer can plausibly say that the residue cleanup should be handled by someone else. Vendors can also say that there are have not seen large scale attacks and that they do not have to protect against these attacks given the risk profile.

The same pattern may emerge for AI agents and handling hidden/microarchitectural residues.

Each AI agent boundary is also an economic boundary. Each layer can plausibly say that the residue is someone else’s problem. The model provider can say the deployment should isolate side effects. The deployment/orchestrator can say the runtime should enforce cleanup. The runtime can say the operating system should provide better isolation. The hardware vendor can say software should avoid sensitive colocation. The user experiences the combined risk of all these but usually has the least ability to inspect or repair it!

The real answer is that every party involved in agentic execution should fix its own leaks and share responsibility for security and privacy. But each party also has reason to argue that the cost is too high, especially when the economic benefits are difficult to measure and the harms are difficult to attribute.

So the likely outcome here is not hard to guess. Hidden microarchitectural residue handling is treated as an afterthought, and agents end up reflecting the incentives that shaped it, viz., agents get more capable, overt residue cleanup improves through ad hoc clean up attempts, and create a very long tail of hard to detect, microarchitectural residues that expands the attack surface.

The best chance for security is while these systems are being designed and deployed. AI-agent platforms designed now should at least treat residue management as first-class design requirement. That, however, means finding ways to incentivize designers to care about hidden, microarchitectural residue before users are harmed. If we treat microrchitectural residue management as an optional, “nice-to-have”, “less-important-than-overt” security feature, we will spend the next decade patching a massive, distributed attack surface.

About the Author: Simha Sethumadhavan is a Professor in the CS department at Columbia University. He would like to thank  Profs. Roxana Geambasu. Martha Kim and Tanvir Ahmed Khan for thought provoking comments and feedback.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

Architecture & Systems are Changing: The Architect’s Role in the Era of Agentic Co-Design

The AI datacenter stack is built on hardware-software contracts and abstractions that were never designed for the workloads datacenters now serve. Memory systems strain under terabyte-scale capacity. Heterogeneous accelerators have been pressed into deployment. With datacenters projected to consume over 1,000 TWh annually, surpassing Japan (the world’s fourth-largest economy), renegotiating the hardware-software contract is no longer optional.

AI was enabled by decades of hardware and software efficiency gains. The next leap requires two orders of magnitude more, on a stack whose workloads, infrastructure, and economics bear little resemblance to the one the contract was written for.

That is not a problem any single layer of the stack can solve. It is a co-design problem, and it is unfolding while the design process itself is changing across systems and architecture.

The contract so far

Computer architecture has long been guided by a quiet contract with three commitments: abstractions, interfaces, and transparency. Layers that hide hardware complexity from programmers; interfaces like the x86 ISA that let decades-old binaries still run on Linux today; and microarchitectural state largely hidden behind a model programmers can keep in their heads. Together, these commitments deliver the property programmers care about most: programmability.

This contract is not arbitrary: it is what lets billions of lines of legacy software keep running while architects rebuild underneath. But the contract was negotiated for a world where humans wrote all of the code and humans designed all of the hardware. Both halves of that world are changing at the same time, and the architect’s job is evolving with them.

Plenty of room at the Top

In 2020, Leiserson, Thompson, Emer, Kuszmaul, Lampson, Sanchez, and Schardl argued in Science that post-Moore performance gains would have to come from the “Top” of the computing stack: software, algorithms, and hardware architecture, rather than from the “Bottom” of semiconductor physics. They were right, and the half-decade since has only sharpened the point.

The harder claim in that paper is the one we want to dwell on. The Top has plenty of room, but the gains are “opportunistic, uneven, and sporadic,” subject to diminishing returns. The Top has historically been mined by hand, one paper and one design cycle at a time. What is changing now is the rate at which it is mineable. The two directions we describe next change that rate. Same Top, mined faster, mined more systematically, and mined by tools the field did not have until recently.

Two directions are reshaping the design loop

Two complementary directions are converging on how we build system software and hardware: embedding learning inside low-level mechanisms, and using AI agents to explore the architectural design space itself.

The first direction has a deep history. Perceptron-based branch predictors put a lightweight learning model on the critical path more than two decades ago, and the catalog has steadily grown since. On the cache-hierarchy side, Mockingjay uses a trained reuse-distance predictor to imitate Belady’s optimal replacement policy. On the prefetching side, Hashemi et al. framed memory access patterns as an LSTM prediction task, Pythia recast the entire prefetcher as an online reinforcement-learning agent, and Micro-Armed Bandit showed that lightweight bandit-based RL can match more complex agents at a fraction of the storage cost. Outside the cache hierarchy, reinforcement learning has been applied to chip floorplanning, learning-based memory allocation replaced hand-tuned allocator heuristics with predictors trained on real telemetry, and Seer applied deep learning to predict QoS violations in cloud microservices before they materialize. Most recently, our work on learned virtual memory (LVM) eliminated address-translation overhead with a learned index that fits in two cycles of integer arithmetic. The principle generalizes: fixed designs are being replaced with principled, hardware-realizable models that adapt to workload shifts in ways hand-tuned heuristics cannot.

The second direction is newer, and arguably more disruptive. AlphaEvolve demonstrated that LLMs paired with evolutionary search can discover algorithms across domains, from mathematical constructions to data-center scheduling. ADRS extended the idea to broader systems research, and recent work from Google has applied the same approach to cache replacement. The same paradigm has reached the software side of the machine: agentic systems that generate and tune CUDA and Triton kernels, including NVIDIA’s AVO and Meta’s KernelEvolve, are now in use across heterogeneous accelerators. Sankaralingam captured the bigger picture in Computer Architecture’s AlphaZero Moment, arguing that the field is approaching a regime where architectural discovery itself becomes a search problem, beyond per-mechanism tuning. In our own recent work, Agentic Architect, we coupled LLM-driven code evolution with cycle-accurate simulation to explore microarchitectural design spaces, and found that the loop matches or exceeds state-of-the-art designs on cache replacement, prefetching, and branch prediction. 

These two directions are not alternatives. They differ in what they decide and when. The first decides how a fixed mechanism behaves at runtime: a branch predictor that learns its own weights, a cache policy that adapts to the workload. The second decides what the mechanism looks like in the first place: the predictor, the policy, the prefetcher itself, evolved before deployment. Both move judgment that used to live in tight loops written by experts into search problems that can be scored and re-evaluated.

Figure 1. Agentic Architect, a framework for Computer Architecture Design Space Exploration and Optimization.

Why the loop closes here

Computer architecture has a structural advantage that is easy to take for granted: from the beginning, the field has organized itself around shared, quantitative empirical evaluation. SPEC, PARSEC, CloudSuite, DeathStarBench, and a long list of others encode a community-wide agreement about what a “fair comparison” looks like. The metrics are equally well established: IPC, MPKI, miss rates, area, power, energy-delay product. Every paper in the field is, in effect, a measurement against an agreed instrument.

An agentic loop needs exactly this kind of discipline to close. The loop’s productivity is bounded by the cost and clarity of its fitness signal: how cheaply can a candidate be evaluated, and how reliably does the resulting score reflect the property we actually care about? In domains where evaluation is subjective, expensive, or contested, agentic exploration struggles. In computer architecture, the cycle-accurate simulator gives the loop reproducibility: controlled-environment evaluation against well-defined metrics. Production profiling, hardware performance counters, tracing, and system telemetry give it realism: behavior under load and access patterns that synthetic benchmarks cannot reproduce. The two together are what close the loop.

That has a practical consequence. It means the field does not have to invent its evaluation infrastructure to take advantage of agentic co-design; it has to connect it. The benchmarks, the simulators, and the metric vocabulary are already in place. What is missing is the throughput and the integration: simulators that can serve hundreds of evaluations per study, fitness functions that compose IPC with area and power as primary terms, and training/evaluation splits that let us measure generalization instead of overfitting. We will return to this agenda below.

When code is co-authored, what does “programmable” mean?

Some of the architectural conservatism we aimed to maintain was justified, decades ago, by a single phrase: but no one will program it. The Cell processor’s programmer-managed SPEs and local stores are an example: an elegant design that proved very hard to program in practice.

The cost of programmability used to be borne almost entirely by humans. That is no longer true, and it changes the calculation.

In April 2025, Satya Nadella reported that 20% to 30% of Microsoft’s code was AI-generated, with internal acceptance rates rising monotonically. Google sits in a similar regime: a quarter in Q3 2024, half by fall 2025, and 75% of Google’s code by April 2026, with Sundar Pichai describing the shift as “truly agentic workflows” in which engineers orchestrate fleets of AI agents rather than writing each line themselves.

These numbers describe authorship of characters, not accountability. But they should change how we evaluate the programmability constraint. When agents can routinely program across ISAs, generate platform-specific code paths, write test harnesses, and bridge unfamiliar interfaces given a clear specification, the cost on the programmer is no longer a sufficient veto on a hardware design choice. Designs that were dismissed because they imposed too high a cost on human programmers warrant a fresh look when most of that cost falls on agents instead.

Programmability still matters. Clarity, debuggability, verifiability, and predictable performance remain real properties humans need, and increasingly properties agents need too. Abstractions still matter, perhaps more than ever. Deciding which to expose, which to hide, and which to make machine-checkable is now a question for the programming-languages and systems community alongside architects. But the most consequential lever may not be what we add; it may be what we remove. Many of the layers in today’s stack exist to hide hardware from human programmers and cost cycles and area to maintain. When agents absorb that complexity, the layers come off, and the performance and efficiency we have been paying to abstract away come back.

A widening design space

Reshaping the loop only matters if the space it has to cover is tractable. Increasingly, it isn’t.

A modern AI datacenter spans CPUs, GPUs, AI accelerators, a widening memory landscape (DRAM, CXL, HBM, HBF, SSD), and rack-scale integration with NVLink and optical interconnects. The software and hardware layers have not caught up: a single agentic query may dispatch dozens of model invocations across heterogeneous devices and tool calls on CPUs, all with different abstractions. The operating system (OS), the layer that has historically reconciled such mismatches, must evolve at an unprecedented pace to keep up with growing hardware capabilities and software demands. For example, it only has partial visibility into the GPU, despite its prominent role in AI workloads. Our recent work LithOS has established a beachhead for OS-level control over GPUs, but extending that contract to coordinate the full heterogeneous stack is open. At every level of that stack, energy and power are hardening from secondary considerations into primary constraints.

Each of these pressures is, individually, a multi-year research program. Together, they describe a design space defined by heterogeneous compute, evolving memory hierarchies, rack-scale integration, software-level coordination, and workload regimes that did not exist five years ago. Covering this space by hand is increasingly difficult, even for a large team of architects.

That is the practical case for agentic co-design. The space is outgrowing human-only exploration, and the tools to cover it are finally here.

A proof point

In our recent work, we introduce the Agentic Architect, an agentic framework for architecture design space exploration and optimization. We evaluate it across three of the most studied microarchitectural domains: cache replacement, data prefetching, and branch prediction. We chose them precisely because they are mature. They have decades of literature, well-understood baselines, and limited remaining headroom; if the loop produces gains in these domains, the result is meaningful. The evolved cache replacement policy matched and slightly exceeded Mockingjay; the evolved prefetcher beat SOTA by 17%; the evolved branch predictor improved over Hashed Perceptron on workloads where branch behavior is the bottleneck.

Figure 2. Storage versus performance for data prefetchers. The evolved prefetcher (87 KB) is Pareto-optimal: it delivers the highest geomean speedup over no prefetching at a smaller storage budget than the next-best design.

The more interesting result is what the loop discovered, and what it didn’t. The components in the evolved designs are almost entirely known techniques: stride engines and delta correlators for prefetching, reuse-distance predictors and signature tables for replacement, perceptron variants for branch prediction. None of these primitives is new.

What is new is the coordination. The evolved prefetcher continuously re-evaluates each predictive engine and throttles speculative ones under memory pressure. The evolved replacement policy arbitrates between three independent predictors based on their recent accuracy. The recurring structure across all three domains is the same: preserve the seed’s core, add orthogonal known features, integrate them through new coordination, and adapt at runtime. The novelty lies in the coordination. The loop refines the foundation; the architect still chooses it.

The infrastructure needs to evolve

If agentic co-design is going to do useful work across this design space, the bottleneck moves to infrastructure. The benchmarks and metrics are already there. What we need to build is throughput, multi-objective scoring, and cross-layer reach. The agenda is concrete:

• New tools for agentic architecture design space exploration. Cycle-accurate simulators were built for human-paced experimentation; an agentic loop wants hundreds of evaluations per study, with storage, area, power, and timing as terms in the score rather than afterthoughts that disqualify the result later. We need simulators, search strategies, and metrics purpose-built for this regime: search loops that respect hardware constraints and balance exploration against exploitation, and composite metrics that combine performance, area cost, and generalization into signals the search can rank against.
• Cross-component and cross-layer co-evolution. Co-design across the OS/hardware boundary is now the norm rather than the exception. Taking virtual memory as an example, TLB design, page-table walkers, translation footprint in the caches, and huge-page promotion in the kernel are tightly coupled, and optimizing any one in isolation may capture only a fraction of the available improvement. RTL backends, full-system simulators, and formal verification each let the loop close around a different surface.
• Open source has to evolve. Releasing code is no longer enough. We need structured artifacts that span the full stack, from prompt, seed, and scoring function down to traces, simulator and system configurations, and where applicable RTL, packaged so an agent can clone a repo, re-run the search that produced a published result, and compare new candidates against the same baseline.

The architecture and systems communities are uniquely positioned to drive that work.

Renegotiating computer architecture and systems

A stack co-authored by humans and agents needs renegotiation along the three axes of the old contract. Each is now reweighed against a new deliverable: programmability for agents and humans alike, rather than humans alone.

Abstractions. Many layers exist precisely to hide hardware from human programmers, and they cost cycles and area to maintain. With agents absorbing that complexity, some of those layers can come off; performance and efficiency we have been paying to abstract away come back.

Interfaces. The boundary between hardware and software was drawn for human programmers. As agents become the primary author of low-level code, the interface that carries the contract forward needs redrawing: machine-checkable, composable, and accessible to tools rather than only to humans.

Transparency. The property that lets a programmer model the CPU in their head gives way to a stricter need: explainability. The architect must verify the result against intent, explain why it works, and check that it generalizes beyond the workloads it was trained on. None of these come for free; the field needs methods, metrics, and tooling that make them routine.

Leiserson and colleagues told us in 2020 that there was plenty of room at the Top of the computing stack. The half-decade since has been about confirming they were right; the next half-decade will be about whether we build the tools to actually live there. Agentic co-design, paired with learning embedded inside the system itself, is a strong candidate for addressing the “opportunistic, uneven, sporadic” character that delivered those gains so far.

Architecture is changing. The contract still holds, but the terms are up for negotiation. The next generation of the stack will be defined as much by what we remove as by what we add. The people best positioned to make those calls are the ones who understand both the hardware and the software. That is, by definition, our community.

Acknowledgments

Thanks to the Computer Architecture & Operating System (CAOS) group at Carnegie Mellon and to Prof. Alex Daglis and Prof. Todd Mowry for feedback on this post.

About the Author

Dimitrios Skarlatos is an assistant professor in the Computer Science Department at Carnegie Mellon University. His research bridges computer architecture and operating systems with a focus on AI datacenter efficiency, privacy, and scalability. His work has been deployed in production datacenters and upstreamed into the Linux kernel. He has received the IEEE CS TCCA Young Computer Architect Award, the NSF CAREER Award, the Intel Rising Star Award, a Linux Foundation Faculty Award, an ISCA Best Paper Award, two ASPLOS Best Paper Awards, a CACM Research Highlight, four IEEE MICRO Top Picks, the joint ACM SIGARCH & IEEE CS TCCA Outstanding Dissertation Award, the David J. Kuck Outstanding PhD Thesis Award, and over a dozen industry faculty awards from Amazon, AMD, Intel, Meta, Oracle, and VMware. His recent work led to the founding of LithosAI.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.

TL;DR: The history of parallel computing is a history of shifting what we put at the center of the computer. The first axis, control-level parallelism (CLP), is control-centric and schedules around the program counter: it gave us the high-performance computing (HPC) era. The second axis, data-level parallelism (DLP), is data-centric and schedules around tensors: it gave us the artificial intelligence (AI) era. A third axis is now emerging: value-level parallelism (VLP), where narrow data bitwidth exposes a small number of unique values and lets the architecture deduplicate redundant computation. Two recent works, Carat (ASPLOS ’24) and Mugi (ASPLOS ’26), make the case concretely: VLP eliminates redundant computation in both linear and nonlinear operations on AI workloads. This article argues that VLP is not a point of optimization but the beginning of a value-centric computing paradigm, one that is crucial for addressing the escalating energy demands of next-generation intelligent systems.

Traditional Parallel Computing

The First Axis: Control-Level Parallelism

The HPC era saw a diverse set of workloads. The metric of success made the goal explicit: instructions per cycle (IPC) normalizes performance to how fast instructions are consumed, not to what data the instructions are operating on. Therefore, computer architecture in the HPC era was control-centric. Michael Flynn formalized the design space in 1966 with his taxonomy [1]: SISD, SIMD, MISD, MIMD, outlining the orthogonality of instruction and data. For decades, this was the right framing: transistors were scarce, control logic was expensive, and the most valuable thing an architect could do was to issue one more instruction per cycle. 

The actual implementation to exploit the CLP to execute multiple instructions in parallel arose from the independence between instructions. We enjoyed the technology evolution from pipelining, branch prediction for SISD, superscalar issue, out-of-order execution, simultaneous multithreading, chip multiprocessors for MIMD and beyond.

The Second Axis: Data-Level Parallelism

Entering the AI era, powered by large language models (LLMs), transistors became plentiful but the memory bandwidth became scarce due to the large data volume in AI tensors. Consequently, we hit the memory wall and turned to data-centric architectures, expanding more along the data dimension in Flynn’s taxonomy. Success is now measured by how well we apply one operation to many data elements while feeding them efficiently from memory (e.g., throughput, goodput, and arithmetic intensity). TPUs with systolic arrays and GPUs with tensor cores are renowned examples to exploit the rich DLP opportunities from high-dimensional tensors.

Diving deeper, we see that compute arrays, i.e., dataflow architecture, are becoming the first class citizens. Dataflow architecture follows the philosophy of letting data drive control, with early works from MIT [2] [3]. With regular compute and memory patterns in AI tensors, dataflow architecture builds massively parallel compute arrays to maximize the computational density and minimize the control overhead. 

Another line of research to exploit DLP for AI workloads targets the von Neumann bottleneck, envisioned by John Backus as early as 1978 [4]. These solutions move the computation closer to the memory (e.g., in/near-memory/storage processing) by attaching additional compute logic next to the memory blocks, unlocking massive DLP on the already wide enough memory blocks.

The Third Axis: Value-Level Parallelism

Though Flynn’s taxonomy, with dimensions of instructions and data, has been followed for decades, there are untouched landscapes. While CLP and DLP focus on concurrency and parallelism, neither asks the next question about the content of the data: can the patterns in data values benefit the computation efficiency? VLP is value-centric and the third axis in this regard, i.e., it targets computational redundancy inherent to the data patterns of workloads. It recognizes that when identical values flow through a pipeline, the arithmetic becomes deterministic and, therefore, avoidable. Consequently, we move from executing every instruction and data to computing only each unique data value.

Origins for GEMM

Carat (Pan, San Miguel, Wu — ASPLOS ’24) [5] is the paper that coined VLP and materialized it in hardware architecture. The insight is simple. As deep learning inference moves to larger batches and lower precisions (e.g., FP8 being the de-facto data format in DeepSeek v3), the number of unique values shrinks rapidly while the frequency of each grows. Here, we give an example. For a scalar-vector multiplication for an arbitrary scale weight w and 1k UINT4 inputs, conventional hardware would compute 1k multiplications for the weight w and each UINT4 vector element. Looking closely, there are only 16 unique products, i.e., 0 x w, 1 x w, 2 x w, …, 15 x w. Thus, conventional hardware would compute 1k/16=64 times more than needed.

Figure 1. Overview of VLP for scalar-vector multiplication.

Figure 1 above outlines how VLP is constructed in Carat. In Figure 1 (a), VLP consists of value reuse, which accumulates the weight w over time, each accumulation result, called a partial product, is used to compute the next partial product. Then each input just subscribes to the proper partial product as the correct output. To materialize the subscription, we leverage temporal coding, often seen in the brain, which generates a spike at the cycle indexed by the data value. For example, a data valued 8 will generate a spike at cycle 8, as shown in Figure 1 (b). Therefore, there exists a temporal correspondence between the spike and the accumulated partial product. Each input subscribes to their correct output in parallel, giving the rise to value-level parallelism. The scheduling unit is no longer the instruction or the array element; it is the unique product value, made available to many input consumers via temporal coding. Given this formulation, we see that lower precision produces fewer unique values, while larger batches create more inputs to share the unique values.

Generalizing Beyond

So far, VLP in Carat targets GEMM optimization for large-batch, low-precision, symmetric-format use cases. However, these assumptions may no longer hold in more recent LLM workloads: the batch size is small (e.g., 8~16) to ensure real-time response, the data formats are asymmetric (e.g., INT4-FP16) to minimize the memory footprint of the weight and KV cache, and the nonlinear operations are heavy and complicated (e.g., softmax, GELU, SiLU) to ensure high accuracy.

Mugi (Price, Vellaisamy, Shen, Wu — ASPLOS ’26) [6] is a follow-up work that closes the gap and generalizes VLP for both linear and nonlinear operations for broader AI workloads. Figure 2 shows VLP for elementwise nonlinear operations that can be done through a four-phase pipeline on input floating-point numbers with a sign (S), mantissa (M) and exponent (E). The first phase is input approximation, which converts wider inputs to narrower bits without sacrificing the LLM accuracy too much. The inputs to nonlinear operations are always in higher precision (e.g., BF16, FP16, or FP32) in AI workloads than model weights. The approximation to narrower bits ensures a shorter temporal signal for high throughput. The second phase is value reuse with opportunities from large GEMM shapes in LLMs. Unlike value reuse in Carat accumulating the partial product, value reuse in Mugi loads the precomputed nonlinear results, where higher accuracy is allocated to more critical inputs. The third phase performs temporal subscription on the mantissa bits (M) of the inputs. For each input, the selected output corresponds to the nonlinear results with the same mantissa but different exponents (E). Finally, the fourth phase performs temporal subscription on the exponent bits of the inputs. For each input, the selected output corresponds to the correct mantissa and exponent.

Figure 2. Overview of VLP for elementwise nonlinear operations.

Mugi essentially cascades VLP to construct high dimensionality in the value space to compute nonlinear operations. The resulting architecture unifies the datapath for both linear and nonlinear operations, leading to savings in silicon area.

What Makes VLP Different

Conventional architectures pay arithmetic costs to process every instruction and data element while advanced techniques (e.g., memoization, tabulation) leverage computation reuse and pay memory costs to store past results and refer back to them when needed. VLP pays for neither: it minimizes both arithmetic and memory access, instead spending its silicon budget on value delivery, i.e., the network and temporal converters that route unique results to many input consumers in parallel. The type of computation fundamentally changes to something new: a form of temporal subscription.

Future Opportunities and Open Questions

In its current form, VLP relies on temporal coding, which is actually inspired from how the brain works. Though the community has focused predominantly on deep learning, VLP opens up a different research direction: what are potential synergies between neuromorphic and classical computing in computer architecture? Looking beyond, VLP raises several questions to answer in the AI era. 

• Where does VLP stop paying off? Though Carat and Mugi work for varying batch sizes, both of them now are designed for low precision to create more opportunities for value reuse. There is presumably a design space with high precision and low value redundancy. It is essential to understand the mechanism to exploit VLP for such scenarios and quantify the potential gain.
• Is VLP an ISA-level concept or an accelerator-level one? Both Carat and Mugi are accelerator designs. A real-world question is whether VLP can inform CPU and GPU microarchitecture. What would a VLP-based tensor instruction look like? Could it be a drop-in replacement of tensor cores with better efficiency?
• What should the software stack look like? VLP for nonlinear involves approximation, which naturally introduces inaccuracy to the task. This falls back to the question of approximate computing, but in the context of new hardware primitives. We probably shall build a co-design framework to deploy VLP under approximation errors.
• Can VLP live with sparsity? Sparse computation has been a major optimization since the start of deep learning, and more opportunities are emerging from weight, KV-cache spanning across the bit level, value level, block level and even request level. It is meaningful to study how VLP synergizes with such use cases.
• How does VLP interact with memory? Despite efforts in computation, memory stays at the core of AI. A natural question is whether we can optimize the memory system with VLP, or more broadly, value-centric computing. There have been associative memory-based AI accelerators, and whether there could be VLP-based alternatives?

Final Thoughts

Architecture research is all about how to compute faster and more efficiently. Control-level parallelism has let us argue about IPC and pipelines for thirty years. Data-level parallelism has let us argue about FLOPS and dataflow for fifteen. Value-level parallelism, the third axis, now shows promise for emerging AI workloads (thanks to Carat and Mugi) and paves the way for exciting synergies between neuromorphic and classical computing. Here’s hoping one day computer architects see the value in it.

About the authors:

Di Wu is an assistant professor at the University of Central Florida. 

Zhewen Pan is a PhD candidate at the University of Wisconsin–Madison. 

Joshua San Miguel is an associate professor at the University of Wisconsin–Madison.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.