Summary: The Evolution of KV Caching in LLM Inference

The provided paper traces the architectural progression of Key-Value (KV) caching strategies, focusing on how recent LLM developments—such as long-context agents and heterogeneous model architectures—have forced a move from simple in-memory caching to complex, hierarchical, and specialized cache management systems.

#### 1. Foundational Concept: Prefix Caching and RadixAttention
Traditional KV caching stores Key and Value vectors to avoid recomputing prefill tokens. RadixAttention (SGLang) revolutionized this by utilizing a Radix Tree to manage KV blocks. This structure enables:

• Prefix Sharing: By mapping token sequences to paths in the tree, the system can identify common prefixes across multiple requests, skipping redundant prefill calculations.
• Granular Matching: Token-level precision in matching allows for efficient reuse of cached states in multi-turn dialogues and agentic workflows.

#### 2. Scaling Memory: HiCache and Hierarchical Storage
As GPU HBM (High Bandwidth Memory) becomes a bottleneck, HiCache introduces a three-tier storage architecture (L1: GPU, L2: CPU, L3: Remote/Disk).

• Data Plane Efficiency: To overcome the overhead of scattered memory access in PagedAttention, HiCache employs GPU kernels to perform direct memory access (DMA) transfers, effectively parallelizing the movement of scattered pages.
• Layout Reordering: It shifts from the standard *layer-first* layout to a *page-first* layout, allowing for atomic I/O operations (reading/writing an entire page across all layers at once), which significantly optimizes latency for offloading.
• Control Plane Strategies: The system manages offloading via intelligent prefetching. During idle periods or queueing, it proactively moves data from L3 to L2, and from L2 to L1, masking latency through pipelining where the transfer of layer $L_{n+1}$ overlaps with the computation of layer $L_n$.

#### 3. Adaptation to Hybrid Architectures (SWA)
Modern models (e.g., Xiaomi MiMo-V2.5) often use hybrid architectures, combining Full Attention with Sliding Window Attention (SWA). This breaks the assumption that cached KV states remain valid indefinitely.

• Window-Safe Length: Systems now enforce a policy where hits are only valid if the SWA tail remains intact within the cached window.
• Decoupled Management: By separating Full Attention and SWA pools while maintaining a unified logical view, these systems optimize memory utilization, achieving high cache hit rates (93%+) even with restricted storage.

#### 4. The Shift to Flat Paged Pools: TokenSpeed and Heterogeneity
The emergence of models like *Inkling* (featuring mixed attention mechanisms and convolutional states) has led to the deprecation of the Radix Tree in some implementations in favor of Flat Paged Pools with Heterogeneous Views.

• The Problem: Rigid Radix Tree structures struggle with "state heterogeneity," where different layers require different caching rules (e.g., infinite growth vs. bounded window vs. fixed state).
• The Methodology:
  • Unified Physical Allocation: Memory is managed as a flat block pool, simplifying allocation and reducing internal fragmentation.
  • Heterogeneous Interpretation: The same physical block is viewed differently by distinct manager groups. For instance, a block might store $256$ tokens for Full Attention but only $16$ tokens for convolutional states.
  • Logic-Physical Separation: By delegating the caching policy to group managers (e.g., best-effort eviction for SWA, permanent storage for Full Attention) while keeping the memory allocator centralized, the system gains architectural scalability without compromising performance.

#### Conclusion
The evolution of KV caching reflects a transition from "simple reuse" to "context-aware state management." Whether through the meticulous repair of Radix Trees (MiMo) or the fundamental shift to flat, multi-viewed memory pools (TokenSpeed), the industry is converging on strategies that separate physical memory allocation from logical state requirements to support increasingly complex, multi-modal, and long-context LLM architectures.