The paper introduces Memory Caching (MC), a technique designed to bridge the gap between memory-bounded recurrent neural networks (RNNs) and memory-intensive Transformers. While Transformers offer a growing memory capacity that scales quadratically with sequence length ($O(L^2)$), RNNs compress information into a fixed-size memory, leading to $O(L)$ complexity but often underperforming in recall-intensive, long-context tasks due to forgetting. MC allows the effective memory capacity of RNNs to grow with sequence length, providing a flexible trade-off with $O(NL)$ complexity, where $N$ is the number of cached segments.

The core idea of MC is to segment the input sequence $x \in \mathbb{R}^{L \times d_{in}}$ into $N$ segments, $S^{(1)}, \dots, S^{(N)}$, each with length $L^{(s)}$. For each segment $S^{(s)}$, a memory module $M^{(s)}$ compresses the segment's information. The memory update within a segment $s$ at time $t$ is defined as $M_t^{(s)} = f(M_{t-1}^{(s)}; k_t, v_t)$, where $k_t = x_t W_k$, $v_t = x_t W_v$, and $q_t = x_t W_q$ are key, value, and query projections of the input token $x_t$. The last state of each segment's memory, $M_{L^{(s)}}^{(s)}$, is cached.

The key innovation lies in how the model computes its output $y_t$. Instead of relying solely on the current "online memory" $M_t^{(s)}$, MC augments this with access to all previously cached segment memories. The output is computed as:
$y_t = \text{Agg}\left(\{M_{L^{(i)}}^{(i)}(\cdot)\}_{i=1}^{s-1}; M_t^{(s)}(\cdot); q_t\right)$
where $\text{Agg}(\cdot)$ is an aggregation function combining the online memory with the cached memories for the query $q_t$. The paper proposes four variants for this aggregation:

1. Residual Memory: This is the simplest aggregation, summing the online memory's output with the outputs of all cached memories when queried by $q_t$.

$y_t = M_t^{(s)}(q_t) + \sum_{i=1}^{s-1} M_{L^{(i)}}^{(i)}(q_t)$
For strictly linear memory modules (e.g., linear attention), this formulation can mathematically collapse into a standard fixed-size memory if segments are trained sequentially from the previous state, as cached memories can be pre-summed. However, even this simple form shows empirical benefits, acting as a retention mechanism.

2. Gated Residual Memory (GRM): To introduce query-dependent selectivity, GRM incorporates input-dependent gating parameters $\gamma_t^{(i)} \in [0,1]$ for each segment.

$y_t = \gamma_t^{(s)} M_t^{(s)}(q_t) + \sum_{i=1}^{s-1} \gamma_t^{(i)} M_{L^{(i)}}^{(i)}(q_t)$
The gates $\gamma_t^{(i)}$ are typically parametrized as similarities between a linear projection of the current input $u_t = x_t W_u$ and a mean-pooled representation of segment $S^{(i)}$, $\text{MeanPooling}(S^{(i)})$. Specifically, $\gamma_t^{(i)} = \left\langle u_t, \text{MeanPooling}(S^{(i)}) \right\rangle$, often followed by a softmax normalization. This prevents the collapse observed in the linear residual case, as the gates are token- and segment-dependent.

3. Memory Soup: Inspired by model souping, this variant combines the *parameters* of the cached memory modules into a single, data-dependent memory $M_t^\ast$ before retrieval. If $M_{L^{(i)}}^{(i)}$ is parameterized by $\theta_{M_{L^{(i)}}^{(i)}} = \{W_1^{(i)}, \dots, W_c^{(i)}\}$, then the aggregated memory's parameters are $\theta_{M_t^\ast} = \left\{\frac{\sum_{i=1}^{s} \gamma_t^{(i)} W_1^{(i)}}{\sum_{i=1}^{s} \gamma_t^{(i)}}, \dots, \frac{\sum_{i=1}^{s} \gamma_t^{(i)} W_c^{(i)}}{\sum_{i=1}^{s} \gamma_t^{(i)}}\right\}$. The output is then $y_t = M_t^\ast(q_t)$.

This approach is distinct from GRM when using deep or non-linear memory modules, as it creates a specialized non-linear retrieval function by interpolating parameters, rather than interpolating outputs.

4. Sparse Selective Caching (SSC): To mitigate memory overhead for very long sequences, SSC selectively chooses a subset of cached memories. A router computes relevance scores $r_t^{(i)} = \left\langle u_t, \text{MeanPooling}(S^{(i)}) \right\rangle$ for each segment $S^{(i)}$. It then selects the top-$k$ most relevant segments to form a set $R_t$, which are used in the aggregation.

$y_t = \gamma_t^{(s)} M_t^{(s)}(q_t) + \sum_{i \in R_t} \gamma_t^{(i)} M_{L^{(i)}}^{(i)}(q_t)$
This reduces the computational cost of retrieval by operating only on a sparse subset of cached memories.

The paper discusses two main approaches for managing memory states across segments:

• Checkpointing a single memory: In this view, the model trains an associative memory, and the cached states $M_{L^{(s)}}^{(s)}$ are checkpoints of this optimization process. This implies $M_0^{(s)} = M_{L^{(s-1)}}^{(s-1)}$, meaning the memory for a new segment starts from the last state of the previous segment.
• Independent compressors: Each segment is processed by an independent memory module that starts from an initial point $M_0^{(s)}$ unrelated to $M_{L^{(s-1)}}^{(s-1)}$. This views each cached memory as a compressed representative of its specific segment, avoiding interference between segments.

As a proof of concept, MC is empirically validated on architectures like Linear Attention (LA), deep memory module Titans, Sliding Window Linear Attention (SWLA), and Deep Linear Attention (DLA) across language modeling, long-context, and retrieval tasks. The results indicate that MC enhances recurrent model performance, extends their effective context length, and performs competitively against Transformers in in-context recall tasks, narrowing the performance gap with state-of-the-art recurrent models.