Nested Learning is a novel machine learning paradigm introduced to address the challenge of catastrophic forgetting in continual learning, where models lose proficiency on previously learned tasks when acquiring new knowledge. It posits that a machine learning model should not be viewed as a single, continuous process but rather as a system of interconnected, multi-level optimization problems. This approach unifies the traditionally separate concepts of model architecture (network structure) and optimization algorithms (training rules), asserting they are fundamentally different "levels" of optimization. Each level possesses its own internal flow of information, termed "context flow," and a distinct update rate, or frequency, at which its parameters are adjusted.

The core methodology of Nested Learning is built upon the insight that these interconnected optimization problems can be ordered into "levels" based on their update frequencies. This perspective reveals a new dimension for designing more capable AI by enabling the construction of learning components with deeper computational depth.

Key technical aspects and applications of Nested Learning include:

1. Associative Memory as a Foundation: The paradigm formalizes both the training process (specifically, backpropagation) and key architectural components (like the attention mechanism in Transformers) as associative memory modules. For backpropagation, the model learns to map a given data point to its local error, which quantifies the "surprise" or unexpectedness of the data. For attention mechanisms, they are seen as modules learning mappings between tokens in a sequence. This formalization suggests that well-known architectures, such as Transformers and memory modules, can be interpreted as linear layers with differing frequency updates.

2. Deep Optimizers: Nested Learning views standard optimizers (e.g., momentum-based optimizers) as associative memory modules. It highlights that many existing optimizers rely on simple dot-product similarity, given by $\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i$ for vectors $\mathbf{a}$ and $\mathbf{b}$, for their updates, which does not adequately account for relationships between different data samples. By reframing the optimizer's underlying objective to a more standard loss metric, such as L2 regression loss (e.g., $(\hat{y} - y)^2$ for a single prediction $\hat{y}$ and true value $y$, or the mean squared error $\frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2$ over $N$ samples), new formulations for concepts like momentum are derived, enhancing their resilience to imperfect data.

3. Continuum Memory Systems (CMS): This concept extends the typical short-term (sequence model) and long-term (feedforward neural networks) memory distinction in Transformers. Within the Nested Learning framework, memory is conceptualized as a "spectrum of modules," with each module updating at a specific, distinct frequency rate. This multi-time-scale update mechanism, analogous to multi-time-scale updates in the human brain, creates a richer and more effective memory system for continual learning.

As a proof-of-concept, the authors designed "Hope," a self-modifying recurrent architecture based on Nested Learning principles. Hope is a variant of the "Titans" architecture, which are long-term memory modules that prioritize memories based on their surprisingness. Unlike Titans, which have two levels of parameter updates resulting in first-order in-context learning, Hope leverages unbounded levels of in-context learning. It is augmented with CMS blocks to scale to larger context windows and features a self-referential process that allows it to optimize its own memory, theoretically leading to "infinite, looped learning levels."

Experimental results demonstrated Hope's superior performance in language modeling (lower perplexity) and common-sense reasoning (higher accuracy) compared to modern recurrent models and standard Transformers. Furthermore, Hope exhibited better memory management in long-context Needle-In-Haystack (NIAH) tasks, confirming the efficacy of CMSs in handling extended sequences of information efficiently. The paradigm suggests a robust foundation for bridging the gap between the limited, forgetting nature of current large language models (LLMs) and the remarkable continual learning capabilities observed in the human brain.