Deep Delta Learning (DDL) introduces a novel architecture that generalizes the standard identity shortcut connection in deep residual networks, addressing its limitation of imposing a strictly additive inductive bias that restricts the modeling of complex state transitions. While traditional ResNets approximate an ODE $\dot{\Xb} = \Fb(\Xb)$ via an additive update $\Xb_{l+1} = \Xb_l + \Fb(\Xb_l)$, DDL proposes a more flexible layer-wise transformation.

The core of DDL is the Deep Delta Residual Block, which updates the hidden state matrix $\Xb \in \RR^{d \times d_v}$ using the rule:
$$\Xb_{l+1} = \underbrace{(\Ib - \beta_l \kb_l \kb_l^\top)}_{\text{Delta Operator } \Ab(\Xb)} \Xb_l + \beta_l \kb_l \vb_l^\top$$
Here, $\Xb_l$ is the hidden state at layer $l$, $\Ib$ is the identity matrix, and the parameters $\beta_l \in \RR$, $\kb_l \in \RR^d$, and $\vb_l \in \RR^{d_v}$ are learnable and data-dependent. Specifically, $\beta_l$ is a scalar gate, $\kb_l$ is a reflection direction vector, and $\vb_l$ is a value vector. The first term, $(\Ib - \beta_l \kb_l \kb_l^\top) \Xb_l$, represents a rank-1 perturbation of the identity matrix applied to the current state, termed the Delta Operator $\Ab(\Xb)$. This operator modulates the identity shortcut with a learnable, data-dependent geometric transformation. The second term, $\beta_l \kb_l \vb_l^\top$, is an injection term. This formulation innovatively couples the "erasure" of old information (via projection onto $\kb_l$) with the "writing" of new features (via injection of $\vb_l$), both scaled synchronously by the gate $\beta_l$.

The expressive power of DDL is rooted in the spectral properties of the Delta Operator $\Ab(\Xb)$. Its eigenvalues are $\{1, \dots, 1, 1-\beta\}$, meaning the scalar gate $\beta$ deterministically controls the operator's spectrum and thus its geometric interpretation. This allows the network to dynamically interpolate between three fundamental linear transformations:

• Identity Mapping ($\beta \to 0$): When $\beta$ approaches $0$, the eigenvalues are $\{1\}$. This corresponds to a skip connection, preserving the signal for deep propagation, such that $\Xb_{l+1} \approx \Xb_l$.
• Orthogonal Projection ($\beta \to 1$): When $\beta$ approaches $1$, the eigenvalues become $\{0, 1\}$. This operation represents an orthogonal projection onto the hyperplane $\kb^\perp$, effectively "forgetting" or erasing components of the state parallel to $\kb$. In this regime, $\det(\Ab) \to 0$.
• Householder Reflection ($\beta \to 2$): When $\beta$ approaches $2$, the eigenvalues become $\{-1, 1\}$. This corresponds to a Householder reflection, which inverts the state along the direction $\kb$. This introduces negative eigenvalues, enabling the modeling of oscillatory or oppositional dynamics within the network's state transitions, with $\det(\Ab) \to -1$.

Furthermore, DDL establishes a theoretical link to efficient sequence models through its "Depth-Wise Delta Rule." Expanding the DDL update rule reveals a classic Delta Rule structure:
$$\Xb_{l+1} = \Xb_l + \beta_l \kb_l (\vb_l^\top - \kb_l^\top \Xb_l)$$
This formulation highlights how the network explicitly controls the spectrum of its layer-wise transition operator. It acts like a "target-minus-current-projection" update, where $\vb_l^\top$ is the target value and $\kb_l^\top \Xb_l$ is the current projection of the state onto the direction $\kb_l$. Unlike architectures like DeltaNet which apply this "Delta Rule" over the time dimension, Deep Delta Learning applies it over the depth dimension, allowing the network to selectively "clean" or "rewrite" specific feature subspaces layer-by-layer. This mechanism prevents the accumulation of interference, a common issue in standard additive ResNets, while preserving the stable training characteristics of gated residual architectures.