The DeepSeek-V4 series introduces a high-efficiency Mixture-of-Experts (MoE) architecture designed for long-context intelligence, supporting up to one million tokens. The series includes the DeepSeek-V4-Pro (1.6T total parameters, 49B activated) and DeepSeek-V4-Flash (284B total parameters, 13B activated). The architecture aims to mitigate the quadratic complexity of standard attention mechanisms through hardware-aware optimizations and novel structural innovations.

Core Methodological Innovations

1. Hybrid Attention Architecture (CSA and HCA): To handle 1M-token sequences, the model employs a combination of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).
   • CSA: Compresses the KV cache by a factor of $m$ by combining adjacent entries and then uses a "Lightning Indexer" to perform sparse attention. The indexer computes scores using low-rank projections of query hidden states $h_t$ to select the top-$k$ most relevant compressed KV blocks.
   • HCA: A more aggressive compression strategy that reduces the KV cache by $m'$ (where $m' \gg m$) while maintaining dense attention for the remaining entries. Both mechanisms are supplemented by a sliding window of local KV entries to preserve fine-grained local dependencies.

2. Manifold-Constrained Hyper-Connections (mHC): DeepSeek-V4 replaces standard residual connections with mHC to improve stability during deep stacking. The residual transformation $X_{l+1} = B_l X_l + C_l F_l(A_l X_l)$ is constrained by ensuring the residual mapping matrix $B_l$ resides within the manifold of doubly stochastic matrices ($M \in \mathbb{R}^{n \times n} | M1_n = 1_n, 1^T_n M = 1_n, M \ge 0$). This ensures the spectral norm is bounded by 1, preventing signal explosion in deep networks. The matrix $B_l$ is computed via the Sinkhorn-Knopp algorithm: $M^{(t)} = T_r(T_c(M^{(t-1)}))$.

3. Optimizer and Training: The model utilizes the Muon optimizer, which enhances training stability and convergence speed. The architecture adopts the DeepSeekMoE framework with fine-grained routed experts. A significant shift in the routing mechanism involves replacing the sigmoid-based affinity computation with $\text{Sqrt}(\text{Softplus}(\cdot))$ to improve expert selection stability.

4. Infrastructure and Efficiency:
   • Inference Efficiency: In a 1M-token context, DeepSeek-V4-Pro requires only 27% of the FLOPs and 10% of the KV cache relative to DeepSeek-V3.2.
   • Quantization-Aware Training: The models incorporate FP4 quantization for routed expert weights and the indexer query-key (QK) path, significantly reducing memory footprint without degrading core reasoning capabilities.
   • Kernel Optimization: The implementation leverages TileLang for custom, batch-invariant, and deterministic kernels, ensuring high performance across various hardware settings and bitwise reproducibility.

Post-Training Pipeline

The models undergo a two-stage post-training process:

• Specialist Training: Independent training of domain-specific experts (mathematics, coding, agentic workflows) using Supervised Fine-Tuning (SFT) and Reinforcement Learning via Group Relative Policy Optimization (GRPO).
• On-Policy Distillation: A unified student model is trained to consolidate the diverse proficiencies of the specialist experts by minimizing the reverse KL divergence between the unified model and the specialized teacher models.

Performance Summary

DeepSeek-V4-Pro-Max demonstrates state-of-the-art performance for open models, rivaling frontier closed-source proprietary models in reasoning and coding benchmarks. The series effectively resolves the efficiency bottleneck of long-context LLMs, providing a foundation for advanced test-time scaling and long-horizon agentic task completion.