The paper details the development, technical architecture, and practical applications of Solar Open 2, a 250B parameter open-weight large language model (LLM) developed by Upstage with government support. The primary focus of the model is achieving frontier-level reasoning and agentic performance while maintaining high operational efficiency for enterprise office environments.

Core Methodology and Technical Architecture

The model's development followed a rigorous multi-stage training pipeline, involving pre-training, instruction tuning, and massive-scale reinforcement learning.

#### 1. Hybrid Attention and Structural Innovation
To support long-context capabilities (up to $1M$ tokens) without the exponential memory and computational costs associated with standard attention, Upstage implemented a Hybrid Attention mechanism. This design uses a $3:1$ ratio of softmax attention to linear attention, which reduces memory consumption and computational complexity by approximately $1/4$ compared to pure softmax implementations.

#### 2. Architectural Adjustments

• Removal of Rotary Positional Embeddings (RoPE): The team removed traditional position encodings, forcing the model to learn token order intrinsically through the transformer structure.
• Stable State Training: To stabilize the training of position-invariant structures, a sigmoid gate was added.
• Negative Eigenvalues: For the linear attention component, the team introduced negative eigenvalues, allowing the model to perform "forgetting" operations on past states, which is critical for complex agentic tasks.
• Weight Transfer: To initialize the 250B model, Upstage developed a mapping technique to transfer weights from their earlier 100B model despite the architectural changes, successfully porting approximately $5.7B$ parameters ($2.3\%$ of the total).

#### 3. Post-Training and Reinforcement Learning (RL)
Upstage shifted from traditional SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) to a sophisticated RL-based approach. The team created 12 domain-specific "teacher" models (specializing in coding, office automation, and document processing) to generate verified, high-quality training trajectories.

• Agentic Environment: A significant portion of R&D effort was dedicated to building a sandboxed environment where models could perform "verifiable" tasks (e.g., executing code, creating Excel/PPTX files).
• Asynchronous Learning: To resolve GPU idling during RL (caused by varying task completion times), the team implemented an asynchronous training pipeline, ensuring high GPU utilization and efficient batching of diverse task signals.

Optimization and Efficiency

In collaboration with Nota AI, the model was optimized for inference through advanced quantization and pruning techniques:

• Calibrating MOE: For Mixture-of-Experts (MOE) structures, they employed a technique called "Pascal-MOE" to ensure balanced data distribution across experts during calibration.
• Router Preservation: A custom loss function was introduced during quantization to prevent "router collapse," ensuring that the decision-making logic of the MOE remains consistent with the original FP16/BF16 weights.
• Global Pruning: Unlike layer-wise pruning, the team utilized global pruning to remove $\sim 20\%$ of weights, allowing the model to fit into lower-memory hardware configurations (e.g., two H100 GPUs) while maintaining context windows of $256K+$ tokens.

Practical Application

Solar Open 2 is specifically tailored for "Office-verse" tasks, demonstrating high proficiency in:

• Automated Data Processing: Generating and analyzing complex documents, CSVs, and Excel reports.
• Legal/Enterprise Workflows: Achieving performance comparable to proprietary frontier models in legal research, case analysis, and document review, while enabling deployment in on-premise, secure environments.
• Agentic Tool Use: The model exhibits high stability in multi-step reasoning, making it suitable for desktop agent tasks where multiple tools must be invoked sequentially.