SWE-Together: Interactive Coding-Agent Evaluation

The paper introduces "SWE-Together," a comprehensive benchmark designed to evaluate interactive coding agents within the Software Engineering (SWE) domain. Addressing the limitations of static benchmarks, SWE-Together focuses on multi-turn, interactive programming tasks that reflect real-world collaborative coding scenarios.

Core Methodology and Evaluation Framework

The benchmark comprises 109 diverse software engineering tasks sourced from real-world repositories, utilizing the "opencode" harness for environment orchestration. The evaluation methodology emphasizes a two-tier success metric:

1. Pass@1: Measures the probability that a single model run successfully resolves a task, representing the baseline performance of the agent.
2. Pass² ($Pass^2$): A more stringent metric defined as the probability that two independent model runs both achieve a judge score of $\ge 0.85$. This serves as an indicator of result stability and reproducibility in an interactive environment.

The performance gap between Pass@1 and Pass² is termed the "unstable tail" (calculated as $Pass@1 - Pass^2$), which quantifies the variance and reliability of the agents' code generation capabilities.

Technical Metrics and Performance Analysis

Evaluation is conducted using a "Model judge" system that scores outputs on a scale from 0 to 1, where higher scores reflect higher adherence to functional requirements. Key performance indicators analyzed across various models include:

• Correction Rate: Measured as a ratio (lower is better), identifying the efficiency of the agent in reaching a correct solution without needing repeated human or heuristic interventions.
• Token Efficiency: Tracked as the average number of tokens (in thousands) consumed per task. Current state-of-the-art models like claude-opus-4.8 demonstrate an optimal balance, achieving a 52% Pass@1 rate with 74.0k tokens, while gpt-5.5 displays higher efficiency, requiring only 29.9k tokens for a 48% Pass@1.
• Temporal Performance: Measured in minutes per task, providing insight into the agent's latency in complex problem-solving.

Key Findings

The study establishes an "Oracle reference" ceiling of ~78% success, derived from gold-patch human solutions, which serves as the upper bound for current automated agents. Comparative data reveals that while claude-opus-4.8 currently leads in Pass@1 (52%) and Judge score (0.801), there is a significant performance variance across models like glm-5.2 and deepseek-v4-pro. The data suggests that as model complexity increases, the primary engineering challenge is narrowing the unstable tail to ensure consistent, high-quality code resolution in autonomous software development environments.