*HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?* introduces a novel benchmark designed to evaluate Large Language Models (LLMs) not by their ability to solve isolated tasks, but by their capacity to engineer the execution infrastructure—the "harness"—that enables agentic behavior.

Core Premise and Motivation

Current agent benchmarks typically treat the harness (the execution loop, tool-use logic, context management, and verification mechanisms) as a static experimental variable. The authors argue that as agents move toward real-world deployment, the ability to build and refine these systems is a critical skill. HarnessDev shifts the evaluation unit from a single task output to a persistent, runnable, and revisable execution system. The performance of an agent is defined by two metrics:

1. Capability: The success rate of the harness on held-out downstream tasks.
2. Efficiency: The consumption of executor-model tokens required to achieve that performance.

Methodology

The benchmark operates through two distinct stages of development:

1. Creation (RQ1): A creator LLM starts with a "weak seed" ($H_{seed}$), which provides only a basic I/O interface and non-orchestrated primitives (e.g., file I/O, search, tool-calling portals) without any built-in planning, loop, or verification logic. The model must construct a complete, functional agent harness.
2. Evolution (RQ2): The model takes its own created harness ($H_0$) and iteratively revises it based on downstream execution feedback from a designated set of tasks.

The formal evaluation follows the relationship:
$$(L_C, D) \rightarrow H, \quad (H, L_E, x) \rightarrow y \xrightarrow{J} \text{score}$$
Where $L_C$ is the creator, $D$ is the development environment, $H$ is the generated harness, $L_E$ is the executor, $x$ is the task, and $J$ is the judge.

To prevent cheating or overfitting, the benchmark enforces rigorous constraints. Harnesses are frozen before evaluation. The scorer relies on verifiable, objective outcomes (e.g., repository diffs in code tasks, final environment state) rather than the harness’s self-reported success.

Key Findings

• Creation Success: While models successfully generate runnable systems, they consistently lag behind mature, human-engineered references, particularly in domains requiring long-horizon planning like "Search and Research" and "Code." Conversely, models perform competitively in "Writing" and "Machine-Learning Experimentation."
• Cost-Performance Disconnect: There is no direct correlation between high token consumption (execution cost) and benchmark success, indicating that more expensive, "heavier" harnesses are not necessarily more capable.
• Evolution Instability: While LLMs are capable of using feedback to iterate on their harnesses, the evolution process is inherently unstable. Performance often fluctuates across revisions, and improvements frequently fail to transfer to held-out tasks.
• Executor Dependence: The paper demonstrates that a harness's effectiveness is tied to the specific executor LLM ($L_E$). A harness optimized for one model may collapse when paired with a different model, highlighting a significant lack of transferability across architectures.

In conclusion, the authors demonstrate that while frontier models possess the latent capability to act as "system builders," the ability to maintain and improve these systems over time remains an open, challenging frontier.