Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Automating harness evolution fails when the action space is messy, the traces are huge, and edits cannot be attributed. AHE fixes all three with observability, turning every edit into a falsifiable contract so the loop improves instead of flailing.

A coding agent's performance rests on its harness as much as its model: the system prompt, the tools that expose the shell and file system, the middleware that manages context and execution, and the memory. Harness engineering is still a manual craft because automating it runs into three walls: the action space is heterogeneous across editable components, trajectories bury the actionable signal under millions of tokens, and the effect of any single edit is hard to attribute. AHE (Agentic Harness Engineering) closes the loop by matching each wall with a form of observability.

The result is that every edit becomes a falsifiable contract, a self-declared prediction checked against the next round's outcomes, so evolution proceeds without collapsing into trial and error. Ten iterations lift Terminal-Bench 2 pass@1 from 69.7% to 77.0%, beating the human-designed Codex harness (71.9%) and self-evolving baselines, and the frozen harness transfers to other benchmarks and model families without re-evolution.

The problem it attacks

As base models advance, the manual loop of reading traces and hand-crafting harness edits cannot keep pace, and it faces two structural obstacles. Long, unstructured trajectories yield little actionable signal, so an evolving agent drowns in tokens. And tightly coupled harness code makes it hard to say which edit caused which change, so the loop cannot tell improvement from noise. Without solving both, automatic harness evolution degenerates into random edits. AHE's bet is that if the evolution agent is given structured context over a clear action space, with each edit's predicted effect later verified, it can reliably converge.

How it works

AHE rests on three matched observability pillars. Component observability gives every editable harness component a file-level representation at a fixed mount point: system prompt, tool description, tool implementation, middleware, skill, sub-agent config, and long-term memory, each a loosely coupled file, so the action space is explicit and every edit is revertible. Experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus the evolving agent can actually consume. Decision observability pairs every edit with a self-declared prediction (which tasks it should fix, which it might regress), verified against the next round's task-level deltas.

The falsifiable-contract idea is the linchpin. Each edit declares a targeted fix plus a predicted impact (expected fixes and at-risk regressions). The next round compares the predicted-fix and predicted-regression sets against the observed task-level deltas, producing a per-edit verification. The paper finds the evolve agent is reliable at predicting fixes but largely blind to regressions, which it flags as the clearest target for improvement: regression foresight.

Results

The harness transfers without re-evolution

The harness is evolved once on Terminal-Bench 2 with GPT-5.4, then frozen and reused. On SWE-bench Verified it reaches the highest aggregate success while spending 12% fewer tokens than the seed, where the self-evolving baselines ACE and TF-GRPO both regress below the seed and spend more. Across three alternate model families the frozen harness gives consistent gains, and crucially the largest gains land on the bases furthest from saturation.

Cross-family gains beating within-family ones is the evidence that the evolved components encode general engineering experience (coordination patterns weaker models lean on more) rather than benchmark-specific tuning fitted to GPT-5.4.

Where the gain lives

A component ablation localizes the improvement. Tools, middleware, and long-term memory each carry the gain on their own, while the system prompt edited alone regresses. This matches the recurring harness-engineering finding that the durable wins are in the factual structure (tools, context management, memory), not in prompt wording, and it explains why the harness transfers: factual structure is model-agnostic in a way prompt phrasing is not.

What it changes

AHE's contribution is making harness evolution observable enough to be autonomous. Prior self-evolving harness work either edits a single surface or treats the harness as an opaque blob; AHE tunes the full harness as a combinatorial whole, but only because component observability gives a clean action space and decision observability makes each edit's effect attributable. The falsifiable-contract framing is the transferable idea: an edit you can verify against a prediction is an edit a loop can learn from, which is exactly what separates AHE from trial-and-error self-evolution.

Where it sits among prior work

Limitations

The evolution runs on Terminal-Bench 2 with GPT-5.4 and a step budget and per-task timeout fitted to that setting, which partly explains the smaller within-family gains. The decision-observability check shows the evolve agent predicts fixes well but is blind to regressions, so the contract is only half-closed; unforeseen regressions still slip through until a later round catches them. As with the other systems in this study, the gains are measured on the benchmarks the harness was evolved or transferred against, and the verifier is the benchmark's own pass signal, so a harder held-out distribution is untested. Marginal regressions appear on the three smallest SWE-bench repositories.

Learnings

1. Observability is what makes self-evolution work, not the search. The three pillars (explicit components, distilled experience, verified decisions) are what stop the loop from flailing. Give the evolver structure and attribution and it converges.
2. Falsifiable edits beat blind edits. Pairing every change with a checkable prediction turns evolution into something auditable, and it surfaces the weak spot (regression foresight) precisely.
3. Durable harness gains live in tools, middleware, and memory, not the prompt. The ablation is a clean restatement of the LIFE-HARNESS lesson: factual structure transfers across models; prompt wording does not, and editing it alone can regress.
4. Evolve on cheap, transfer to the rest. A harness evolved once and frozen carries the largest gains to weaker, less-saturated models, so the cost amortizes across a model family.

Glossary