Scaling Laws for Agent Harnesses via Effective Feedback Compute

Counting tokens, tool calls, and wall time barely predicts whether an agent succeeds. Count only the feedback that is informative, valid, non-redundant, and retained, and the scaling law snaps into focus.

A harness decides how a model calls tools, receives feedback, verifies intermediate states, stores memory, and revises solutions, which makes harness design a form of test-time scaling. But the usual scaling analyses parameterize that process by raw expenditure: tokens, tool calls, operations, wall time, cost. None of those distinguishes useful feedback from redundant or unstable interaction. This paper introduces Effective Feedback Compute (EFC), a trace-level coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for later decisions, normalized by task demand when comparing tasks with different feedback needs.

The payoff is a much better scaling law. Raw tokens and tool calls explain limited variation in failure rate (R2 of 0.33 and 0.42); a strong multivariate baseline reaches 0.88; EFC-based coordinates reach 0.94, and task-demand-normalized Oracle-EFC reaches 0.99. The conclusion: harness scaling is governed less by how much computation is spent than by how efficiently raw budget is converted into durable, task-sufficient feedback.

The problem it attacks

Test-time scaling lets you spend inference-time compute to get evidence from the environment instead of growing the model. But unlike pretraining, where model size and data are clean axes, harness scaling has no good coordinate. Measuring raw budget treats a useful verification and a redundant retry as equal, so the scaling curve is noisy and the lessons are unreliable. If you cannot measure the right thing, you cannot say whether spending more helps, and you certainly cannot compare harnesses fairly.

How it works

EFC measures the amount of feedback a trace actually puts to work. A unit of interaction counts toward EFC only if it clears four tests: it is informative (it changes what the agent knows), valid (it is correct, not a hallucinated or erroneous signal), non-redundant (it adds something not already known), and retained (it is carried into later decisions rather than forgotten). Two derived quantities follow. Harness efficiency, EFC divided by raw compute, measures how much effective feedback a harness extracts per unit of budget. Task-demand normalization, EFC divided by task demand, measures whether the extracted feedback is sufficient for the task at hand.

For real settings without oracle access to the environment state, the paper provides Estimated-EFC (recoverable from the trace before the outcome is known) and NRS-EFC (non-redundant stable EFC, emphasizing retained over transient feedback). The decomposition is the conceptual core: harness design controls the raw-to-EFC conversion (efficiency), while the task sets the demand, and success depends on both extracting effective feedback and having enough of it relative to demand.

Results

The matched-budget intervention is the clincher

The most convincing result holds raw cost and tool calls fixed and varies only feedback quality. Success rises from 0.27 to 0.90. Same budget, same number of tool calls, radically different outcome, because the feedback the agent retained and used was better. This is direct causal evidence that the quantity to optimize is effective feedback, not spend. On mixed real traces, raw-compute baselines have near-zero or even negative fit while NRS-EFC/Dtask reaches 0.92, and Estimated-EFC/Dtask reaches 0.93 even when computed before the outcome is known, so the coordinate is usable predictively, not just in hindsight.

What it changes

This paper does not propose a harness; it proposes the right axis to measure one. That matters for the whole self-improvement program, because every harness-evolution method in this study optimizes against some success signal, and EFC explains what actually drives that signal. It reframes "spend more compute" into "convert budget into retained, valid, non-redundant feedback, and make sure there is enough for the task." For anyone building or evaluating a self-improving loop, EFC is a diagnostic: if a harness edit raised raw spend but not EFC, it should not be expected to help, and the data backs that up.

Where it sits among prior work

Limitations

Oracle-EFC needs access to the environment's true state to judge validity and informativeness, which is available in controlled and executable settings but not in general; the paper's answer is Estimated-EFC and NRS-EFC, which approximate it and predict slightly less well. The four properties (informative, valid, non-redundant, retained) require operational definitions that may not transfer cleanly to every domain, and efficiency is shown to be slice-dependent, so a single number does not capture every regime. As a measurement paper it predicts success rather than improving it, so the practical payoff depends on harness designers actually optimizing EFC. The prospective holdout (R2 = 0.85) is strong but lower than the in-distribution fits, the expected gap when moving to new data.

Learnings

1. Measure feedback, not spend. The single most useful idea: raw compute is a bad scaling axis for harnesses because it counts redundant and invalid interaction equally. EFC fixes the axis, and the fit jumps from 0.33 to 0.99.
2. Feedback quality is causal, not correlational. The matched-budget intervention (0.27 to 0.90 with budget fixed) is the cleanest demonstration that what the agent retains and uses, not how much it spends, drives success.
3. Normalize by task demand. Enough feedback for an easy task is too little for a hard one; dividing EFC by demand is what makes the coordinate comparable across tasks.
4. This is the missing yardstick for self-improvement loops. For the RSI study, EFC gives a principled way to ask whether a harness edit actually helped: did it raise effective feedback, or just cost. A loop that cannot prove gain per effective-feedback unit is the cost liability the book warns about.

Glossary