HarnessDev.

What a harness is and how it's tested

• What a harness isThis surrounding execution infrastructure, commonly termed the agent harness, manages the execution loop, tool use, context, failure recovery, and result verification that turn model outputs into actions.
• Same model, different harnessWith identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 but 49.6% inside Codex CLI.
• The questionA practical question is whether LLMs can assist harness engineers--or even take over such a role--in building and continually improving the harness.
• Not ordinary code editingWhen a model modifies its own harness, it is editing the execution substrate through which it acts: the change alters how the model itself observes, plans, and recovers in all future tasks.
• A model is not a systemThe rapid growth of FDE hiring reflects a simple fact: a capable model is not yet a working system, and today the gap is closed by human engineers.
• Creation and EvolutionIn Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it.
• Capability and efficiencyCapability measures whether the harness works: we run it on held-out downstream tasks and report task-level success rates. Efficiency measures how many executor-model tokens the frozen harness consumes.
• A frozen, reusable artifactThe submitted artifact is a runnable harness that is frozen and then reused across downstream tasks.
• The weak seedThe seed has no agent loop, task decomposition, tool policy, context management, persistent task state, verifier, retry or recovery logic, or stopping rule.
• The seed scores zeroUnmodified, it produces an empty or partial artifact and scores zero on every downstream benchmark.
• One to three practice casesThe creator receives a task-family specification, tool and permission constraints, a short design tutorial, and one to three development cases. It never sees the human implementation or the hidden evaluation set.
• Six control modulesThe creator must implement the six control modules: execution loop, tools, context, state, lifecycle and verification.
• Four domains, five benchmarksCreation covers four domains and five downstream benchmarks; together, the suites contain 2,207 unique downstream instances.
• Six creator modelsWe evaluate six creator LLMs: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro.
• Three builds eachFor RQ1, we independently create and evaluate three harnesses for each creator-benchmark pair and report avg@3.
• No shortcuts foundNo harness obtained score through a prohibited route, and no run is excluded on these grounds.

Creation: what models built

• Best creator vs referenceUnder Self-Eval, Opus 4.8 has the highest overall score (67.8), but remains below the human-engineered reference (86.2).
• The gap depends on the domainWriting harnesses approach the reference, whereas Search shows the largest gap and Code also remains behind.
• ML experimentsOpus 4.8 and Gemini 3.1 Pro lead MLE-bench with medal rates of 32.9 and 32.4.
• Creation scores (Self-Eval)Opus 4.8: SWE-Pro 69.3, Terminal-Bench 64.8, MLE-bench 32.9, EQ-Bench3 84.6, BrowseComp 52.4. Human references: 80.0, 88.8, 24.0, 83.7, 92.2.
• Harness defects, not the model77.8% of failed Data tasks are attributed to harness defects, showing that the bottleneck is not only executor capability.
• Fewer lines, best scoreGemini adds the fewest lines (1,006) yet obtains the best Terminal-Bench score (68.8), suggesting that focused changes and frequent verification matter more than code volume.
• Components that never fireOf 108 component instances in Code, 72 trigger in real runs, 18 have only partial evidence, and 18 are never observed; all unobserved instances concern state and memory.
• State and memory unusedState and memory are the clearest gap. No checkpoint event appears in 26,679 recorded task trajectories.
• Checks are mostly syntacticValidation is also mostly syntactic; 441 of 2,325 executed Data tasks produce degenerate submissions that no harness detects.
• Read, fix, re-verifySelf-test count alone is a weak signal. Testing helps when the creator reads the failure, makes a targeted change, and re-verifies it.
• Cost varies, payoff doesn't followMLE-bench token use varies by about nineteen-fold, yet higher cost does not reliably produce a higher score.

Transfer and evolution

• Swap the runnerOpus shows the opposite pattern: its Self-Eval SWE-Pro score falls from 69.3 to 33.0 under Gemini.
• A hard-coded step limitAn Opus Code harness that performs well under Self-Eval but nearly collapses under Gemini because it hard-codes a 120-step limit around the original executor.
• Repeated searchesIn the Opus Search harness, the duplicate-query rate rises from 10.1% to 88.2% after the executor changes.
• Some harnesses gainSeveral Qwen and DeepSeek harnesses improve under Gemini; Qwen gains 17.6 points on BrowseComp and 12.9 on MLE-bench.
• When capability transfersCapability transfers only when its prompts, tool protocol, budgets, and stopping rules remain compatible.
• Ten full evaluationsDuring development, it receives results from a fixed 100-task SWE-Pro feedback set and all 89 Terminal-Bench tasks. The controller provides a budget of ten post-H0 full-evaluation pairs.
• Held-out tasksEvery official version is additionally evaluated on 630 SWE-Pro instances disjoint from the feedback set, and these scores are never shown to the creator.
• Gains shrink on unseen tasksAll five self-runtime creators improve on the visible feedback pair, but the gains shrink on held-out tasks.
• Feedback vs held-out gainsFeedback-pair gains of +3.0 to +13.9 points under self runtime; held-out-630 gains of +1.43 to +4.44. Under fixed Gemini, held-out changes run from +2.70 to -10.32.
• Local program searchEvolution therefore resembles local program search around runtime feedback, where deletion can be as useful as addition.
• Diagnosis is the weak stepFailure diagnosis remains the weakest step: the dedicated trajectory interface is called only twice.
• 99 said success, 48 passedOpus gives the clearest positive example: it finds that 99 of 100 runs report success while only 48 pass, traces the gap to premature completion, and adds a completion check.
• Held-out gainsAll five self-runtime declarations improve over H0 on held-out tasks, with gains of +1.43 to +4.44 points and a mean gain of +3.11.
• Under a fixed runnerTransfer is weaker under the fixed Gemini executor: only Opus improves on held-out tasks, while the other three lineages regress.
• Close to a coin flipAcross 64 comparable switches, feedback and held-out scores move in the same direction only 34 times (53.1%), and only 2/9 declared versions are held-out optimal.
• Where intelligence accumulatesIf model weights are one place intelligence accumulates, the harness is another: explicit, inspectable, testable, reusable, and continually improvable.

Limits

• Not human abilityExceeding 100% means exceeding that external reference, not exceeding human ability.
• References are not pairedThese references pair a human-engineered harness with the executor model used by that system; they are not scores obtained with one common executor.
• Run-to-run noiseEvolution is not monotonic. 27 report gains that remain inside the repeated-run noise band, two have clear positive evidence beyond the noise band. The same commit can vary by about ±4.75 pair-score points.
• Unreliable for picking a versionVisible feedback is therefore useful for local search but unreliable for final selection.
• One trajectory per cellEvolution currently has one trajectory per creator-runtime cell and one unfinished main-runtime cell, and its post-freeze held-out evaluation covers SWE-Pro only.
• Coverage and baselinesThe four categories cover many but not all real deployments. Human baselines are uneven and not guaranteed optimal.
• Treat as untrusted codeAnyone reusing the generated harnesses should treat them as untrusted code and isolate them more strictly than we did.