Building a Self-Revising LLM Harness Loop with Held-Out Evaluation Sets
Instead of hand-coding agent call structures, use an isolated scoring set and execution reports to automatically generate, grade, and select harness file revisi
On October 19, 2026, X user @virgilxbt summarized a paper titled The End of Harness Engineering by Chinese researchers and shared a workflow in which the harness (the orchestration code or configuration file around an LLM that manages tool calls, prompt routing, and execution order) rewrites itself across rounds instead of being edited by hand. The core takeaway from the post is straightforward: "grade the file, don't write it."

Image source: https://x.com/virgilxbt/status/2111890105028022608
In conventional agent development, engineers manually wire up tool sequences and track changes through commit messages—an approach @virgilxbt characterizes as "a guess with a commit message." Rather than treating the orchestration layer as a permanent, hand-crafted artifact, this workflow treats the harness file as disposable and shifts human effort entirely toward defining evaluation criteria.
Core Principle: Keep the Solver Frozen and Own Only the Scoring Set
Throughout the loop, the underlying model—referred to as the solver (the base LLM executing the actual task)—remains completely frozen with no weight or model updates. Only the outer harness file that organizes how the model runs is modified from round to round.
The human engineer writes and maintains only one artifact: the scoring set (an isolated collection of benchmark tasks used strictly to measure correctness). According to the source post, three rules govern how that scoring set is handled:
- Hidden from the proposer: The model generating harness edits never sees the scoring set.
- Never optimized against: Candidate edits are not tuned directly on test items, preventing overfitting to benchmark questions.
- Used solely for grading: Every candidate harness produced in a round is evaluated against this held-out set to pick a winner.
As @virgilxbt frames the shift, developers stop arranging model calls and tools by hand and instead focus on deciding what counts as correct and reading the resulting scores.
Four-Step Mechanics of the Self-Revising Loop
The self-revising harness loop described in the post repeats a four-step cycle:
- Feed in the execution report: Pass the execution report (logs and failure traces from the previous round's run) into the proposer.
- Generate 8 candidate edits: Produce 8 modified versions of the harness file based on what failed or stalled in the report.
- Score on held-out questions: Run all 8 candidate harnesses against held-out questions (evaluation tasks kept separate from the proposer) and record each candidate's score.
- Keep the winner and repeat: Retain only the highest-scoring harness file as the baseline for the next round.
According to @virgilxbt, the cited paper reported that across 41 rounds—without a human ever opening an editor on the harness—the held-out score rose from 0.31 to 0.64. Documenting a parallel run of the same loop in practice, @virgilxbt reported using 300 Kimi K3 workers across 41 rounds, resulting in 37 automated rewrites with zero manual edits.
Caveats and Unverified Details to Keep in Mind
Based on the source thread and available metadata, several practical constraints apply before adopting this workflow:
- Dependence on grading set integrity: In the thread,
@wwardennnoted that the evaluation set is the component that requires the strictest protection, while@open_is_freepointed out that selection pressure does not disappear—it moves one level up to the question of who grades the grader. If the scoring set is narrow or flawed, the loop will optimize the harness toward those flaws. - No direct paper link in the source: While the post names The End of Harness Engineering and cites the 0.31-to-0.64 score progression, neither the main post nor the replies include a direct arXiv or repository URL, so those benchmark figures rely solely on
@virgilxbt's summary. - Compute and parallelism footprint: The author's own 41-round run relied on 300 Kimi K3 workers, meaning teams replicating an 8-candidate-per-round search on smaller setups will need to budget token usage and wall-clock runtime accordingly.