Agentic Evals: Grade What the Agent Changed, Not What It Said

The original author is Sergii Makarevych (@sermakarevich), who shared an article on agentic evals in an X post dated October 11, 2026. The core claim is simple: grade an agent run by the state it changed, not by the sentences it produced.

This tip carries over the grading order the author stated himself in the thread. The point: what a run changed matters more than the answer.

Start from the same database and check five things in order

In the thread, the author pinned down the grading order. The point: state matters more than the answer.

1. Check that the end state matches the expected state.
2. Check that the required tools were called.
3. Check that no forbidden tool was called.
4. Check that the run stayed inside the step budget.
5. Only then look at the reply.

In the article's example, every run starts from the same rebuilt database. After the run, the grader checks the five items above. Do not reverse the order and read the reply first. A plausible reply with a wrong end state is a failure.

Remember the 10 runs that reply-only grading missed

The numbers the author published show the reply-only trap plainly. A reply-only grader passed 58 of 60 runs. Applying the state check dropped the pass count to 48. The other 10 runs said words to the effect of "your refund is on its way" while no refund existed.

Pass on words, fail on state. That is the first thing to change when introducing agent evals. The unit of grading should be the external effect (record changes), not the fluency of the reply.

Note the limits this method does not cover

The caveats raised in the thread are preserved here along with the source. These five checks cover the happy-path run result. Even when a run passes, the score says nothing about the reach the run still held (what it was allowed to do). One thread comment argues for scoring permissions next to the result.

Unreachable task specifications need separate handling. As another thread comment notes, spending days fixing an agent when the specification asks for a state it can never reach is wasted time. Before writing the grader, verify that the task specification describes a state the agent can actually reach.

Original source

• Original post by Sergii Makarevych (@sermakarevich), 2026-10-11: Agentic evals: how to know whether an AI agent did the job, and will do it again
• Original article: Agentic evals (X Article)
• Author's own grading-order explanation (reply to Leo Oliemans): State matters more than the answer