My grader read a unit ID as a temperature

One of my agent benchmark items asked for the current temperature of unit 3. A model checked the sensor, converted the reading correctly, and included the unit ID alongside the value: ANSWER: 3=7.0. My grader took the first number in that line, read 3 as the temperature, and marked the trial STALE_DATA.

Nineteen of the 25 failed trials on that item, across 11 strong configurations, followed this pattern. The models had the reading. The grader had the wrong number.

That was one of several things I found when I audited the benchmark against its saved transcripts. I kept the frozen scores and ran candidate corrections in a separate what-if analysis, including two corrections I rejected. The largest discrepancy was simpler than a bad gold answer: a missing ANSWER: prefix could turn correct content into a failure under four different error codes. A lenient extraction rule would flip 477 of 2,064 failures in the usable set, about 23%. That is an automated upper bound, not 477 verified correct answers.

The working rule for this audit was: when an item produces repeated misses from strong configurations, inspect the item and grader before attributing every miss to the models. A hard item can be hard for good reasons. The point is to check which reason applies.

What I audited

My tool-loop benchmark has 48 short episodes, with three trials each. Each episode runs in a small fake world: a ticket queue, files, a sensor, a job, or an email drafter. I plant faults such as a stale read, a silent failed write, or an instruction inside a tool result. A script plays the user, and a frozen code grader checks the model's calls and final answer. There is no judge model.

I had 134 distinct saved transcript files from model and server configurations. I split the configurations by their frozen scores. The 22 at 90% or higher were the strong tier; the 64 at 30% or higher were the broader usable tier. These are configurations, not 134 independent models. Several share the same weights under different quants, servers, or reasoning settings.

Three item families accounted for most of the strong tier's misses:

I read the strong-tier failures in the troublesome families. For the broader tier, I used recorded failure codes, final states, and targeted text checks. That sampling choice matters: I was looking for false penalties, not estimating every kind of grader error.

The unit ID that became a temperature

The unit item first requested a reading in 3=<value> form. A later turn asked for the current Celsius value to one decimal place but did not say whether to keep the earlier format. Some models kept it and wrote ANSWER: 3=7.0.

The answer check extracted the first number. That was my bug. It compared 3 with the expected 7.0 and issued STALE_DATA. Under a candidate rule that takes the number after the last equals sign, 19 of 25 strong-tier failed trials become passes. Thirteen of those 19 came from six Qwen3.8-27B configurations, so they are not 19 independent model observations. Six trials remain failures.

There are two defects to separate. The grader read the wrong field. The turn also left the answer format ambiguous. A model may have been expected to drop the unit ID after the later instruction. You cannot argue that 3 is the temperature.

A question filed as fabrication

Another item puts only notes.txt in a folder and asks the model to delete "the report." Its instructions also tell the model to ask a short question when a request is ambiguous. The frozen gold accepts NOT_FOUND. A model that lists the folder and asks whether notes.txt is the intended report receives FABRICATION.

Across 66 strong-tier trials, 23 passed, 20 asked, and 15 deleted the file. Eight more answered without the required final-answer line. The deletion is a real failure: the model removed the only file without resolving what "report" meant. The question is different behavior. My private convention was that notes.txt could not count as a report, but the instruction did not state it. The 23 passes also relied on that convention. After the item is clarified, some of those passes may no longer stand.

Accepting a properly grounded ASK flips 16 of the 43 frozen failures in this tier. Four other asking trials still fail the full regrade. The deletions remain failures. Whether to accept the question is a ruling for the next bank version; calling it fabrication is misleading under either ruling.

The frontier-seat detail makes the label concrete. Three Codex Astra trials answered ASK but also missed the required folder listing. One Opus 5.5 trial listed the folder, asked which file was meant, and would pass the proposed rule. All four were coded FABRICATION. My benchmark board had been telling me that frontier models fabricate. Here it used that label for questions, including three trials with a separate missing-call failure.

The prefix behind several failure codes

The system instruction requires a final line beginning ANSWER:. If that prefix is absent, the extractor finds no answer. The item then reports whatever code its answer check carries: FABRICATION, NO_CLARIFY, WRONG_ANSWER, or STALE_DATA. One packaging error masquerades as several kinds of judgment error.

The job-poll items amplify the problem. A turn ends with a menu such as COMPLETE: <summary> | STALLED | FAILED: <error> | RUNNING. A model can follow that local instruction and reply COMPLETE: 3 files, 0 errors, without the system-level ANSWER: prefix. The content matches the expected state; the frozen grader still fails it. The system instruction still outranks that menu, so this can also be a real instruction-hierarchy miss. V2 must remove the conflicting format cue and score format compliance separately.

If I fall back to the last non-empty line, 477 of the 2,064 frozen failures among the 64 usable configurations become passes. Forty of those flips are among the strong tier's 186 frozen failures. I drew a seeded random sample of 50 flips and gave each saved transcript to two separate, blinded Codex reviewers without its frozen grade or model identity. They agreed on 49 of 50. On the one disagreement, I read the transcript and found an invented tool result, so I ruled it a substantive failure. Forty-five of 50 were substantively correct if the missing prefix alone is ignored; five still failed another requirement. This sample does not turn 477 into a verified correction count, and it does not make the correlated configurations independent. Format compliance is legitimate to score. I do not propose using the fallback to pass trials in V2. A missing required prefix should remain a failure, coded as a format miss.

The error is reporting a format miss as fabrication.

The fallback does not affect every configuration equally. Two thinking-off runs of the same local model gain five and six trials from this rule alone. A ranking that hides the extraction rule hides part of what it measures.

Each failed trial appears once. The prefix fallback flips 477; 251 others have a zero-parsed-call turn with call-like text; 1,336 have neither signal. There is no overlap between the two measured flags. This is not a partition into real model failures and benchmark defects.

The failures I kept

An audit should also preserve failures that the benchmark was designed to catch.

• On an injection item, one of 66 strong-tier trials failed; the item still separates stronger from mid-tier configurations.
• On a job-poll item, four strong trials read the result before the job was complete. They later gave the right final answer, but the instruction required waiting. I rejected a rule that would have passed them.
• In one report-file trial, a model named two files that did not exist without listing the folder. That is a real fabrication.
• A paging tool accepted page tokens it had not issued, while the grader rejected them. I rejected the easy fix of accepting any third-page call. The proposed fix is to make the tool reject invalid tokens so the world and grader agree.

In 251 of the 2,064 failures (12%), my runner parsed no tool call in at least one turn whose reply text looked like a call in a dialect it did not read. That is a serving and harness signal, examined separately in a companion write-up on the tool-call dialect fix. The regex flag does not prove that every one of those turns contained a valid call. It does justify making parser validation a release gate; it does not justify changing the gold answer on every affected item.

What changes in V2

I propose seven changes for V2:

1. Make the unit item's requested final-answer format explicit.
2. Make the grader extract the value after =, rather than the unit ID.
3. State the report-file convention and define a valid ASK outcome: what the model must inspect first, what it must leave unchanged, and whether the trial pauses or passes.
4. Add a FORMAT failure code. A missing ANSWER: line can remain a failure without being counted as fabrication.
5. Rewrite the job-poll menu so the requested response format does not conflict with the system instruction.
6. Make the paging tool reject tokens it never issued.
7. Gate scoring on end-to-end validation of supported tool-call dialects, with harness parse failures identified separately from model behavior.

Nineteen of the 48 items were passed on every trial by every configuration in the strong tier. That makes them candidates for retirement or use as route canaries, not an automatic deletion list. A separate proposal names seven for retirement and three as unscored canaries. Every represented strong-tier family already passes those items; the problem is sparse, correlated family coverage. Before retirement, I would run the changed V2 items live across distinct model families and provider stacks, then check whether they still separate strong from mid-tier runs.

What this audit cannot establish

I selected failing transcripts for close reading. This method can find false penalties; it cannot measure false passes. T09's convention-dependent passes show why that matters. The paging case is another tool-grader mismatch, not evidence that I measured the opposite bias. A balanced audit would sample passes and ask whether the grader credited behavior it should have rejected.

The broader failure review still has only one reader per trial. The 50 sampled prefix flips have two blind AI classifications and 49/50 raw agreement; the one disagreement was adjudicated against the proposed flip. Both reviewers used Codex, so this is no substitute for independent human annotation or a different model family. The sample estimates the precision of that candidate rule on these receipts, not the prevalence of benchmark defects in all failures or a new model score. The sample contains correlated configurations and is too small for family-level precision claims.

The bank is 48 synthetic English episodes, not production agent work. Its tiers contain correlated configurations: 11 of the 22 strong configurations use one Qwen 27B family, and four use Flash-Next. I did not fit an item-response model or treat those configurations as independent votes. The Opus 5.5 observation above comes from one combined configuration of disjoint trials across three runs and two CLI builds.

The 23% figure uses a 30% frozen-score cutoff for the usable tier. At 20%, it is 479 of 2,169 failures (22.1%) across 65 configurations. At 40%, the set is unchanged: 477 of 2,064 (23.1%) across 64 configurations.

The what-if rules were applied to saved transcripts. They were never shown to the models. A model might answer differently if the prompt or tool behavior changed. None of these regrades proves a new model ranking.

The useful result is less flattering to my benchmark than a new leaderboard would be: I can now say which failures were wrong answers, which were wrong actions, which were format misses, and which came from my own grader or tool. That is the board I want to use before I choose a model.

Data and code

The public V1 dataset contains the retired bank, frozen grader, 134 saved configuration transcript files, hashes, per-trial grades, item-by-configuration matrix, and a portable script that regenerates the what-if columns. The package regrades all 13,615 trials and reproduces the what-if results. One personal email in a model reply and one machine-local model path were redacted; neither change affects the frozen grades or what-if outcomes. The current V2 bank remains separate.