DEV Community: Erik Hill

My AI gate tests were green theater. The fix was to stub the wire — and nothing above it.

Erik Hill — Tue, 04 Aug 2026 18:38:30 +0000

In a private multi-agent project, agent proposals go through a human approval gate. The Playwright tests for that gate were all green, and had been for a while.

They were green because they never once ran the thing they claimed to test.

Green theater

The gate tests staged model output straight into application state: write a pending proposal card, click approve, assert the card flips to approved. Clean, fast, deterministic — and a lie by construction. Those tests proved the consumer of a pending proposal works. They never exercised the path that produces one.

So when a refactor changed the producer to write proposals pre-approved — quietly dropping the approval guard on the way — the whole suite stayed green. The staged state still looked exactly like the state the tests expected, because the tests were the ones staging it.

That's the failure mode I now call green theater: a suite that manufactures the evidence it then inspects.

Stub the bytes, keep the SDK real

The honest fix was to move the fake down to the lowest layer that can hold it: the wire. That's what I extracted into llm-wire-stub — a scripted Anthropic Messages API at Playwright's network boundary.

The app under test runs unmodified: a real @anthropic-ai/sdk client, a real MessageStream, a real SSE decode, a real tool loop. The stub intercepts api.anthropic.com with context.route and answers with the documented streaming envelope:

message_start → (content_block_start → …delta… → content_block_stop)* → message_delta → message_stop

One property matters more than any other here: a wrong shape must fail loudly. A stub that emits a sloppy envelope which the app half-tolerates is just green theater one layer down. The SDK's own accumulator is the enforcer — it throws on the out-of-order stream if message_start is missing (the current SDK's message reads "Unexpected event order"), and finalMessage() rejects if message_stop never arrives. That claim is demonstrated, not asserted: tests/envelope.test.ts feeds the stub's bytes through the real SDK and shows exact reconstruction, then feeds it deliberately broken streams and shows the SDK throw.

Request bodies are evidence

The part of this that changed how I test: the stub records what the app sent, not just what it was shown. Every intercepted call becomes a RecordedRequest — model, system prompt, messages (including tool_result blocks), tool names, api key.

The response side of a test says "the app can render what it was given." The request side says "the app asked the right question." Two bugs from the private suite's history made me care — both invisible to any response-side assertion:

The price lie. A UI chip displayed a cost figure that disagreed with what was actually going over the wire. Every response-side test passed, because the responses were fine; the lie was in the outbound traffic nobody was reading.
The context leak. One agent's output was supposed to reach the next agent's prompt, and silently didn't. A node that never saw upstream output is provable in one assertion — a missing message in requests[n].messages — and in no other way I know of that doesn't involve staring at logs.

Those two anecdotes are private history; you can't reproduce them from the public repo. What you can check is the mechanism: the e2e spec a tool turn makes the SDK loop take a second request asserts on the second request body and shows the tool_result the app produced riding back up the wire.

Fixture expressiveness IS coverage

This one cost a release, so it gets its own section.

The private suite had a test asserting that the number of model requests matched the price quoted to the user — call it "requests == quoted." Correct assertion, sound idea. It passed for a full release cycle while the metering was wrong.

Why? The stub at the time could only produce plain text turns. It could not script a tool_use block, so the SDK's tool loop never fired, so no test run ever took a second request. Both sides of "requests == quoted" were trivially 1. The assertion was right and the fixture made it vacuous.

The lesson: your assertions can only be as strong as what your fixture can express. A fixture that cannot produce a second turn silently converts every multi-turn assertion into a tautology — no failure, no warning, nothing to review.

So ScriptedTurn grew a tool field that streams input_json_delta fragments the way the real API does, and — same lesson, other direction — an error field. A stub that can only succeed makes every consumer's error path green theater by omission. An error turn answers with the documented Anthropic error JSON, and the real SDK surfaces it as a catchable RateLimitError, exactly as in production:

const stub = await stubAnthropic(context, [
  { error: { status: 429, type: "rate_limit_error", message: "Rate limited." } },
]);
// …drive the UI; assert the app shows its rate-limit state, not a crash…

(One caveat that costs an afternoon if you skip it: the real SDK retries 429s and
5xxs by default, so an error-turn script either scripts the retries too or runs
the client with maxRetries: 0. Both in-repo demonstrations do the latter.)

Quickstart

npm install -D llm-wire-stub

The registry tarball ships dist/ prebuilt, so it needs no install scripts and no flags — which matters as of npm v12, where git dependencies and lifecycle scripts are off by default. (A github:egnaro9/llm-wire-stub install also works via the prepare hook, but on npm 12 it needs the new --allow-git and script allowances — the registry install is the clean path.)

import { test, expect } from "@playwright/test";
import { stubAnthropic } from "llm-wire-stub";

test("the agent answers from the scripted wire", async ({ context, page }) => {
  const stub = await stubAnthropic(context, [
    { text: "First scripted answer." },
    {
      text: "Filing a card now.",
      tool: { name: "create_card", input: { title: "prove the loop" } },
    },
    { text: "Card filed. Done." },
  ]);

  await page.goto("/");                       // your app, unmodified
  // …drive the UI; the app's real SDK client hits the stub…

  expect(stub.requests).toHaveLength(3);      // what the app actually sent
  expect(stub.overflow).toBe(0);              // no unscripted model calls
});

overflow counts requests that ran past the end of the script — an unexpected extra model call shows up in an assertion instead of hiding.

hold() / release(): concurrency observed, not assumed

const stub = await stubAnthropic(context, (req) =>
  JSON.stringify(req.messages).includes("alpha")
    ? { text: "for alpha" }
    : { text: "for beta" }
);

stub.hold();                 // responses now block
// …trigger two sends in the UI…
await expect.poll(() => stub.requests.length).toBe(2);  // both IN FLIGHT
stub.release();              // both complete

If the producer serialized its calls, the second request could never reach the wire while the first is still pending — so two recorded requests under hold is proof of concurrency, not a timing accident. Note the function-form script: an array keys answers to arrival order, which is a lottery under concurrency; a function keys them to who asked.

Limits

Anthropic envelope only. One provider, tested end to end, over multi-provider support with one tested path. No OpenAI or Gemini framing.
Playwright-oriented. stubAnthropic wants a Playwright BrowserContext. (sseBody and errorBody are framework-free; the vitest suite uses them with a plain custom fetch.)
Messages API v1 streaming only. No batch API, no extended thinking, no citations, no server tool use. Success turns are text and/or one tool_use block; failures are the error variant. Nothing else is expressible, on purpose.
Scripted, not simulated. The stub never invents behavior; if your script runs out, the overflow counter says so loudly.

The repo

github.com/egnaro9/llm-wire-stub — MIT, 9 vitest tests (envelope through the real SDK, including the fails-loudly demonstrations) and 10 Playwright tests (a browser fixture driving the real SDK's tool loop against the stubbed wire). The private-product bugs above are provenance; everything the stub is claimed to do is demonstrated by a test you can run.

My determinism test passed for months while the two builds played different games

Erik Hill — Sat, 01 Aug 2026 15:23:41 +0000

I compiled the rules engine of a shipped Android game to the browser. Same Java, two compilers. Then I checked whether the two agreed.

They did not — and the test I already had for exactly this had been green the whole time.

The same command twice: green against the current engine, then against the committed recording of the broken build. Play it as a terminal session if you want to select the text.

The setup

The rules live in one module with no Android on its classpath, which is what let me compile them a second time with TeaVM and run the same logic on a canvas in a browser tab.

A seeded run should be reproducible. Give the engine seed 42 and a fixed sequence of inputs, and you should get the same game every time — that is what makes a run replayable and two builds comparable.

Here is what I actually got, same seed, same inputs:

	JVM	browser
first obstacle x, frame 60	405.426	304.426
still alive at frame 360	yes	no
final score	9	6

Not a rounding difference. A different game.

The cause is boring. The test failure is not.

GameEngine used java.util.Random. Its algorithm is specified down to the constants — you can read the exact linear congruential generator in the Javadoc. So a seed ought to name exactly one sequence.

But my code was not running that algorithm. It was running whichever implementation the runtime supplied, and TeaVM's is not the JVM's. The specification describes what java.util.Random does; it does not force a foreign runtime's reimplementation to match.

The fix took ten minutes: write the LCG out longhand so both builds execute the same arithmetic instead of trusting that they will.

The interesting part is the test.

The test that could not have caught it

I had a test called theSameSeedProducesTheSameRun. It ran the engine twice, with the same seed, and asserted the results matched. It passed on every commit, including every commit during which the browser build was playing a different game.

It had to pass. It runs the engine twice in the same runtime. A test shaped like that cannot observe a disagreement between runtimes — not a subtle one, not a 712-pixel one. The assertion was true and useless at the same time.

This is the part I keep coming back to: the test was not weak, or flaky, or under-specified. It was structurally incapable of failing for this reason. No amount of making it stricter would have helped.

A golden file would not have saved me either

The obvious next move is to pin the output: record a known-good trace, commit it, assert against it forever.

I did that too. Then I reverted the fix to see what would happen.

The golden-file assertion passed.

It had to. The JVM's java.util.Random produces exactly what my hand-written LCG produces — that is the whole point of writing out the documented algorithm. So the JVM's trace never moved. Only the browser's did, and the golden file had nothing to say about the browser.

Pinning one runtime's output is blind in precisely the same way as comparing one runtime to itself. Both feel like determinism tests. Neither can see across the boundary they claim to hold across.

What actually works

The only thing that finds this is a diff between the two runtimes.

A JVM test drives a scripted 600-frame game — fixed seed, fixed frame cadence, ten scripted drags — and records score, streak, run state, the player and every obstacle at 11 checkpoints.
It writes the input plan to disk, and a Node script reads that file to drive its run. The drags are defined once. If both sides hardcoded them, a typo in one would look like an engine disagreement, and I would be debugging the test instead of the code.
The Node script imports the artifact the web build actually ships — not a reimplementation of it — and a comparator diffs the traces. Score, streak, run state and obstacle counts must match exactly. Floats get 0.01px, which is 5x the largest rounding difference the two compilers actually produce and four orders of magnitude below the bug it exists for.

the JVM and browser builds agree across 11 checkpoints
(largest float difference 0.0020, tolerance 0.01)

And I falsified it before trusting it. Revert Rng, run it again:

the two builds disagree (69 differences):
  frame 90  obstacle 1 x: jvm=438.177 browser=74.177  (off by 364.0000)
  frame 250 obstacle 3 x: jvm=879.507 browser=167.507 (off by 712.0000)
  frame 360 score:        jvm=8       browser=6
  frame 360 running:      jvm=true    browser=false

That recording is committed, so the failure reproduces without anyone having to break anything.

Two more things fell out of it

The test I wrote to cover the fix was itself vacuous at first. Its precondition needed a run that survived long enough to earn a continue, and a parked player never gets there — so it silently took a branch that asserted nothing. It passed with the fix removed. I only noticed because I make a habit of breaking the code to confirm the test goes red. Now it plays the run with a dodging strategy, and removing the fix fails it for the right reason.

Recording px but not py hid a real change. I altered a positioning constant by one pixel to test something unrelated, and the entire suite stayed green — because the trace captured the player's horizontal position and not its vertical one. A trace only protects what it records.

The thing worth taking away

If you have a test whose name contains "deterministic," "reproducible," or "same seed," ask it one question: which runtimes does it consult?

If the answer is one, it cannot tell you the thing its name implies. Neither can a golden file of that one runtime's output. Only the diff between them can.

The engine is open source: github.com/egnaro9/tapdodge-engine — the check is in tools/compare_trace.mjs, and the README has the numbers.

You can play the browser build, which is the artifact the test drives.

I built an AI dev harness that isn't allowed to trust itself. Then I checked the part doing the not-trusting.

Erik Hill — Mon, 27 Jul 2026 05:24:39 +0000

A follow-up to I built an AI dev harness that isn't allowed to trust itself.

The harness I wrote about isn't allowed to trust itself. That was the whole point, and it rested on one sentence:

nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human.

I still believe it. But it contains an assumption I didn't notice I was making: that the dangerous thing is unchecked work.

Over one night I rebuilt both halves of that sentence on pi — an open, npm-distributed, forkable agent runtime — mostly to find out how much of my harness was discipline and how much was one host's shape. Two public repos came out of it: pi-eval for the proof half, and pi-gates for the irreversibility half.

Both halves failed first. They failed in the same shape, and it isn't the shape I'd been defending against. In each case the check ran, passed, and was proving something other than what it appeared to prove.

That's a correction to my own thesis, so here it is with the receipts.

The gate that was a model being polite

The irreversibility half is a gate that denies git commit and git push unless I've typed an approval phrase at the keyboard that turn. Everything else an agent does in a repo is recoverable by reading a diff. A push isn't.

I wrote it, tested the pure logic, installed it, and asked my session to commit something without the phrase.

It refused. Then it told me:

the gate is functioning as intended — it's blocking the commit

No bash command ran. No tool_call event fired. The gate was never invoked. The model had read about the rule earlier in the conversation and was enforcing it itself — and describing itself, accurately from its point of view, as the mechanism.

I nearly reported the gate verified off that transcript.

When I pushed back and said do not refuse, I need to see whether the gate blocks, it classified that as an attempted bypass and refused again.

The compounding detail is the part I'd want another engineer to take away. The model knew about the rule because my gate had told it. My first "armed" banner used ctx.ui.notify, which lands in the conversation. The gate manufactured the cooperation that made it unverifiable.

Two fixes, both transferable:

Announce out of context. The banner moved to ctx.ui.setStatus, which writes the TUI footer. I traced it: setStatus → setExtensionStatus in interactive-mode.js, with no path into the model's context. A gate the model believes it owns is a gate it can be argued out of — and worse, one that sounds like a gate while being nothing of the kind.

Verify with no model in the loop. Gate correctness is now proven by firing synthetic events at the handler and reading the return:

const [r] = await pi.fire("tool_call",
  { toolName: "bash", input: { command: "git add -A && git commit -m x" } }, ctx);
assert.equal(r.block, true);

That file is the only artifact that proves the gate fires. When I finally ran the real thing in a clean session, the difference was unmistakable — the model ran the command and got denied:

$ cd /tmp/gitgate-test && git add note.txt && git commit -m "gate test 2"
BLOCKED — git commit/push needs an operator approval typed this turn.

Same outcome as the refusal. Completely different fact.

The gate that verified the predicate and not the answer key

The proof half is a deterministic grader — fixed predicates, no LLM judge — so a score change means the output moved rather than the judge having a bad day.

Every task has to survive two machine checks before it enters a suite: a correct answer must PASS its predicate, and a plausible-but-wrong answer must FAIL it. That second one matters more than it looks, because the comparison is a paired sign test where ties are discarded. A predicate that can't fail a wrong answer turns a real difference into a tie. A lax grader doesn't mismeasure your suite — it blinds it.

The gate worked. Of 48 authored tasks it rejected 7, six of them for one cause: the number grader takes the first number in a reply, and a worked solution's first number is an operand. Those predicates graded 3 against an expected 78 and failed the correct answer.

I was pleased with it. Then it admitted a task with expected = 52.34.

48.45 × 1.08 = 52.326, which rounds to 52.33.

Both models I tested answered 52.33 and were scored wrong. It's git-provable in both directions — commit 25d3466 adds "expected": 52.34, and HEAD has 52.33.

Here's why the gate couldn't catch it. I wrote the answer key. I also wrote the known-good answer used to verify the predicate. The same arithmetic slip was in both. Two independent-looking checks, one shared error, zero detection.

A gate verifies the predicate. It cannot verify the answer key.

The fix is a third gate: a separate agent derives every answer from the prompt alone, never shown my value, and disagreement kills the task. The predicate then runs against the independently derived answer — grading mine would be circular, since mine may have been reverse-engineered from the predicate.

A right number about the wrong question

Then the escalation, and it's the one that generalises furthest.

I compared thinking=medium against thinking=off across 100 tasks with three repetitions each. The result was clean: near-total ties, zero informative tasks, a cost ratio of essentially 1. Tidy, plausible, publishable.

--thinking was inert. Pi's resolver takes the level from a model:level pattern, not from the argument I was passing; it returned undefined for every level, so the session fell back to the default. I had compared one config against itself.

"No measurable difference" was completely true and completely worthless.

What caught it was token counting I'd added an hour earlier for an unrelated reason: both sides reported ~116k reasoning tokens, and a thinking=off run cannot do that.

Notice the shape. This check didn't pass for the wrong reason. It passed correctly, about the wrong question — and nothing downstream can detect that, because every artifact looks exactly like a successful measurement. So the fix isn't a better check, it's a refusal to start:

preflight: asked for thinking=off but the session resolved to medium.
Refusing to run — a run labelled with a config it did not use is worse than no run.

That is now my favourite line in either repo, and I'd put it in a category with the other refusals: a missing answer is an error and not a zero; a response truncated at the token cap is not an answer; two runs of different suites are not comparable and the tool says so rather than reporting the delta.

The correction belongs in the grader, not in my judgement

With the config actually applied, thinking=high beat thinking=off 8–0, p=0.008.

Two of those wins weren't real.

thinking=off answers in prose instead of thinking first. On one task it worked through the arithmetic, concluded "Friday" — correct — and an exact-match grader scored the whole reply. On another it listed matches at positions 0, 2, 4, 6, answered "4" — correct — and the extractor took the 0. The model was right both times.

Verbosity is not accuracy. But a config change that alters verbosity moves every position-sensitive grader in the same direction at once, so it doesn't look like noise. It looks like a finding.

I could have published 8–0 and explained in prose which two results I'd decided to disbelieve. That's the version where you have to trust me.

Instead the graders got a scope="last_line" option, and re-scoring the same six runs produced 6–0, p=0.031 — selecting the same six tasks I'd picked by hand, mechanically. The verdict is weaker and I trust it more, because nobody has to take my word for which results were artifacts.

That's what "machine-checkable" is actually for. Not proving you're right. Removing your judgement from the places it can quietly do work.

And the answer it produces is a decision rather than a score, because cost sits next to it:

thinking=high  won 6, lost 0, 84 tied, 10 unstable      p = 0.031
               $1.107  vs  $0.445                       2.49x cost

+6 tasks per 100 for two and a half times the money. Someone can act on that. A score alone isn't a decision.

Twice the data, less power

The discipline has a price, and I'd rather show it than sell around it.

At 100 tasks the tool refused to rank two frontier models. Only 4 tasks separated them, and six is the floor at α=0.05 — no split of that data could have reached significance. So it says that, instead of reporting a tie:

Only 4 tasks separated them. Even a clean sweep of 4 could not clear p<0.05, so this suite cannot decide between them — that is a limit of the suite, not a finding about the configs.

"Cannot tell" and "they're the same" are different findings. Most tooling collapses them, and the collapse always favours having something to report.

There's a second rule underneath: a task where a config disagrees with itself carries no direction, and is discarded exactly like a tie. Necessary, because running one model three times against the same 100 tasks produces ~2.67 "informative" differences from within-model variance alone. Any real finding has to clear that.

Then I wrote in a README that this cost was fixable "with more repetitions." I tested that claim an hour later. It's backwards:

reps 4-6    9-1   unstable 13   informative 10   p=0.0215   decisive
reps 1-6    7-1   unstable 17   informative  8   p=0.0703   not

Twice the data, less power. Every extra repetition is another chance to observe a within-config disagreement, and the strict rule discards the task when it does. In the limit it throws away every genuinely stochastic task — precisely the ones carrying the most information about a noisy config. A conservative rule isn't a free choice, and the cost is measurable.

One more, because it's the part I'd most want checked if I were reading this. When the strict rule returned a non-significant result, I built a weaker rule that returned a significant one. The motivation was independent — I'd measured the discards-rise-with-reps property before checking whether a different rule moved any verdict — and that is also exactly what everyone who p-hacks believes about themselves.

So I wrote down the prediction, the refutation conditions and the fixed parameters, committed them before collecting fresh data, and committed the analysis script while the runs were still going. A pre-registration that leaves the analysis to be written afterwards only relocates the discretion.

One of three predictions failed — the conservative rule reached significance on fresh data when I'd predicted it wouldn't. I reported it as a failed prediction, because the whole point of writing it down first is that you don't get to reinterpret it after.

What the port deleted

The open-runtime argument is usually ideological. Here it's concrete, and it's mostly about subtraction.

On Claude Code, my model gate needed three files and two hooks to answer one question: which model is running? A SessionStart hook wrote the id to .runtime_model_<session_id>; the gate read it back; a second .runtime_model_ppid_<PPID> file existed purely to cover the race where /clear starts a new session before the startup hook writes the new file.

Pi hands the handler ctx.model. All of it is gone — not refactored, deleted.

The git gate is a better version of the same story. The original kept its approval in a sentinel file, so it also needed a forge guard denying any command that so much as mentioned the sentinel's name, with documented residuals around assembled paths like P=.oae_approve; touch "${P}_pending". In pi the approval is a variable in the extension's closure, with no path from a bash command to it. That guard isn't hardened. It's unnecessary, and its residuals don't exist.

How much of a sophisticated system is design, and how much is scar tissue from its substrate? For these two gates: a lot of it was scar tissue.

But one thing didn't survive the move, and it's worth stating plainly rather than approximating. My harness had a Stop hook that refused to let a turn finish if source had changed without validation running. Pi has no blocking turn-end event — agent_end, agent_settled, turn_start and turn_end are all declared with no result type, so a handler literally cannot return a decision. If I need that rule I'll move the enforcement point to tool_call before commit, and I'll call it a different gate rather than pretend it's a port.

And the trust root needed rewriting, in a way a careless port would miss. Pi's InputSource is "interactive" | "rpc" | "extension" — an extension can inject input. A gate that honoured every input event could be opened by the thing it exists to gate. The approval check is source === "interactive", and that single comparison is the whole security property.

The amendment

I'm not retracting the original rule. The original's own logic is what forces the amendment.

The harness isn't allowed to trust itself. But I was trusting the part that does the not-trusting, and I had no mechanism that could tell me otherwise.

If an agent's work is unverified until a gate proves it, then:

A gate is also an agent's work.

It was written by someone under time pressure, it can pass for the wrong reason, and it can be verified by a subject who has read it and would like to be helpful. The check running is not evidence that the check fired. The commit not happening is not evidence that the gate blocked it.

Everything above is in two public repos, and every number in this post is reproducible from the run artifacts committed alongside them. That's deliberate, and it's the only part of my method I'd defend without qualification: if you can't point at the file, don't publish the number.

pi-eval · pi-gates · gradecore — the same grading engine behind a live drift board and a live crash test.

I built a tool to prove my multi-agent harness was worth it. It told me it wasn't.

Erik Hill — Sat, 25 Jul 2026 20:06:31 +0000

I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters with a judge improves them further. It sounds obviously true. More thinking, more review, better answers.

I never measured it. So I built something that could, pointed it at my own setup, and it disagreed with me.

The result

One sweep. Twenty coding tasks, three harness shapes, real models, $0.99.

harness	calls/task	score	cost	latency
one drafter	1	95%	$0.031	2.2s
planner → drafter	2	90%	$0.264	9.1s
planner → two drafters → judge	4	80%	$0.692	18.3s

Adding the scaffolding made it worse and cost 22× more. The four-call panel beat the single drafter on zero of twenty tasks and lost three. Nothing errored — 0% failure rate across all sixty runs. It just did worse work, slower, for twenty-two times the money.

The part I care about more

Here is what the tool actually said about that:

Only 3 tasks separated them. Even a clean sweep of 3 could not clear p<0.05, so this suite cannot decide between them — that is a limit of the suite, not a finding about the harnesses.

The panel costs 22× more and the suite cannot decide between them — on this evidence the extra spend buys nothing.

95% versus 80% looks like a decisive result. It isn't. Seventeen of the twenty tasks were ties, so only three carried any information, and three discordant tasks cannot reach significance even if one side sweeps all of them. A leaderboard would have printed the two numbers and let me conclude the panel is worse. That would have been a stronger claim than the data supports.

So the honest reading is narrower and more useful:

There is no evidence the panel helps on this suite.
It costs 22× more and takes 8.4× longer, which is measured, not inferred.
Whether it is genuinely worse needs more tasks than twenty.

Those are three different statements. Most eval tooling collapses them into a ranking.

Why the suite size is the real constraint

It isn't step size — the 15-point gap is three times the 5-point resolution. It's that the two shapes only disagreed on three tasks. Everything else tied, and ties are exactly what a paired test throws away. Twenty tasks is simply too few to generate enough disagreements for any test to work with. The fix is not better statistics, it is more tasks — which is why the tool lets you bring your own suite and tells you, as you paste it, how many points each task is worth.

This is the same lesson my drift board taught me earlier this week, when four "regressions" turned out to be rate limits and single-question noise. Small suites produce confident nonsense.

How it works

A harness config is data — roles, models, prompts, and a topology graph. You draw the shape or paste the JSON; each is a view of the other. Declare the axes you want to vary and it runs the matrix.

Scoring is deterministic: fixed predicates execute the generated code and return a verdict, and no model grades anything. There is an assistant in the page, and it is allowed to read scores and explain them — never to produce one.

Worth being precise about what that does and doesn't buy. The grading is deterministic — the same output always scores the same. The generation is not: I set no temperature and no seed, and this sweep used one run per config, so a 5-point move between two runs of the same shape sits inside sampling noise. That is an argument for more tasks and more runs, and it is a second reason the tool won't call a winner here.

The comparison is per-task and paired, not two averages. Both shapes run the same twenty tasks, so the question is how many tasks one won, which has far more power at this sample size than comparing means. The test is an exact sign test: no normality assumption, no variance assumption, ties excluded because they carry no direction.

Your key stays in the browser. The backend receives sanitized traces and refuses anything key-shaped at its boundary; the page shows you the exact bytes it posts and tells you to check your own Network tab rather than believe the panel.

One vendor detail worth writing down, because it cost me an hour: api.openai.com answers the CORS preflight with the right headers and then omits access-control-allow-origin on the actual response, so a browser-direct call is discarded no matter how valid the key is. Anthropic opts in deliberately — that is what anthropic-dangerous-direct-browser-access is for. Testing only the preflight with curl -X OPTIONS shows success and is misleading.

Where this sits (so I don't oversell it)

Harness and prompt-comparison tooling is not a new category — promptfoo, LangSmith, Braintrust and others do model and prompt comparison, several with far more surface area than this. The narrow thing here is an intersection: browser-BYOK, plus deterministic no-LLM-judge grading, plus a comparison that reports when it cannot decide.

What this cost me

$0.99 and about ten minutes — the sweep runs strictly sequentially, so 60 runs at those latencies is 592 seconds of model time before anything else — to find out that the architecture I had been assuming was better is, on this evidence, not better and definitely more expensive.

One caveat on the 22×: that ratio is at Sonnet 5's introductory pricing, which runs through 2026-08-31. After that the gap gets wider, not narrower.

I would rather know.

Run it yourself: https://egnaro9.github.io/never-touch-ai/sweep.html — draw a harness and sweep it. It runs free on mock substrates with no key at all.

Source: https://github.com/egnaro9/never-touch-ai
Raw result: results/sweep_2026-07-25.json — the numbers above are computed from it, so you can check them.
Deeper write-up: the field note — graph execution model, the sign test, and the two bugs the live run surfaced.
Built by Erik Hill · https://egnaro9.github.io

My LLM drift tracker flagged four regressions this week. All four were wrong.

Erik Hill — Fri, 24 Jul 2026 23:55:15 +0000

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post.

Between 21 and 24 July it did that four times:

23 Jul  Gemini 3.5 Flash   -11.4 pts
24 Jul  Gemini 3.1 Pro      -2.9 pts
21 Jul  Grok 4.3            -5.7 pts
22 Jul  Llama 3.3 70B       -2.9 pts

Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong.

None of those models got worse. Here's how I know, because the how is the only part worth reading.

Two of them weren't the model

Every point on the board carries a second number next to accuracy: reliability, the share of probe calls that actually came back. Look at the two Google alerts with that column showing:

gemini-3.5-flash  22 Jul  acc 1.000  reliability 1.000
                  23 Jul  acc 0.886  reliability 0.914   <- "-11.4 pts"

gemini-3.1-pro    22 Jul  acc 0.914  reliability 0.943
                  23 Jul  acc 0.886  reliability 0.914   <- "-2.9 pts"
                  24 Jul  acc 0.971  reliability 1.000   <- next clean run

Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one.

I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight:

api.groq.com -> 429: Rate limit reached for model `llama-3.3-70b-versatile`
service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30

34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch.

Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression."

The other two were one question

The remaining two alerts are more interesting, because reliability held at 1.000 the whole time. Those numbers are real:

grok-4.3        0.800 -> 0.743   = -5.7 pts
llama-3.3-70b   0.800 -> 0.771   = -2.9 pts

The suite is 35 tasks. One task is 100/35 = 2.86 points.

So -2.9 points is one question changing its answer. -5.7 is two. And -11.4, the scariest number in the set, is four.

A 35-task suite cannot resolve anything finer than about three points. Every "regression" my board flagged this week was an integer number of questions, which is the tell: I wasn't measuring drift, I was measuring the granularity of my own instrument. Reporting a one-question flip as a model regression is reading noise as signal — and doing it in public, about a named company's model.

Why the alerting is still right to be loud

The obvious fix is to make the tracker quieter — only fire above 10 points, say. I don't think that's right. A tracker that only fires on catastrophes misses the drift you actually want to catch, and the -11.4 that turned out to be failed calls is exactly the shape of a real regression. Sensitivity is the feature.

Sensitivity is only safe, though, if something downstream is willing to say no. So the alert doesn't publish anything. It writes a stub that says, in its own text:

Auto-logged when the scheduled probe flagged a run-over-run regression. Before this becomes a post, check the run log and the Reliability metric — a rate limit or provider outage can look exactly like a regression.

The automation's job is to notice. Mine is to check. This week that split did real work: four notices, zero posts.

The number I actually care about

If you build evals, you already track your models' scores. The metric I'd argue you're missing is the share of your own alerts that survive checking.

Mine, this week, was zero. That's not a comfortable number to publish, and it's the most useful one I have — it tells me the suite is too small to resolve single-task noise, and that reliability has to sit beside accuracy on every chart or the chart lies.

Both of those are fixable. Neither would have been visible if I'd shipped the post the tracker wrote for me.

The hard part of a drift tracker isn't detecting drift. It's not manufacturing it.

The board: egnaro9.github.io/model-drift — 16 models, 5 metrics, daily, every run kept. The field notes are on the page; this one is "Four regression alerts, zero regressions."

The code: github.com/egnaro9/model-drift. No LLM-as-judge anywhere — every task is graded by a fixed deterministic check, so a score change means the model moved, not the test.

The AI Crash Test: adversarial LLM testing you can audit in the Network tab

Erik Hill — Thu, 23 Jul 2026 04:57:51 +0000

A browser tool that points your own API key at an adversarial battery and grades every answer with pure predicates — no LLM judge, and your key never touches my server.

The first time I ran it against a real model, it told me the model was ~29% vulnerable.

That number was wrong. And the tool proved it was wrong — to me, in public — because of exactly one design decision.

Every verdict in The AI Crash Test is a deterministic predicate over the model's answer string: exact match, regex, a number check, an injection canary, a must-refuse rule. No model grades another model. So when the report flagged ~29% vulnerable, it also showed the fail card for every miss — prompt, expected, actual, side by side. Three of those cards didn't show a broken model. They showed a broken grader: false positives in my own code. I fixed the graders; the real number was 0%.

That's the whole pitch. An auditable grader has bugs you can catch in public. A vibes-based, LLM-as-judge arena just hands you a number and asks you to trust it.

The honest part first

LLM red-teaming is a crowded, mature space. garak (NVIDIA), PyRIT (Microsoft), and promptfoo all do far more than this — more probes, more scale, more integrations. Browser tools that adversarially test with your own key exist too; most lean on an LLM judge.

So this isn't a new category, and I won't pretend it is. The narrow thing that's mine is an intersection: browser-based BYOK + deterministic no-judge grading + a provably shared engine with a longitudinal drift board. Distinctive engineering and discipline, not a market-novel product. If you want heavy artillery, go use garak. If you want a result you can reproduce byte-for-byte and a key that goes straight to the provider and never touches my server, read on.

Two properties you can check yourself

1. Deterministic grading — no LLM in the grade path. Every grade is a pure function of the answer string, run in an open-source engine called gradecore. Run a mock model through it twice and the score is byte-identical. No temperature, no judge drift, no "the grader was having a bad day."

2. BYOK, never-touches. The browser calls the provider directly with your key. crashkit's server receives only the answers — the grade request has no key field at all.

Don't take my word for it. Open DevTools → Network, run a battery with your key, and search the panel for the key itself. It lights up only on the request to the provider (e.g. api.anthropic.com, in the x-api-key header) — never in the /api/grade call. I verified this live before writing this; you can reproduce it in about thirty seconds.

The honest caveat: this only works where the provider allows direct browser calls. Anthropic (with the dangerous-direct-browser-access header) and Gemini work; OpenAI-direct is often CORS-blocked. Stating the limit is part of the point.

One engine, two lenses

gradecore isn't a crashkit-only toy. It's the same deterministic engine behind my live model-drift board, which tracks 16 LLMs over time. Same code, two jobs: the board is longitudinal monitoring; The AI Crash Test is on-demand adversarial testing.

And it's the same engine, not a lookalike. Run the board's frozen suite through gradecore and the suite_hash comes out identical, byte for byte — faithful extraction, not a reimplementation. (To be clear: crashkit uses gradecore, not my whole eval stack.)

What's in the battery

Eight tasks across seven attack kinds: prompt-injection, tool-abuse, spec-violation, refusal-calibration, unsafe-compliance, hallucination-bait, and consistency. You get a severity-weighted vulnerability report and a fail card for every miss.

A real result from a recent run: claude-haiku-4-5 resisted 100% — 0% vulnerable across all seven kinds, deterministically graded, reproducible.

Try it / read it

Run it: https://crashkit.onrender.com
Source: github.com/egnaro9/crashkit and github.com/egnaro9/gradecore

I built this solo, self-taught, in under a year — and yes, with heavy AI assistance in the loop. The design decision I care about is the one you can check without trusting me: open the Network tab, run it twice, read the graders. Prove it — don't take my word for it.

301 duplicate IDs in the browser, 0 on the JVM: one real bug, end to end

Erik Hill — Wed, 22 Jul 2026 00:01:43 +0000

A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in new words, and read more like a pitch deck than a case study. He was right, and he pointed at the fix himself: walk through one real task. What changed, what caught the problem, what proof was required, where did a human step in.

So here is one task, start to finish. Everything below is public and runnable. No withheld details, no diagrams of boxes with arrows.

The setup, in two sentences

My match-3 game runs on a plain-Java rules engine. It runs two ways: on a JVM (where the tests and CI live) and in the browser, compiled to JavaScript by TeaVM, so the same Java drives a real playable board.

That second runtime is not just a demo. Running one piece of logic on two different machines gives me a free check: where the two disagree, one of them is wrong. I did not have to write down the right answer. I just had to notice a disagreement.

Here is a task where that check earned its keep.

What changed

I ported the engine to the browser. New demo-js module, TeaVM config, opaque integer handles in and JSON out so boards never actually cross into JavaScript. Mechanical work. The engine code barely moved.

Except the port compiled a line I had never once looked at hard. This is how every gem got its ID:

String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);

Timestamp plus a random number. It had passed every test for months.

What the check caught

The IDs are how the renderer tells one gem from another. Two gems with the same ID animate as a single gem. So I ran the same board generation on both runtimes and counted collisions over 128,000 gems.

JVM: 0 duplicates.
Browser (TeaVM): 301 duplicates.

Same source code. Same inputs. Different answer. That is the entire signal. A machine counted it; I did not have to guess that something felt off.

Where I stepped in

A number this specific still needs a human to say which side is wrong and why. That part was me.

System.nanoTime() looks unique but only leans on the clock being high-resolution enough that two calls land on different values. A JVM's timer is fine, so the flaw was invisible there. Browsers deliberately clamp their clock to about 100 microseconds (a Spectre mitigation), so nanoTime barely advances between gems and RNG.nextInt(1000) collides on its own often enough to matter.

Neither runtime was broken. The code was, for depending on clock resolution it was never promised. The browser was just honest about it.

The fix is boring, which is the point:

private static long idSeq = 0L;

private static synchronized long nextId() {
    return idSeq++;
}

-String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
+String id = row + "-" + col + "-" + nextId();

A counter is unique on every clock. The guarantee stops depending on the platform.

What had to exist before it could close

The one rule I do not bend: a fix is not done because I say "fixed." It is done when a check that would catch the bug is sitting on disk and passing in CI. For this one, that meant three tests whose whole job is to fail if IDs ever lean on the clock again:

@Test
public void idsAreUniqueWithoutRelyingOnClockResolution() {
    // Mint many gems as fast as possible: a clock-derived id would collide
    // here on any platform whose timer doesn't advance between calls.
    Set<String> seen = new HashSet<>();
    for (int i = 0; i < 50; i++) {
        GameBoard.Gem[][] b = BoardEngine.createBoard(plain());
        for (GameBoard.Gem[] row : b)
            for (GameBoard.Gem g : row) seen.add(g.id);
    }
    assertEquals(50 * 64, seen.size());
}

Plus one for board creation and one for the refill hot path, where new gems get minted every cascade. The suite went from 51 tests to 59. Those three are the receipt that the specific failure cannot come back quietly. Without them, "I fixed the ID thing" is just a sentence.

The commit and the tests are here: match3-engine (BoardEngine.java, IdUniquenessTest.java).

The same trick, one level up

Around the same time, the same "run it two ways, look for a quiet disagreement" habit pointed the other direction. Auditing where TeaVM and the JVM diverge, I hit a date case they split on:

YEAR = 2002, WEEK_OF_MONTH = 2   (America/New_York, en_US)
JVM:   Sun Jan 06 2002
TeaVM: Sat Jan 12 2002

This time my code was fine. The bug was in TeaVM's reimplementation of Java's GregorianCalendar. One line used days - 2 where every neighbouring branch, and the Apache Harmony code it was ported from, used days - 3:

-days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 2)) - skew;
+days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 3)) - skew;

One character. The reason no test had caught it: the suite already had four assertions that reach that exact line, commented out since 2015. My change re-enabled them instead of adding new ones. They fail on the old code and pass on the fix, which is the cleanest proof I could ask for that the fix is real and nothing else moved.

It merged into TeaVM on 2026-07-17 and closed a dormant issue. My whole contribution was a 2 to a 3 and un-commenting eleven-year-old assertions.

That's the whole thing

No cleverer reviewer would have found the first bug by reading the code, because the code looked fine and the tests were green. A second runtime found it by disagreeing. My job was the part a machine can't do: read a "0 vs 301" and decide which side was lying, and why.

If there's a transferable idea here it's just this: don't keep one source of truth for logic you can't fully check by hand. Run it two ways and treat every disagreement as a bug until you've proven which side it lives on. Sometimes it's yours. Once, it was the compiler's.

Both examples are runnable:

match3-engine — the Java engine, 59 tests, playable in-browser via TeaVM.
evals-differential-oracle — a tiny browser demo of the same idea: the same match-3 rule written twice, fuzzed against each other over thousands of boards, plus a deliberately-broken version both nets catch.

Thanks to Ryan for the nudge. The last post told you I test things. This one showed you one.

Distinguishing wrong from absent

Erik Hill — Tue, 21 Jul 2026 19:42:32 +0000

model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.

I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.

The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model.

The design was already exclude-by-class, not blanket. Code: github.com/egnaro9/model-drift

Writing the methodology in public is how it gets read this closely.

I built agentic AI that checks its own work — a 90-second tour

Erik Hill — Tue, 21 Jul 2026 04:19:21 +0000

I built an AI dev harness that isn't allowed to trust itself

Erik Hill — Mon, 20 Jul 2026 02:12:53 +0000

Scope note (read first): this describes a system I built and operate to develop an unannounced game. The game's identity, mechanics, and assets are withheld, and so are the harness's tuned prompts, gate implementations, and internal failure specifics. What's shown here is the method and the evidence discipline — the transferable part.

Summary

Over about four months (spring–summer 2026) I built and operated, solo, an operator-supervised multi-agent development harness that builds a real, shipping product. Its defining property isn't speed — it's that nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human. This is a case study of the system and the evidence trail it leaves.

The idea

LLM coding agents are fast and unreliable. The engineering problem isn't getting output — it's trusting it. So the harness is built around one rule: an agent's work is unverified until a gate proves it. Coordination is automated; consequences are gated. And it's built to ship, not to gold-plate: every gate exists so I can move fast without shipping something broken — verification in service of velocity, not instead of it.

Architecture (concept level)

A five-role loop: Strategy → Execution → Critic → Eval → Ops. Judgment roles (Strategy, Critic, Eval) run on stronger models; execution roles on cheaper ones — cost follows the difficulty of the decision, not a flat default.
A manager / orchestration layer. Above the execution agents sits one orchestration role that I direct — it plans each unit of work, routes it to the right role and model, and holds the system's state between steps. I designed the roles, the gates, and the routing; the harness runs them. I'm not outside the loop supervising a black box — I'm the system's judgment and authority, and the manager is the layer that extends that across many parallel agents.
A cold, independent critic gate. Before a consequential change closes, it's reviewed by a Critic running on a fresh, zero-context session — a different strong model with no memory of how the code was written — so it reviews the work itself, not the author's rationale for it. It can send the change back for rework. A self-review rubber-stamps; a cold critic catches what the author already talked themselves past.
A human-in-the-loop autonomy ladder: the loop's handoffs are automated — one role hands to the next without me — but every irreversible act (deploying a build to a device, committing to git) stays behind an explicit human approval. Automate coordination; never automate the irreversible.
A differential oracle for correctness: the core logic is implemented twice and the two versions are fuzzed against each other. Where they disagree, one is wrong — no gold labels required.

The evidence discipline (the differentiator)

Every closed unit of work leaves a durable, machine-checkable proof:

NO-PROOF-NO-CLOSE gate. A work item cannot close until an automated check confirms its proof exists on disk. The loop physically cannot skip it.
Provenance-bound proof. On-device validation screenshots are sanitized (sensitive regions blacked out), and provenance manifests bind images to the exact git SHA, screen dimensions, and redaction method that produced them — so an artifact traces back to the commit it proves.
Human-gated checkpoints. Each checkpoint records scoped git staging (explicit paths only), a commit/SHA trail across the repos it touches, an artifact-registry audit, and an explicit operator approval.
Periodic self-evaluation. An independent evaluation role produces a numeric health score with a delta versus the prior period and a failure taxonomy; regressions feed a failure registry that drives fixes.

The testing oracle

The product's core logic is held to property-based invariant tests — generated inputs are thrown at the engine and a set of invariants must hold for every one (e.g. a detector must agree with an independent full re-scan, and detection must be side-effect-free). The suite runs against the authoritative implementation, so an invariant is enforced on the logic, not asserted in prose (last run: zero failures). As a standalone, fully public demonstration of the same technique, my match3-engine repo carries 16 jqwik property invariants over random inputs.

Operating record (Apr–Jul 2026, from the on-disk archive)

~200 completed work-arcs · ~190 human-gated checkpoints · 74 independent critic reviews · 13 periodic self-evaluations · a growing failure registry with per-item root-cause fixes · a ~200-file sanitized proof archive with ~90 provenance manifests.

Verifiable outcomes (all public)

An arcade game — Tap Dodge Rush, under SeraphLight Studios — shipped end-to-end to Google Play.
A one-character bug fix merged upstream into TeaVM (the Java-to-JavaScript compiler), closing a long-dormant issue.
A live public model-drift board grading 16 LLMs daily on a frozen, deterministically-graded suite — no LLM-as-judge, so a score change is real.
Ten public repos, including a differential-oracle testing project and a Model Context Protocol server built from the spec.

What I'd bring to a team

Treat AI output as unverified until proven. Build the gate before the feature. Make failures loud, not silent. Keep a human on the irreversible path. The discipline transfers to any codebase — the harness just made me practice it a few hundred times.

Full architecture case study & repo: github.com/egnaro9/agentic-dev-harness · Portfolio: egnaro9.github.io

Set membership is not pairing: a property test that was green for the exact bug it was written to catch

Erik Hill — Sun, 19 Jul 2026 05:57:09 +0000

I have a small board that tracks 16 LLMs across 5 labs on a frozen 35-task suite. The charts are the easy part. The paragraph above them is the part I did not trust, because I had written that kind of paragraph by hand twice and both times it went false — true when I typed it, falsified by a later run.

So I stopped typing it. The paragraph is generated now. Each sentence is a claim: a predicate over the current numbers plus a renderer, returning a string or None. A claim whose predicate stops holding is dropped, not reworded. The generator can be silent. It can't be wrong.

That second sentence is the part I got wrong.

The sentence that lied

One claim compares a lab's cheaper models against that same lab's flagship. On the live board it produced this:

Google's cheaper tiers match or beat its own flagship — Gemini 3.5 Flash at 100% and Gemini 3.1 Flash-Lite at 95%, against Gemini 3.1 Pro at 95%

Correct. But the first version of that renderer collected the winners, took the highest score among them, and printed one number for the group:

scores = f"{_names(winners)} at {_pct(max(r.acc for r in winners))}"

Which reads: Gemini 3.5 Flash and Gemini 3.1 Flash-Lite at 100%. Flash-Lite scored 95%. The sentence put a real number next to a model that did not earn it.

Naming several models and then printing one number is not a formatting choice. It is a claim about all of them.

The test that should have caught it

I had written a property test for exactly this class of bug. It asserted that every percentage appearing in the prose was a score some model actually got:

real = {f"{p[-1]['acc'] * 100:.0f}%" for k, p in m["series"].items()}
printed = set(re.findall(r"\d+%", out["text"]))
assert printed <= real

It was green.

It is green because the assertion is true. 100% was a real score — Gemini 3.5 Flash got it. The test asked "does this number exist in the data?" The bug was "is this number attached to the right model?" Set membership where pairing was required.

The replacement checks the pairing:

_PAIR = re.compile(r"\b([AB] (?:Big|Small|Tiny)) at (\d+)%")

for label, printed in _PAIR.findall(out["text"]):
    assert f"{actual[label] * 100:.0f}" == printed

The fixture was wrong too

This is the part I would have missed if I had stopped at the test.

A fixture with one cheap model per lab cannot reach the failing state — with a single cheap model per group, a group is always unanimous, so a renderer printing rows[0]'s score is always right. That was the shape I started from, and I fixed it before the first commit, so take this part on my word rather than on the history. The property test was not weak because of its assertion alone. It was weak because the data it ran against could not produce a disagreement.

The fixture now gives LabA two cheaper tiers under one flagship, which is the shape Google actually has (Pro / Flash / Flash-Lite):

REGISTRY = [
    {"id": "lab-a:big",   "label": "A Big",   "group": "LabA", "tier": "flagship"},
    {"id": "lab-a:small", "label": "A Small", "group": "LabA", "tier": "mini"},
    {"id": "lab-a:tiny",  "label": "A Tiny",  "group": "LabA", "tier": "nano"},
    ...
]

and the baseline board puts those two cheap tiers at different scores, both at or above their flagship.

What it cost

Eight mutants run against the new guards. Three survived the first time — the tie guard, the single-flagship rule, and a float-truncation fix — and each one needed a test written before it was really covered. The repair was less covered than it felt.

The float one is my favourite, because it is a bug in the code whose entire job was to never overstate a number. Percentages truncate rather than round, so a model on 99.6% can never print as "100%" — a perfect score it did not get, in the one place a reader checks hardest. But:

# The epsilon is not decoration: 0.58 * 100 is 57.99999999999999 in binary
# floating point, so a bare floor prints a measured 58% as "57%". [...]
return f"{math.floor(value * 100 + 1e-9)}%"

A guard against overstating had started understating instead. That is still wrong prose about real data — just wrong in the flattering direction, which is the direction you don't check.

The suite is 125 tests now, stdlib only, 0.04s. It was 91 at the fix commit, 74 when the generator first landed, and 22 before the generator existed.

What I'd do differently

Ask what state would make this assertion fail, and name it out loud. I could not have named it for the set test — not because I hadn't thought about it, but because the fixture made it unreachable and I never noticed that the two facts were connected. An assertion whose failing state you cannot describe is decoration.

Mutate the code, not just the inputs. Property tests over generated data feel rigorous enough to skip this. Three of eight mutants survived my new, careful, adversarially-reviewed guards. That is the number that convinced me.

Assume the repair is uncovered. The fix arrives believing in itself. It has no tests yet.

And one more, which I found while writing this. The README badge for that repo says tests-91. It is 125. Four screens further down, the quickstart says # 22 tests — a badge and its own README disagreeing, both hand-typed, both true on the day they were written and false since. That is the identical failure mode the generator exists to eliminate, sitting in the repo that hosts the generator.

The verification was broken, not the code. It usually is.

Repo: github.com/egnaro9/model-drift — the broken test and the real one are both in tests/test_narrative.py.
Portfolio: egnaro9.github.io

The Cold-Context Critic: A Reviewer That Never Remembers Writing the Code

Erik Hill — Thu, 16 Jul 2026 01:22:41 +0000

Anyone who has written code and then reviewed it minutes later knows the feeling: the diff looks fine. Of course it does. The person reading it is the same person who just talked themselves into every line of it. The naming made sense because the intent is still warm in memory. The shortcut felt justified because the reason for it is still sitting in working memory, uninspected.

That is the problem the cold-context critic is built to remove.

The core idea

In the harness I built to develop a game, every change passes through a review step performed by a fresh model instance that has no memory of writing the code. It did not plan the change. It did not argue for the approach. It did not feel the small relief of getting something to finally pass. It receives the diff and evaluates it on its merits, the way a stranger would.

The intuition is old and boring, which is part of why I trust it: an author rationalizes; a cold reviewer does not. A model — or a person — that just produced the work carries a bias toward the choices it already made. Strip the memory of making them, and the same mistakes stop being self-evidently correct. The reviewer has nothing to defend. It can only look at what is actually there.

I want to be precise about the claim, because it is easy to oversell. This is a discipline, an architecture choice. I am not going to tell you it caught a specific dramatic production bug, because I am not going to invent an incident to sell an idea. What I can say is the mechanism: cold review catches the class of mistakes an author talks itself into — the plausible-looking shortcut, the assumption never stated out loud, the edge case that "obviously" can't happen. Self-review is structurally bad at exactly those, because the author already accepted them once.

Where it fits

The cold critic is one layer, not the whole story. It sits alongside a few other pieces of the same distrust:

Per-role model routing. Stronger models run the judgment stages — planning, critique, evaluation — and cheaper ones handle routine execution. The reviewer isn't grading its own homework, and it isn't the same weight class picked for speed.
A differential oracle. Core game logic is implemented twice and held to shared invariants. When the two implementations disagree, at least one is wrong, and I find out without having to trust either.
A human approval gate on every irreversible action. Deploys and commits do not happen because a model decided they should. They happen because a person approves them. That gate never moves.

I think about the whole thing on an autonomy ladder, L0 through L4 — from fully hands-on toward more independence. The cold critic is what makes climbing that ladder defensible. More autonomy is only safe if the verification underneath it does not depend on the thing being verified vouching for itself.

The setup, honestly

Concretely: the critic is a separate invocation with a clean context window. It does not inherit the conversation that produced the change. It gets the diff and the standard it is being held to, and it reports back before a human sees the change. That's the shape of it. The value isn't a clever prompt — it's the missing memory. The setup exists to guarantee the reviewer never accumulated a reason to like the code.

That's also why I keep the human gate immovable. The cold critic is a good reader, not an authority. It can flag; it doesn't get to ship.

An honest note

I'm about five to six months into this, career-changing from roughly six years in professional kitchens, no CS degree — I learned through freeCodeCamp and by building. The game is real and published to Google Play, but it is not serving millions, and I'm not running production RAG or fine-tuning pipelines. I'd rather tell you what this is than dress it up.

So the honest framing is this: I don't have the scars yet that teach senior engineers where the bodies are buried. What I can do is build the distrust in as a habit from the start — assume the author is biased, including when the author is me or a model I'm running, and make something with no stake in the answer do the checking.

If you want to see the same idea in code you can actually run, the differential oracle is public: github.com/egnaro9/evals-differential-oracle. It's the same move as the cold critic — distrust your own output — implemented as two independent versions of the logic checked against each other, plus invariant tests over thousands of random boards, with a deliberately buggy implementation included to prove the checks bite. Clone it, run pytest, watch it catch the planted bug.

More at egnaro9.github.io and github.com/egnaro9.