An AI Code Review Is Only as Good as Its Refuters 🧪

Trail of Bits opens last week's post, Auditing in the age of (good enough) AI, by noting that security firms have published plenty of posts about pointing an agent harness at a codebase and finding dozens of bugs, theirs included. Their point is that agentic code review is only one part of how they use AI. The part they want to talk about comes earlier: agents building the tooling and formal models that make the review deeper.

We agree, and our own review logs from this week show why.

What they actually spent the time on

The target was the Miden VM, a new zero-knowledge VM with its own assembly language, MASM, and almost no developer tooling. Over six months before the review, their agents built an LSP server, a decompiler, a static analysis engine and a Lean model of the VM executor. The analyses found over 400 locations where type validation could be improved, plus one high-severity bug: a prover-supplied remainder in mod_12289 was never validated, which would let a malicious prover forge Falcon signatures. The Lean work produced 95 machine-checked correctness proofs and found two subtle bugs that the existing unit tests had missed.

The method matters more than the totals. While building the decompiler, agents decompiled a randomized set of procedures and compared the output with the original MASM, and each issue became a regression test. The Lean kernel checked the proofs, so humans only had to audit the theorem statements. In both cases the agent produced the work and something independent decided whether it held.

A bug count measures the finder

"Our agent found N bugs" tells you that candidate findings are now cheap. It leaves out the cost of reading the list: each item is a claim someone must reproduce, rank, then fix or dismiss. A finder that is wrong a third of the time is not two-thirds as useful. It is a queue with no way to tell which third is wrong. A finding that states an input and the wrong output, by contrast, is one anyone can run to see whether it holds.

Trust comes from what happens after a finding is found:

1. A concrete failure scenario: an input and the wrong output.
2. An independent attempt to refute it.
3. A test that fails without the fix, proven by putting the bug back.
4. A probe of the running system after release.

What the loop caught

This week we ported a batch of SEO and routing fixes from one of our sites to a second site built from the same codebase. Before deploying, we gave each of several reviewer agents one lens: the canonical/hreflang registry, middleware and redirects, robots and content, or build and deploy. Every finding needed an input, the wrong output and a fix. Two independent skeptic agents then tried to refute each one, marking it refuted if they could not reproduce it from the actual code.

Twelve findings were upheld and none were refuted. The worst: the blog index had been changed to fall back to another language's posts when a locale had none of its own, but the cards still built links under the page's locale. Posts are prerendered only in their own language and dynamic route params are off, so in 13 of 15 locales every card on the blog index linked to a 404. Others: Chinese legal pages newly self-canonical in the sitemap still had English titles and descriptions; a language switcher linked to removed documentation; the AI assistant prompt pointed to a removed FAQ page; the HTTPS redirect kept a non-standard port, sending http://host:8080/ to https://host:8080/, where the CDN never serves TLS; and a shared open-redirect guard sat in a file the cross-repository drift check did not cover, where the copies already differed by one harmless line.

A second, smaller change added a one-hop redirect for mis-cased locale prefixes (/zh-hant to /zh-Hant) and routed tRPC paths through the middleware. It went through a design phase first: one agent designed each change and a separate critic tried to break each design. Investigating the design turned up a real hole on both production sites. Before any authentication logic ran, the middleware skipped all checks for any path matching an unanchored .png/.jpg regex. So /api/trpc/edge/forum.search,x.png?batch=1 returned data (HTTP 207) without passing the API's default-deny. The fix: the static-asset skip never applies to a path that needs a session.

What nearly slipped through

The critics found no runtime flaw in the designs, but three in the proposed tests. An inventory test listed 8 files when there were 9. Adding one new test file would have pushed the lint error count one over the release gate's ratchet and blocked the release. And one source-order test searched for a string that was missing from two of the three copies of the file, so the test could not fail:

// Vacuous: a missing needle makes indexOf return -1,
// so any position is "greater". This can never fail.
const canonical = src.indexOf("<anchor that is not there>");
const hop = src.indexOf("<anchor that is there>");
assert.ok(hop > canonical);
 
// Fixed: order only means something if both anchors exist.
assert.notEqual(canonical, -1, "canonical anchor not found");
assert.notEqual(hop, -1, "hop anchor not found");
assert.ok(hop > canonical);

The quieter near miss came later. The post-implementation review used 4 lenses and upheld 0 findings, but 3 low-severity findings were never verified: the verifier agents hit a usage limit and died. We did not count them as refuted. We read them ourselves, and 2 were real: the drift check missed a shared smoke-test script, and an inventory test recognised only one spelling of an import path. Both were fixed. Defaulting to refuted is right for a skeptic that tried and failed to reproduce. A skeptic that never ran has not given a verdict.

Every new test was mutation-checked. We put each bug back (reverted a fix, deleted a guard, changed 308 to 307, removed a decoding step, allowed a loop), confirmed the test failed, and restored the code. On that change the tests caught all 9 mutations. After each deploy we ran live checks against production: 44 of 44 on one site, and 29 of 29 regression checks plus 12 of 12 gateway checks on the other.

A short checklist

• No scenario, no finding. Require an input, the wrong output and a fix.
• Refute with a different agent that defaults to refuted when it cannot reproduce the finding from the real code.
• Track three states: upheld, refuted, unverified. Unverified goes to a human, never to the refuted pile.
• Mutation-check every new test. Put the bug back and watch the test fail.
• Fail on empty lookups. Any indexOf, regex or file list a test depends on must fail when it finds nothing.
• Probe production with the same scenarios after release.

Trail of Bits notes that a failed side project now only costs tokens. Findings are cheap in the same way. What stays expensive is treating an unchecked finding as true, or as false. Put the effort into whatever decides whether each one holds.

Source: Trail of Bits