Agents review everything, humans review red

I want agents reviewing everything and humans reviewing red. We spend human review attention uniformly across pull requests, as if every change can do the same damage. Start with one price: a human produces tokens about a thousand times more expensively than a model does (the arithmetic is in Human Tokens). That attention needs to earn its keep.

Humans fail too, at measured rates

The usual comparison puts one agent pass up against a human who never misses anything. I'd like to meet that human. The measured version looks like this:

• The figures McConnell compiled put delivered code anywhere from 15 to 50 defects per thousand lines, depending on whose average you take (old numbers, still directionally sound).
• The figures put a normal code review at 20 to 35 percent of defects caught. Formal inspection gets to 55 or 60. That's the kind with a moderator, a checklist, and a scheduled meeting. Same book.

I use those numbers for exactly one purpose: human review leaks. They describe humans reviewing human-authored code. Agent-authored code goes wrong differently, plausible-but-wrong logic and the occasional invented API instead of typos. Those numbers can't tell me how well an agent reviews code.

I tried to stretch them that far in an earlier draft. I was borrowing authority, and calling it evidence.

Most teams buy one review pass because a second reviewer doubles an expensive line item. One draft by a fatigable author, one four-minute skim ending in LGTM, ship. That's what the price of human attention buys. Calling it negligence won't pay for another pass.

So look at what the same money buys from an agent. An hour of engineer time is $100. That buys millions of output tokens, a lot of review over a feature-sized diff. Two things come with those passes, and neither needs a formula:

• No vigilance decay. The peer-review studies put the human cliff at about an hour and four hundred lines. On a big diff, the long boring middle is where human attention is worst, and where the interesting bug is. The fortieth agent pass arrives as fresh as the first.
• No ego. The reviewing agent didn't write the code. It doesn't have to defend it in standup tomorrow.

The tempting formula is (1−p)ⁿ. A 50 percent catch rate over six passes leaves a 1.6 percent miss. I would love to quote that multiplier. I can't.

The only calibrated p we have is the human one, from the human literature. OpenAI's critic experiments are the closest thing on the model side, and they measured catch rates on bugs planted for the test. That's a benchmark. Treating it as calibration would put me right back in the earlier draft.

Redundancy is nearly free, so buy a lot of it. How much it helps stays unclaimed until someone measures it, which is what the pilot below is for.

Two kinds of defect, and only one yields to redundancy

Passes from one model family don't fail independently. They share blind spots, and those are worse than shared noise. Noise averages out, a blind spot doesn't.

So I have Codex review Claude's work and Claude review Codex's. Different training, different blind spots. That's partial independence. They can still miss the same thing.

• Execution defects. Wrong logic, missed edge case, broken test, bad escaping. These are things a model can plausibly catch. Give each pass a separate lens (correctness, security, "prove this breaks"), give it fresh context, and run tests too. A test gives me a check that isn't another model. Those moves can build partial independence. Redundancy works here.
• Intent defects. The code faithfully implements a misreading of what the business wanted. Or it misses a constraint that lived in one person's head and never got written down. If that constraint never reached text the reviewer can read, the catch rate isn't fifty percent and correlated, it's zero by construction. More passes can't find something missing from the inputs. These are disproportionately the expensive failures, the ones where the code did exactly what it was told.

So I still need human review. Spend it uniformly and most of it goes to execution defects the agents already caught, while the intent defects only a human can catch go hungry.

And "six passes" can make an expensive inbox. Six lenses produce six lists with overlapping and sometimes contradictory findings, and reconciling them takes judgment billed at the human rate. Leave that out of the budget and the savings eat themselves.

Two things keep that inbox under control. Each pass emits a structured verdict, one block per lens with finding, severity, and confidence. Exact matches dedupe themselves, most agreement collapses cheaply, and what's left needs judgment. Then the risk tier decides who makes that judgment. On a low-risk change, the gate does it. Nobody reads the findings unless something crosses a threshold.

So the gate needs rules.

Red, yellow, green

I'd use two axes engineering orgs already trust for deploys: reversibility (can we roll it back cheaply?) times blast radius (how many users and systems feel it?). One-way doors and two-way doors. Same questions, aimed at review attention.

Agents review everything, humans review red.

I want these rules holding the gate shut:

Classification is mechanical first. Paths and facts in the diff set the tier. migrations/ is red. Anything under auth or billing is red. A dependency manifest change escalates, a negative coverage delta escalates. Letting an agent assign its own color is grading its own homework, right where models are worst: their blind spots. The agent may propose. The rules get the last word.

The ratchet. Anything (rule, agent, human) can escalate a tier. Only a human with maintain rights can lower one, with a written reason. The label is itself a one-way door. In code, apply max(existing, computed). Use the fresh computation alone and a routine push after a human escalation silently un-escalates the PR. The human shut the gate, the next push opened it again.

Green-washing is the attack. A red change can wear a green diff. The copy tweak on a form that feeds billing. The dependency bump that changes auth behavior. That's why some escalators can't be overridden. The escalator list is the part of the config worth arguing over.

The matrix guards its own config. The tier rules file and the workflow enforcing it both belong under red paths. Changing the gate requires human review. (The gate doesn't get to wave itself through.)

It's calibrated, not decreed. Every bug traced to a merged PR gets tagged with that PR's tier at merge time. If green leaks, the boundary was wrong. Promote that area and attach the incident as evidence. Humans own where the red line sits. Agents bring the evidence about whether it's in the right place.

What this looks like when it's already running

I run a smaller version of this today, for a different risk. On the machines where my agents work, package installs pass through a gate. It refuses any version published within the last fourteen days, any package under a year old, anything from an unfamiliar forge, and anything under ten thousand weekly downloads.

It fails closed. The agents can't grant themselves an exception. The allowlist is mine.

The last time it fired, an unattended build tried to fetch and run a Cloudflare CLI release that had been on npm for zero days. The gate stopped it. So the build fell back to the repo's pinned copy and carried on.

There's nothing clever in there. The same three properties carry it as the matrix: a mechanical floor the agent can't argue past, escalation only a human can reverse, and a rule that covers the tooling itself. Supply chain was easy to justify. That failure mode has a ten-figure damage bill and a Wikipedia page. Review points the same gate at the thing that actually consumes my week.

The ask: one repo, one month

A well-argued proposal still needs a way to fail. So give this four weeks, one repo.

Week 0. Write the tier rules from that team's own incident history. My examples don't know their codebase. Mirror red paths into CODEOWNERS so red leans on native branch protection, with less riding on custom code that can break silently.

Week 1. Labels only, nothing enforced. File boundary disagreements against the rules file while disagreeing is still free. If more than half of PRs land red, the boundaries are wrong. We've routed attention nowhere and added work to do it.

Week 2. Agent review on every tier, one invocation per lens with fresh context and structured output. Publish token cost per PR next to the baseline human review minutes. In public, both directions.

Week 3. Turn on enforcement. Red requires human approval. Yellow requires agent passes plus a human comment. Green auto-merges on clean checks. That's the one tier where reclaimed minutes are unambiguous, the human minutes go to zero. Then seed three test PRs: one touching migrations, one benign docs change, one modifying the gate workflow itself. They should come out red, green, red. If the third comes out green, stop. Fix the escalators before continuing.

Week 4. Show the numbers. Escaped defects per tier against baseline, review minutes per PR against baseline, token cost per PR, and every boundary move with the incident that triggered it.

Capture the baseline before any of this starts. Without it, I can't falsify the pilot. I can run a marketing campaign.

Write the kill criteria before the data exists too (afterward is a convenient time to discover what I meant). Kill it if green-tier escaped defects exceed the pre-pilot defect rate, or if the team is still disputing more than a fifth of labels in week four. That much dispute means the mechanical floor doesn't fit the codebase.

If the result is ambiguous, extend a month. Expanding to more repos on ambiguity would make this the same unmeasured judgment I'm arguing against.

What isn't proven yet

• No measured catch rate for agents reviewing agent-authored code. The pilot's per-tier escape counts start that work. They don't finish it. I won't quote a defect-reduction multiple until there's at least a quarter of data behind it.
• No generic human error rate borrowed from industrial reliability studies. Those describe procedural tasks under different conditions. The software numbers carry this argument alone.
• Agent-proposed tiers are out of scope for the pilot. The mechanical input has to earn trust before I add a judged one. That's one variable too many.

The method, demonstrated on itself

An earlier draft of this argument went to an adversarial critic agent for about a dollar. Twelve objections came back. One caught a contradiction between two sections that I'd have published without noticing. Four numbers came out because they couldn't survive a hostile reader. I was attached to one of them.

The draft was mine, the skeptical pass was nearly free. Deciding which of the twelve objections were right took twenty minutes of the priciest tokens I've got.

That part stays mine.

Sources

Every stat above, with somewhere to check it. Where a figure lives in a book, the link goes to a page that reproduces it, with the book named underneath.

• Delivered code carries 15 to 50 defects per thousand lines. Reproduced with attribution by Sogeti Labs. Original: Steve McConnell, Code Complete, 2nd ed., chapter 20.
• Informal review catches 20 to 35 percent of defects. Formal inspection reaches 55 to 60. The inspection figures, quoted with attribution, are at Coding Horror. The full table, including the informal-review row, is McConnell, Code Complete, 2nd ed., Table 20-2 ("Defect-Detection Rates"), compiled from Capers Jones and others. Jones's own defect-removal data puts inspections higher, 65 to 85 percent, so the numbers used here are the conservative ones.
• Reviewer effectiveness drops past about 60 minutes and 400 lines. Jason Cohen et al., Best Kept Secrets of Peer Code Review (SmartBear), from the Cisco Systems study of roughly 2,500 reviews across 3.2 million lines.
• Model-side catch rates were measured on deliberately inserted bugs. Nat McAleese et al., "LLM Critics Help Catch LLM Bugs" (OpenAI, 2024).