Nicolas Borson

Checking the fixes an AI reviewer suggests

Tue, 03 Nov 2026 00:00:00 GMT

There is a failure mode that comes up in almost every account I read from teams running AI reviewers at scale. The reviewer leaves a comment. The comment is correct. The author accepts the suggested fix. The fix is wrong.

Wrong in the quiet way, the kind that passes CI and gets merged.

I have no number on how often it happens on our side. As a manager I am rarely the one reading the diff, and that is precisely why I want the habit written down before we lean on these tools. If the people who do read the diff have a checklist, I never need to.

A small example

Florian Bruniaux showed one in his talk "Du code au merge" at Dev With AI in Paris on October 12, from a real PR on his codebase at Méthode Aristote. A test covered two failure cases, a missing document and a forbidden access, and asserted only that each one raised an error. The reviewer flagged it: the test accepts any error, so it can pass for the wrong reason. Correct diagnosis.

The bot's suggested fix was to assert the same error message in both cases. More precise than before, and still incorrect, because the two cases should produce different errors. The right fix was to assert the error specific to each case, then rerun.

Three steps, and the bot got two of them. Verify the problem it reports. Then verify the correction it proposes. Two separate acts, and people skip the second because the diagnosis was so convincing.

Where the reviewer's time goes

Anthropic published numbers from their internal reviewer this month (How the Claude Code team does code review), and one of them explains why this matters at scale.

About 59 percent of what the reviewer kept flagging in later rounds was code that was already there before the PR. Round after round, the reviewer was digging up old bugs in the surrounding code and asking the author to own them.

One PR on their CLI went through more than fifty review rounds overnight before its author stopped it.

Their fixes are worth copying even at a tenth of the volume.

Label the finding. Important, Nit, or Pre-existing. A pre-existing bug deserves a ticket. Fixing it in this PR is someone else's call.

Cap the rounds. After three review-and-fix rounds, the reviewer stops suggesting and asks the author to reconsider the approach. After the first few passes, it reviews only the diff.

Flag the class. If the same mistake appears in five places, one comment that names the pattern, so the author fixes all five, including the four nobody pointed at.

Never repeat. A finding already posted is not posted again.

Every finding gets an answer. Fix it, or resolve the thread with a reason. A later commit that fixes it closes the thread automatically.

The results after those changes, on their internal PRs: the share passing after a single round went from 65 to 78 percent, and PRs needing five rounds or more dropped from 8.9 to 1.5 percent.

Watch it work

The second half of the problem sits with the author.

Agents are good at explaining why their code works. The explanation is usually convincing and not always right. Daniel Visca at PostHog has a rule for this (Stop being the code review bottleneck): do not accept an argument that the code works when you can watch it work.

In practice that means every change ships with a way to observe it. A command to run, a request with its expected response, a screenshot, a state dump. What gets validated is the behaviour you can see. Anthropic's version is the same: the test output goes in the PR.

Which brings up the rule that Bruniaux put on one slide and that I think deserves to be printed above every merge button.

A test that is in progress, skipped or cancelled is not a passed test.

And its sibling: a new push changes what remains to be verified. Commit A was tested and reviewed. The agent pushes a fix, now the branch is at B. A validated does not mean B validated. Before merging B, which tests must rerun, and has anyone read the diff from A to B? After every fix, rerun the tests and the reviews that the fix touched, and only those.

When a bot gets it wrong twice

The loop closes on one more habit, and it is the cheapest one.

When a reviewer makes the same mistake twice, or misses the same case twice, look at the harness around the reviewer before you look at the PR. Bruniaux's team opens a GitHub issue for it, with the exact case, the expected result and the linked PR. Then they fix the cause, which is usually one of three things: a missing regression test, an instruction that needs sharpening, or context the reviewer did not have. Then they replay the case and watch the next few PRs.

Anthropic does the same at a larger scale. Past incident and revert PRs, confirmed by experts, form an eval set the reviewer runs against daily, and every change to the reviewer is scored against all of it. Incidents become tests for the reviewer itself.

The habit is the same at both scales. The reviewer is a piece of software your team maintains. It has bugs. You file them.

The checklist I would hand the team

For a reviewer comment:

Is the problem real? Read the code before you read the comment again.
Is the proposed fix right? Separate question. Often no.
Is it in this PR's scope, or pre-existing? Pre-existing gets a ticket.
After the fix, which tests and which reviews need to run again?

For the reviewer:

Three rounds, then stop and rethink.
Same miss twice, file an issue against the harness.

None of this needs a product. The first checklist is a habit, the second is a label, a round counter and an issue template. That is also why I am writing it down now, before we have a reviewer running on every PR and before the first quiet bug. What it does need is a written statement of what a good finding looks like in your repo, and that is the subject of the next post.

Who merges when the reviewer is a bot

Tue, 27 Oct 2026 00:00:00 GMT

Every discussion about AI code review ends on the same question. If the bot reviews the PR, can the bot merge it?

My instinct was no, and since we have not tried it yet, I spent a few weeks reading to find out whether that instinct was just habit. It turns out the teams with the most volume have already answered, and the answer is more interesting than yes or no. They do let agents merge, under rules the agent had no part in writing.

Trust belongs to the task

Jina Yoon at PostHog wrote the clearest framing I have read (When to trust an agent). Trusting an agent more because the model got better is like skipping the seatbelt because you bought a nicer car. What matters is the task, and two questions are enough to classify it.

Is the work easy to verify with deterministic checks?

Is a mistake cheap to undo?

| Verification | Cost of undo | Level | |---|---|---| | Hard | High | 0. The agent assists, a human does the work. | | Hard | Low | 1. Human in the loop, the work stays a draft until someone validates it. | | Easy | High | 2. Delegated to the agent, with one last net before merge. | | Easy | Low | 3. Full autonomy. |

Her example is worth keeping. PostHog rewrote their SQL parser in Rust and barely read it line by line, because a mechanical oracle existed to check the output. But the parser sits under every query on the platform, so the rollout went through a shadow mode in production and a gradual switch. Easy to verify, expensive to undo, so level 2 with a net.

Auto-merge is level 3 only. Easy to verify and cheap to undo, which in practice means behind a feature flag with a rollback that takes minutes.

Keep the classifier away from the reviewer

The obvious shortcut is to ask the reviewing agent whether the PR is safe to merge. Resist it.

First because an LLM judging its own category is exactly the setup that fails. JudgeBiasBench, a preprint from this spring (arXiv 2604.23178), measured LLM-as-judge at over 50 percent error on bias detection, with style bias dominating. One number from one preprint, but it matches what anyone who has watched a model grade its own homework already suspects. A lone LLM judge makes a poor production gate.

Second because the useful classification is mostly deterministic. Which paths did the diff touch? Does it change a migration, a payment flow, a permission check, a CI workflow file? Does it add code behind a flag or change existing behaviour? Those are rules. A path-based labeler answers them before any model runs.

Anthropic's Claude Code team, who now land more than 30,000 changes a day in their monorepo, sum it up in one rule (How the Claude Code team does code review): the change itself sets the risk tier, whoever wrote it.

In practice, a classifier picks the tier from the diff. A second classifier checks whether a person is needed anyway. Some paths always need one: areas listed as sensitive by their owners, and any CI workflow file.

| Risk | Examples | Human approvals | |---|---|---| | Low | Docs, copy, internal-only features, additive changes behind a flag | None, once automated and security checks are green | | Medium | Ordinary product changes, plus sensitive areas like caching, the auto-updater, system prompts | One | | High | The riskiest code, and paths that always need two people | Two |

Two more rules from the same post, which I think are the whole thing in miniature.

An agent's approval never counts as a person's.

An approval covers one specific commit.

A new push after approval is a new PR as far as the system is concerned. Florian Bruniaux made the same point in his talk at Dev With AI in Paris this month, "Du code au merge": "A validated doesn't mean B validated." Tests and reviews attach to a commit. The conversation around the PR carries nothing.

A ladder with three rungs

Put together, the teams I have read converge on a ladder. You climb it one rung at a time, and most teams should stop at the second.

The agent proposes, the author confirms. Deterministic labelling by path, then an agent that proposes a label and a one-line reason. The author of the PR confirms or overrides. Bruniaux runs this with two GitHub labels, Review AI and Review Human. Review AI is for a bounded, known change that tests can verify, inside a perimeter the team has authorised. Review Human is for access rights, private data, calculations, payments, or anything whose consequences are not well understood. When in doubt, ask for a human. The labels are a signpost for people and a filter for the bots.

Auto-approve, human merges. The agent review counts as an approval on low-risk PRs. A person still presses the button, which keeps ownership attached to a name. PostHog does a version of this with StampHog, a bot that approves low-risk PRs. The NextLevel case study collection on agentic code review (source) covers eight organisations, and in every one of them a human still merges.

Auto-merge on a listed class. Docs, copy, dependency bumps with green CI, additive code behind a flag. The class is written down somewhere, and nothing outside the list qualifies. Only trusted authors, in practice the owning team, get the lighter path. And the guardrail is measured: if reverts or incidents go up, the lighter path closes.

Anthropic's post ends on something I find reassuring. They do not yet have the circuit breaker that tightens approval automatically when revert numbers rise, and their engineers want it before more code moves to zero-human merges. Even the team furthest along is not comfortable without it.

What scales down

We are nowhere near 30,000 changes a day, and we have not climbed any of this ladder yet. What draws me to it is that the first rung costs almost nothing. A path-based labeler, two labels, and a short written perimeter that says which kinds of changes the team is willing to let a bot read alone.

The tooling is the easy half. Writing that perimeter down and agreeing on it takes a team conversation, and the next post is about what the bot does once it is inside the perimeter and finds something.

The bottleneck moved

Tue, 20 Oct 2026 00:00:00 GMT

I manage a squad in a SaaS company. We adopted coding agents this year and we are at the very beginning of rethinking how we work around them. From where I sit, one thing already bothers me more than the quality of the generated code. The queue.

Pull requests arrive faster than anyone can read them. One of my engineers reviews most of the front-end work and the pile in front of him keeps growing. On a recent project we duplicated an existing integration pattern across several new data sources in a few days, because that is exactly the kind of thing an agent does well. Then we spent far longer testing it than writing it.

That is about all I have seen with my own eyes so far. Two symptoms. The rest of this post is what I found when I went looking for whether other teams had the same two, and what they did about it.

None of this is specific to us. Make one step in a chain almost free and this is what you should expect.

When one step becomes free, the next one becomes the bottleneck

Software delivery is a chain. Specify, write, review, test, ship, observe. For twenty years the expensive step was writing. Teams, rituals and tooling were sized around that cost. Review was a fraction of the writing time, so one reviewer per author was enough. Manual QA could keep up with the trickle of features.

Now writing is wide, and review and testing kept their old width. The bottleneck is still there. It moved one step to the right.

The numbers are consistent across sources that do not share an agenda. The 2025 DORA report describes throughput and instability rising together, with downstream bottlenecks exposed in testing and review. Its one-line thesis is that AI amplifies what a team already is. Faros AI, which sells telemetry and therefore has a stake in the story, measured 22,000 developers and found median time in review up 441 percent, PRs merged without review up 31 percent, and incidents per PR up 243 percent between low and high adoption periods (source). Take the exact figures with care. The direction is not in doubt.

Anthropic's own Claude Code team went from about 2,500 changes a day in their monorepo in May to more than 30,000 in September (source). Nobody reads 30,000 changes a day.

Balint Orosz put it in a way I keep coming back to. Writing the code was always 20 percent of the job and testing it was 80 percent. The mistake is trying to compress the 80.

Three words from Toyota

When I try to explain the situation to people outside engineering, the word "bottleneck" falls short. It mixes three different problems. The Toyota Production System has a word for each.

Mura is unevenness. The flow is out of balance. Development outruns review and QA. This is the root cause, and the only one worth fixing directly.

Muda is waste. Work that waits. Features sitting in a column called "ready for test" for a week. Branches that need a rebase because they aged. Context that the author has already lost by the time a comment arrives.

Muri is overburden. The person at the narrow step is drowning. They review faster and worse, or they review well and fall behind on everything else.

The classic trap is to treat muda by manufacturing muri. The features are waiting, so we block afternoons for testing, we ask the reviewer to go faster, we add a second reviewer who is already busy. The queue shrinks for a week and the people burn.

The other trap is quieter. When review becomes too heavy, two behaviours appear. Some people approve as soon as the automated review has nothing to say. Others keep reading seriously and get buried under everyone else's PRs. You lose on both sides, quality on one and people on the other.

Three conclusions I refuse to draw

That we should write less code with agents. The 20 percent got cheap, and that is good news.

That a product will fix it on its own. The teams with the most data on this, Uber, Cloudflare, Anthropic, PostHog, have all ended up building their own review and verification systems after trying off-the-shelf reviewers. Monzo switched on GitHub Copilot's automated review and then switched it off by default, because the suggestions were not actionable. What counts is how many comments change what the author does, and most of those tools produce volume instead.

That the bar has to come down. The question I want the team to work on is how we keep pace at this step without lowering the bar. Getting people to review more is a different question, and it leads to different experiments.

Where this goes next

Several directions at once, and one post will never cover them. Once writing is cheap, at least six other things wobble.

The harness around the agent, meaning the instructions, rules and context the team shares, because an agent without a shared harness improvises with whatever pattern is most common on the internet, which is rarely the one you chose.

Who gets to merge what, and whether an agent's approval should ever count.

How you verify a bot's suggestion when the bot is right about the problem and wrong about the fix.

What happens to prototypes when anyone can build one overnight, to team topology when a backend engineer ships their own front-end, to planning when changing your mind costs nothing, and to the quiet learning that used to happen when a junior asked a senior a question.

I will take them one at a time. The next two posts are about the two I expect to hurt first: deciding who merges, and trusting a reviewer that has no name.

If you have been through this and found a lever that worked, I would like to hear about it. Most of what I read comes from companies ten times our size, and I am trying to work out which parts scale down.