The failure mode is not a wrong comment. It is the two-hundredth comment — after which nobody reads the right one either.
Ask an engineering team that turned off their AI reviewer why they did it, and almost nobody says it was wrong. They say it was tiring. Somewhere between the fifth suggestion to extract a helper and the twelfth reminder to add a test, reading the bot stopped being work that paid for itself, and the whole thing became something to scroll past on the way to the human review.
That is a different problem from accuracy, and it does not get solved by the thing everyone tries first.
In February 2026, Martian published Code Review Bench, the first large independent benchmark for these tools: seventeen reviewers measured across roughly 300,000 real pull requests. It reports three numbers — precision (what share of the reviewer's comments matched a change the developer actually made), recall (what share of the developer's post-review fixes the reviewer had caught), and F1 as the balance between them.
The top of that leaderboard is more sobering than the marketing around it. As of the 30 July 2026 snapshot, Greptile leads with an F1 of 60.8%, the best precision of any tool measured at 76.2%, and recall of 50.6%.
Sit with those two numbers for a second. Best-in-class precision means roughly one comment in four still did not correspond to anything the developer changed. And recall near 50% means the reviewer missed about half of what the developer went on to fix anyway. Further down the same benchmark some tools land near 47% precision — more than half their comments are noise.
This is the state of the art, not a cautionary tale about a bad tool.
Something worth noticing if you are comparing tools: CodeRabbit published that it topped the benchmark for the January–February window. cubic published that it was number one. Greptile published the same for the July snapshot. All three are telling the truth about a different date.
The leaderboard moves, everyone screenshots the week they won, and none of it tells you how a tool behaves on your repository. Treat vendor benchmark posts — including any we ever write — as a claim about one snapshot, not about your codebase.
Precision is a per-comment property. Muting is a per-inbox one, and it depends on volume as much as accuracy.
CodeAnt's analysis of teams running these tools at scale describes teams receiving 200–400 AI comments a week and dismissing 70–90% of them. Industry work on alert fatigue puts the share of alerts ignored at up to 40% once fatigue sets in — and the ignoring is not selective. Nobody carefully evaluates each comment and rejects the bad ones. They stop reading the column.
Which means a tool at 76% precision and high volume can be less useful than a quieter tool at the same precision, because the first one gets muted in week three and the second one is still being read in month six. The metric that matters is not comments produced. It is comments acted on, divided by comments read.
The standard remedy is a severity threshold: show me only the important ones. It underperforms, and the reason is structural.
The severity label was assigned by the same model that wrote the finding. Filtering on it does not add information — it reweights the model's own confidence, which is exactly the thing that was miscalibrated to begin with. You drop real bugs it happened to rank low and keep noise it happened to rank high. The volume falls, the ratio does not improve much, and now you cannot see what you lost.
You are turning down the volume on a signal, not improving the signal.
Underneath this is a conflict of interest that no amount of prompt engineering removes.
A single reviewer performs two jobs that pull in opposite directions. Finding issues rewards suspicion: flag the ambiguous case, because a miss is invisible and a false alarm is cheap. Deciding what to show rewards restraint: stay quiet unless you are sure, because a false alarm costs attention. One model doing both, in one pass, has to be suspicious and restrained simultaneously — and it has already committed to the finding by the time it judges it. Models are poor critics of text they just produced.
That is the mechanism behind the numbers. It is not that the models are not smart enough. It is that the author is grading their own paper.
The interesting work in this space is mostly about separating those two jobs.
More context. Reviewers that read the call sites, the type definitions and the covering tests refute more of their own bad findings, because a lot of noise is a correct pattern applied to a place where the guard already exists three lines up. This is why repository-aware tools score better on precision.
A separate filtering pass. cubic published a detailed writeup of moving from one agent to several, including a dedicated filtering agent whose only job is deciding what survives. Their reported outcome — a large drop in false positives and roughly half the comments per pull request — is the clearest public evidence that the split is the right shape of fix.
Independent adjudication. The step past a filter is a judge that did not write the finding, and ideally more than one, drawn from different model vendors so their mistakes are less correlated. Research on multi-agent debate has consistently found that independent critics catch errors a single self-review does not. This is the direction we are working in, and it is fair to say it is not yet settled which way wins.
Vendor benchmarks will not answer the question you actually have. This will, and it takes two weeks:
If the ratio holds above roughly one in three, your reviewer is earning its place. If it is heading toward one in ten, you are paying for something your team has already stopped reading — and swapping vendors will buy you a few weeks before the same curve repeats.
We are building Dicast around the adjudication idea: a wide, cheap scan proposes findings, and three jurors from three different model vendors — none of which wrote the finding, each asked to refute rather than confirm it — vote on whether it reaches your pull request. Two votes to post, three to block a merge, and everything struck stays on the record with the reason.
We are not going to tell you it beats the tools above, because we have not run that comparison and you should not believe anyone who claims a number they cannot show you on your own code. What we will say is which problem we think matters: not catching more, but being worth reading in month six.
We would genuinely like to hear what you use today and what specifically makes you skim past it. Write to hello@dicast.dev — that answer shapes what we build more than any benchmark does.