Method
How these numbers are produced
A detector that cannot be scored can only be demoed. Everything below exists to make the claim on the front page falsifiable — by someone who did not write it.
Ground truth
A case pairs a repository snapshot with a real dependency bump, and records by hand which call sites the bump actually breaks. Across 9 cases that is 21 sites labeled breaking and 96 labeled explicitly not broken — the near misses, which are what a precision number is really measuring.
Labels were written by reading each tree at its pinned commit: model classes traced to
their base, every .dict() classified as model method or
builtin, every render as
ReactDOM.render or a class method. Two tests re-verify on
every run that each labeled file still exists in the checkout and that the labeled
symbol appears in it, so a label that drifts fails the suite rather than quietly
skewing a number.
Three rules that decide what a score means
The unit is (file, symbol)
File-level scoring hides the failure that matters: a detector that flags the right
file for the wrong reason looks correct and is useless. The qualifier is dropped —
an aliased import RD from 'react-dom' makes the same
API RD.render at one site and
ReactDOM.render at another, so all spellings key to the
member name.
Unlabeled reports are unjudged, not wrong
A real repo has more call sites than anyone will hand-label. Counting every unlabeled hit as a false positive would punish a detector for finding something the labeler missed. They are tracked separately and published alongside — the baseline has 12, the agent 0.
Negative cases are first-class
2 of 9 cases are repos that use the package and break nowhere; the correct output is silence. This is the metric a user actually feels — a tool that cries wolf gets muted no matter how good its recall is. Baseline: 0.0% clean. Agent: 100.0%.
Six ways for text matching to be wrong
The 5 real cases exist because hand-built fixtures isolate one trap each, while real repositories stack them. Each trap below is a different failure mode.
- A same-named method on an unrelated class a component's render(), a plain class's dict()
- A same-named function from a different library supertest's .send(), requests' response.json(), stdlib json.load
- The import specifier itself from 'next/headers' is word-bounded, so an identifier search matches the path
- A non-removed overload res.send('OK') is fine; only res.send(404) breaks
- The recommended replacement repos already using req.body get flagged for the API the notes say to migrate to
- Prose comments, JSDoc @param, and a "bans.json" string literal
The sharpest example is
real-realworld-react-19: one genuine ReactDOM.render
call against eleven class-component render() methods —
8.3% precision for the baseline on that repo alone.
Fixtures flatter a text matcher
4 cases are small repos written for this project; 5 are third-party code at a pinned SHA. Reporting one blended number would let a good score on hand-built cases mask a bad one on real code, so the split is always published.
| Source | grep-baseline | agent · deepseek-v4-flash-free |
|---|---|---|
| Fixtures (4 cases) | 30.0% | 100.0% |
| Real repos (5 cases) | 15.5% | 100.0% |
Precision only. Baseline precision roughly halves on real code; the agent holds — which is the result that would have been invisible on fixtures alone.
One run is a demo, not a measurement
Three full sweeps of the same 9 cases on the same model:
| Sweep | Precision | Recall | Note |
|---|---|---|---|
| 1 | 100% | 100% | — |
| 2 | 100% | 95.2% | real-tf2bot-pydantic-v2 scored 1/2 |
| 3 | 100% | 100% | after the label fix |
Precision was 100% in every sweep: across 96 opportunities to report a site the labels
call unaffected, the agent took none. Recall moved once — and chasing it found a bug in
the ground truth, not the model. The release notes for one case claimed
BaseConfig was removed in pydantic v2. It was not; it is
deprecated. The agent had been wavering on genuinely ambiguous ground, and its "wrong"
answer was a defensible reading of a bad spec.
Three things changed: the notes were corrected with the verification recorded in the case file, the ambiguity was resolved in the system prompt rather than left to the model, and a guard test now fails the suite if a case's notes claim an API "is removed" near one that pydantic merely deprecates.
Reproduce it
Nothing on this site is hand-entered. Every number is read from
evals/results.json, which the harness writes on each run.
uv sync --extra dev uv run pytest uv run python -m evals.run --baseline --agent # writes RESULTS.md + results.json cd web && npm install && npm run dev # this site, on that artifact
Both detector flags belong in one invocation: running a single detector rewrites
RESULTS.md with only that one. The JSON artifact merges
instead, so a baseline-only run keeps the agent's numbers.
What this build was made from
grep-baseline measured 2026-08-26
Site-level detail captured directly from this run's findings.
agent derived 2026-07-31 · oc/deepseek-v4-flash-free
Reconstructed from the per-case scores in the published RESULTS.md (run of 2026-07-31). That run recorded fp=0, fn=0 and no unjudged sites on every case, which fixes the reported set as exactly the labeled-affected sites — so the verdicts below are exact. What that run did not persist is the agent's own prose: no rationale or patch text is available for these sites. Re-run `--agent` to replace this block with measured detail.
Artifact generated 2026-08-26 · schema v3