9 hand-labeled cases · 21 labeled breaking call sites
A dependency just shipped a new version.
Does it break your repo?
The changelog already says what changed upstream. The useful question is whether any of it reaches your code — which means reading the release notes, then reading the repository, and deciding which grep hits are real call sites and which are comments, string literals, or same-named methods on unrelated objects.
Most agent demos cannot tell you how often they are right. This one can, because the question has ground truth.
Both detectors find all 21 breaking call sites — recall is tied. The entire difference is what else they say. The baseline reports 96 sites the labels call not-broken; the agent reports 0.
The baseline is deliberately strong
It extracts every identifier named in the release notes and reports every occurrence in the repo — roughly what a careful person does by hand in five minutes. That gives it perfect recall, which is exactly what makes it a fair opponent: the agent cannot win by finding more, only by being wrong less.
| Detector | Scored | Precision | Recall | F1 | Clean on negatives | Cost/scan | Median |
|---|---|---|---|---|---|---|---|
grep-baseline | 9/9 | 17.9% | 100.0% | 30.4% | 0.0% | $0.00 | <0.1s |
agent oc/deepseek-v4-flash-free derived detail | 9/9 | 100.0% | 100.0% | 100.0% | 100.0% | $0.00 | 33.9s |
Precision is computed over labeled sites only. Sites a detector reports that the labels do not mention are counted separately as unjudged, so no detector is punished for finding something the labeler did not think to write down. How scoring works →
Fixtures versus real repositories
Four cases are small repos written for this project; five are third-party repos at a pinned SHA. A detector that scores well only on fixtures has been tuned to its own test data, so the split is always published.
| Detector | Source | Precision | Recall |
|---|---|---|---|
grep-baseline | fixtures | 30.0% | 100.0% |
grep-baseline | real repos | 15.5% | 100.0% |
agent | fixtures | 100.0% | 100.0% |
agent | real repos | 100.0% | 100.0% |
Every case, scored
Open any case to see the release notes it was given, the hand-written labels, and every site each detector reported — with the source line that produced it.
| Case | Bump | grep-baseline | agent · deepseek-v4-flash-free | |
|---|---|---|---|---|
| express-5-clean fixture · negative | express 4.18.2 → 5.0.0 | 0 / 8 | 0 / 0 | open → |
| next-15-async-headers fixture | next 14.2.3 → 15.0.0 | 2 / 2 | 2 / 0 | open → |
| pydantic-v2-migration fixture | pydantic 1.10.13 → 2.0.0 | 2 / 2 | 2 / 0 | open → |
| react-19-render fixture | react-dom 18.2.0 → 19.0.0 | 2 / 2 | 2 / 0 | open → |
| real-express-4-clean real repo · negative | express 4.16.3 → 5.0.0 | 0 / 39 | 0 / 0 | open → |
| real-fastapi-realworld-pydantic-v2 real repo | pydantic 1.9.0 → 2.0.0 | 8 / 8 | 8 / 0 | open → |
| real-nextjs-15-async-apis real repo | next 14.2.3 → 15.0.0 | 4 / 8 | 4 / 0 | open → |
| real-realworld-react-19 real repo | react-dom 16.3.0 → 19.0.0 | 1 / 11 | 1 / 0 | open → |
| real-tf2bot-pydantic-v2 real repo | pydantic 1.10.7 → 2.0.0 | 2 / 16 | 2 / 0 | open → |
right / wrong — true positives over false positives.
Provenance
grep-baseline measured Run 2026-08-26
Site-level detail captured directly from this run's findings.
agent derived Run 2026-07-31 · oc/deepseek-v4-flash-free
Reconstructed from the per-case scores in the published RESULTS.md (run of 2026-07-31). That run recorded fp=0, fn=0 and no unjudged sites on every case, which fixes the reported set as exactly the labeled-affected sites — so the verdicts below are exact. What that run did not persist is the agent's own prose: no rationale or patch text is available for these sites. Re-run `--agent` to replace this block with measured detail.