9 cases
The evidence
Each case pairs a repository snapshot with a real dependency bump, and records by hand which call sites the bump actually breaks. That hand-labeling is what makes precision and recall mean anything — without it a detector can only be demoed.
- express-5-clean
express 4.18.2 → 5.0.0
A repo that depends on Express but only uses APIs that survive the 4 to 5 bump. The correct output is zero findings.
- grep-baseline
- 0 right · 8 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 0 right · 0 wrong · 20.3s
- next-15-async-headers
next 14.2.3 → 15.0.0
Next 15 made the request-scoped helpers cookies() and headers() asynchronous, breaking every synchronous call site.
- grep-baseline
- 2 right · 2 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 2 right · 0 wrong · 19.8s
- pydantic-v2-migration
pydantic 1.10.13 → 2.0.0
Pydantic v2 renamed the model serialisation methods and replaced the validator decorator.
- grep-baseline
- 2 right · 2 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 2 right · 0 wrong · 39.6s
- react-19-render
react-dom 18.2.0 → 19.0.0
React 19 removed the legacy ReactDOM.render entry point in favour of createRoot from react-dom/client.
- grep-baseline
- 2 right · 2 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 2 right · 0 wrong · 31.2s
- real-express-4-clean
express 4.16.3 → 5.0.0
A real Express 4.16.3 REST API (~2.9k stars) upgraded to Express 5. It uses none of the removed APIs, so the correct output is zero findings.
- grep-baseline
- 0 right · 39 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 0 right · 0 wrong · 33.9s
- real-fastapi-realworld-pydantic-v2
pydantic 1.9.0 → 2.0.0
The RealWorld reference backend (FastAPI, ~3.1k stars, pinned to pydantic ^1.9) upgraded to v2. Two distinct traps: the dict() builtin, and response.json() from the test client.
- grep-baseline
- 8 right · 8 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 8 right · 0 wrong · 118.7s
- real-nextjs-15-async-apis
next 14.2.3 → 15.0.0
Vercel's own Next.js + Supabase + Stripe subscription starter (~7.7k stars), pinned at next 14.2.3, upgraded to Next 15's async request APIs.
- grep-baseline
- 4 right · 8 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 4 right · 0 wrong · 48.7s
- real-realworld-react-19
react-dom 16.3.0 → 19.0.0
The RealWorld reference frontend (React + Redux, ~5.6k stars) upgraded to React 19. One real ReactDOM.render call against eleven class-component render() methods.
- grep-baseline
- 1 right · 11 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 1 right · 0 wrong · 25.5s
- real-tf2bot-pydantic-v2
pydantic 1.10.7 → 2.0.0
A desktop chat-bot app pinned to pydantic==1.10.7. Two real v1 APIs in use against sixteen sites that only look like them.
- grep-baseline
- 2 right · 16 wrong · <0.1s
- agent · deepseek-v4-flash-free
- 2 right · 0 wrong · 89.2s