Agent Regression Flywheel
A closed loop that turns live agent failures into tomorrow's regression suite, and gates every version promotion behind a significance test instead of a raw pass-rate diff.
The question
Most eval harnesses run a fixed benchmark. This one asks what should be in the benchmark, based on what is actually breaking in production, and whether a fix is real or just sampling noise.
How it works
A LangGraph agent runs live and is traced in LangSmith. A failure-mining service embeds the traces and clusters them with HDBSCAN to surface new failure modes, rather than matching a predefined list of known bugs.
Each cluster becomes a Harbor-style task with a sandboxed environment spec and a verifier. Temporal fans regression trials out into isolated Modal sandboxes with retries and resumability.
A candidate version is compared to the baseline with paired trials, and needs a McNemar's-test-significant improvement before it is promoted. FastAPI serves the agent and dashboard APIs, Django runs the task-approval queue, and a Next.js dashboard shows the mining funnel and the significance result.
What testing caught
Two real bugs surfaced along the way: a numpy np.bool_ versus Python bool mismatch in the significance module, and a missing pair-count breakdown in the compare endpoint that the dashboard needed. Both are called out in commit messages instead of being silently fixed.
Stack
Python · FastAPI · Django · LangGraph · FastMCP · Temporal · Modal · Next.js · TypeScript · PostgreSQL · Redis · Kubernetes
Honest notes
One test is a documented skip: the live Temporal server integration, because the build environment had no route to download the ephemeral test server. The activity under test is covered directly by six passing tests.
The 10 Kubernetes manifests are structurally validated only; there was no live cluster to apply them to.