Regression
Regression lets you replay runs and compare two of them with real statistics, not an eyeballed score that moved. It merges the former Replay Runs and Run Comparison screens into one scrollable workspace.
What you see
Two sections, joined by a “Compare runs” scroll-to button: Replay Runs on top, Statistical Regression below.
Arriving at /quality/regression?focus=runs or ?focus=compare highlights and auto-scrolls to the
relevant section.
Replay runs
A table of past runs — target (case or dataset), status, cases-completed progress, cost, and a deploy badge when a run started within two hours of a detected deploy. A run started by the CI gate reads From CI, and one in Fresh outputs mode reads From CI · Fresh outputs: your agent produced new answers for it, so it is the one kind of run here that was not a replay.
New run opens a dialog to replay a whole dataset or a single case against the current scorer suite; results are isolated from the online scores you already have.
Clicking a row opens a detail drawer with per-scorer results; single-case runs get a second tab, Diff vs online, comparing the replay score to the live online score.
Replay re-scores a frozen input/output snapshot with the current scorer suite — it does not re-run your agent. Trajectory, tool-validity, faithfulness, and instruction-following scorers need new outputs from your agent; the trigger dialog points to a Fresh outputs CI run for that.
Statistical regression
Pick a baseline run and a candidate run from two dropdowns (only completed/partial runs are selectable) and click Compare.
The result leads with a conclusion in words, then why, then what to do — all written by Morse’s API. Its badge reads Better, Worse, No reliable difference, Inconclusive, or “The judge or scorer changed between runs — compare with caution”, with the number of common cases.
- When the runs share too few cases: We can’t make a useful comparison yet. The reason says what is missing — for example “The runs share 12 of the 20 test cases needed (8 more). The baseline ran 30 and the candidate ran 12. The candidate run is the short side.” — and the next step: “Run both on the same set of at least 20 test cases (add 8 more shared), then compare again.” Add more cases from recent traces → opens your test cases.
- When the grading changed: “The two runs were graded differently, so this comparison isn’t reliable.” with which part changed (judge model or scorer version) and “Re-run the baseline with the same judge model and scorer versions, then compare again.”
- Otherwise: “The candidate scored higher, and the difference is unlikely to be chance.”, “The candidate scored lower, …” or “No reliable difference between the two runs.”, each with how many scorers moved and a next step.
Under it, the scorer table shows each scorer’s Result in the same words — for example “Can’t compare yet: 12 paired cases of the 20 needed. Add 8 more.” The statistics — the difference (Δ), the p-value adjusted for testing many scorers at once (p (adj)) and the 95% confidence interval — are in a collapsed Statistical details section.
A readiness banner above this section states plainly when there aren’t yet two comparable runs, so you see that before opening the picker.
Binary pass/fail scores are compared with a two-proportion z-test; continuous 0–1 scores use a bootstrap 95% confidence interval (1,000 resamples) instead, since a bounded [0,1] distribution with mass at 1.0 violates the normality assumption a t-test needs.
Every scorer in one comparison is corrected together, so testing many scorers at once doesn’t inflate false positives: the false-alarm rate stays at 5%.
How we know. The correction is the Benjamini-Hochberg false-discovery method at α=0.05 (see multiple-testing correction in the glossary).
A scorer needs at least 20 paired cases (current default) before its comparison returns anything but inconclusive. Below that, Morse says it can’t make a useful comparison yet and names how many more cases you need.
Judge-model drift (different judge model between the two runs) and scorer-version drift are named in the conclusion and flagged on the affected scorer row, rather than being folded into the delta.
How to use it
- Trigger a replay against a dataset (or a single case) to establish a run you can later compare.
- Use the Diff-vs-online tab on a single-case run to see exactly how a replay score differs from what’s live.
- Compare runs of the same kind. A CI gate run in Fresh outputs mode scores new outputs from your agent, while a replay run re-scores recorded ones, so the page’s latest comparison pairs the newest run with the newest earlier run of the same mode, and never mixes the two.
- Once you have two completed/partial runs, pick them as baseline and candidate and compare — read the conclusion and each scorer’s result, not just the headline delta, and check whether the grading changed before trusting a “Worse” call.
- If the result reads “We can’t make a useful comparison yet”, add more cases from Cases rather than trusting a small-N delta.
Before you have data
With no replay runs yet, the Replay Runs table reads “No replay runs yet” with a Trigger your first replay button.
With fewer than two comparable runs, the comparison section shows a dashed banner explaining exactly how many more runs are needed, instead of an empty picker.