Change Verdicts

A change verdict answers one question about one agent: after we changed it, did it get better, get worse, or stay about the same? Morse compares the runs before the change with the runs after it, one lane at a time, and gives each lane an honest answer. When the data cannot answer yet, the answer is “too early”, with what is already ruled out and what would settle it.

A verdict compares before and after. It does not show that the change was the cause. See What a verdict does not say.

The four states

Every lane of every tracked change is in exactly one state.

“Unchanged” is a result, not a shrug: the runs are enough to say any movement is smaller than the margin. “Inconclusive” is the honest answer when they are not.

Lanes

Each change gets a separate verdict for each of five lanes.

Lanes are never combined into an overall score or a single “better or worse” state. A change can make cost fall and error rate rise, and Morse shows both. The headline simply lists the moved lanes and counts the rest, for example “cost per run fell 12% · 3 unchanged · 1 too early”.

Latency lanes show the numbers you expect (“p95 now about 8,350 ms, was 8,420 ms”). The decision behind them compares the share of runs slower than the old percentile, which is what keeps the result valid when the traffic is small.

The success-rate lane uses confirmed outcome labels. When labels are missing, the agent’s History can ask you to mark runs. This prompt appears at most once per agent in 30 days across History and the weekly email. Not now hides it for 30 days; the lane’s evidence and counts remain visible. An agent’s History shows each change’s lane verdicts, under the finding it is linked to or in Unexplained changes, with links to before/after traces.

Why Morse waits

Morse recomputes verdicts every hour while a change is being watched. Checking a result again and again, and stopping when it looks good, is how dashboards produce false wins. Morse’s intervals are anytime-valid confidence sequences, so looking every hour does not inflate the false-positive rate.

• Each lane uses an alpha of 0.01. With five lanes, the chance of any false verdict on one change is at most 5%.
• A timer never decides. Reaching the 28-day horizon closes the change with every lane in the state it already had; an inconclusive lane stays inconclusive.

Floors

No lane gives a verdict before it has enough comparable data.

• 100 runs on each side (before and after).
• 400 runs before the change for the slow-end response time (p95) lane.
• 72 hours since the change.
• For success rate: at least 50 labeled runs old enough for their labels to have settled on each side, and at least 20% of runs labeled.

Before the floors, a lane reads “collecting”, for example “collecting: 37/100 runs on the new version, earliest call about Thursday”. The date comes from the recent run rate.

What “too early” rules out

An inconclusive lane leads with what the data already rules out, once the interval is informative. For example: “cost per run did not rise more than 12%”. That is a bound, not a verdict. It tells you how bad the change can be, even while it cannot tell you whether it helped.

It then says what would settle the lane: roughly how many more runs, how many days that is at the current rate, or what labeling is missing. If settling would take longer than the usual gap between changes to that agent, Morse says the lane is unlikely to settle before your next change and suggests batching.

What the data can detect

How large a change has to be before Morse can call it depends on how many runs you have on each side. These are the smallest changes the lanes can detect at alpha 0.01 with the usual traffic shape:

Small agents will mostly get bounds, not verdicts. With 150 runs per side, only a very large cost change reaches “improved” or “worsened”; anything smaller reads “too early” with a bound such as “did not rise more than 18%”. A before-and-after chart can show a drop of 30% on the same data. Morse holds back because, at that size, a swing that large is also what noise looks like.

Where changes come from

Morse finds a change when the dominant configuration of an agent’s runs changes: a new agent version, a new git revision, or a different model, system prompt or tool schema.

• A change you declare with a new agent_version is tracked automatically, right away.
• A change Morse only detected (a new revision or fingerprint) waits for you to confirm it, unless your org has turned on automatic tracking.
• You can dismiss a detected change, or add one by hand with a time and a note, for example when you change a setting Morse cannot see.

Confirm, or track automatically

Confirming a change vouches for it: the verdict then carries your name and date, and a worsened lane can enter the weekly attention list and email. Automatically tracked changes are computed the same way but are labeled unconfirmed, and an unconfirmed worsened lane stays out of the attention list and the email until someone confirms it.

Org owners and admins can turn on tracking of every detected change in the org’s change settings. The setting is off by default.

Dismiss a change without changing recommendation feedback

Dismissing a detected change leaves feedback on its linked recommendation alone. If the change has a linked recommendation, Dismiss opens a confirmation with an unchecked checkbox: “Also mark the recommendation that prompted this as not useful”. Leave it unchecked to dismiss only the change, then select Dismiss change.

Checking the box opts in to marking the linked recommendation not_useful. Existing accepted or not_useful feedback is preserved even when you opt in; Morse creates feedback if none exists or updates an existing dismissed entry to not_useful.

Label prompts: at most every 30 days per agent

When a tracked change’s success-rate lane is waiting on labels, Morse can include a prompt to mark runs in the weekly email and performance brief. The prompt is capped per agent over a rolling 30-day window, rather than by calendar month: queuing a weekly email containing the prompt starts the window. Opening the performance brief does not start it.

In an agent’s History, Not now hides that agent’s label prompt for 30 days. During the window, History hides the labeling action and Not now control, while keeping the evidence gap visible. The weekly email and performance brief use the same per-agent throttle.

Send the version and revision

Detection needs the runs to say what ran. The two cheapest signals:

• Git revision. Set MORSE_GIT_REVISION in your container or deploy environment. It needs no code change.
• Agent version. Pass agent_version when you initialize the SDK, or per run. This is what makes a change declared rather than only detected.

morse_ai.init(agent_version="2026.10.3")

See Record keys and versions for every route, including OTLP and the REST payload. If an agent sends none of these, there is nothing to detect, and Morse says so instead of showing an empty list.

Comparable traffic

A verdict only compares like with like: the same agent and environment, and a similar traffic mix before and after. If the mix differs too much (more than 0.20 total variation across run size and time of day), the lane reads inconclusive with “traffic mix differs” and names the largest shift.

Other changes that overlap the window, an upstream agent change, a price-catalog change or a volume shift are listed as confounders next to the verdict. Morse names them and never adjusts them away.

Canary, rollback and bundles

• Canary. If both the old and the new version each serve at least 10% of runs for several hours, Morse compares them side by side over the same period instead of before against after.
• Rolled back. If the old version returns to at least 90% of runs for at least 6 hours after the new one passed 50%, the change is marked rolled back and its lanes freeze at that point. A later redeploy is a new change with its own window.
• Bundles. If a newer change replaces one before it reaches the floors, the earlier one reads “superseded before the floors” and folds into a bundle. The bundle compares the last stable version to the current one, for example “since v40, 3 changes: cost fell 14%”. It is labeled as covering several changes and uses the same four states and the same rules.

This is how a team that ships several times a week still gets a verdict: one honest comparison across the stretch, not three that each ended too early.

What a verdict does not say

A verdict compares the runs before and after a change. It does not show that the change was the cause. Prompt edits, model updates upstream, traffic shifts, a new customer or a busy week can all move a lane at the same time. Morse lists the confounders it can see and says nothing about the rest.

Treat a verdict as evidence for a decision you still make: keep, revert, or look closer in the traces. Every verdict links to the before and after runs it was computed from.

Related

• Follow-ups and decisions — record what your team decided to do about a change, and see its result later.
• Record keys and versions — send the version and git revision that make changes detectable.
• Agents — the per-agent pages that changes belong to.
• Cost Intelligence — the spend behind the cost lane.
• Efficiency — what to look at when a cost lane moves.