Cases

Found a bad response in production? Save that exact trace as a regression case in one click — no separate test-writing workflow. This screen is where those cases live, grouped into datasets you replay and gate on.

What you see

Two sections stack top to bottom: Datasets, then a flat table of every case.

Datasets

A card grid, one card per dataset, each showing its case count.

The system-managed Auto-Queue dataset carries a badge and a robot icon and can’t be deleted — it’s populated automatically by auto-sampling rules.

Expand a card to see version history (publish a snapshot to lock in the current case list as a new version, with added/removed counts against the prior version) or, for the Auto-Queue dataset, its review inbox instead.

Non-system datasets also show a coverage-gap panel: agent attribution, source-tag coverage, and reference-output coverage, each flagged when more than half the cases are thin on that dimension. Scorer- and failure-category coverage aren’t shown — the case model doesn’t carry per-case scorer results yet, and the page won’t fabricate that view.

Cases

A table of every case: name, input summary, an origin badge (From trace, Manual, or Imported), a reference-availability badge, and a link back to the source trace when one exists.

Arriving from a trace’s “Save as test case” flow with ?source_trace_id= filters the table to just that trace’s cases, with a Clear filter affordance.

Label cases as an expert

Open the Auto-Queue review inbox and switch to To label. This is where people on your team tell Morse what the right answer was, so you can see how often each judge agrees with them. See Judge reliability for how those labels turn into a figure.

Each item carries a source chip that says how the case was picked:

• Flagged: picked by the auto-sampling triggers.
• Random: drawn at random from the judge’s traces. These are the cases that count toward judge agreement.
• Representative: drawn from cases the judge passed.

Choose Label as expert to open one case per screen. You see the rubric and the trace evidence, and Pass, Fail or Unsure buttons. Press P, F or U to answer and N for the next case. Add a note of up to 500 characters if you want to explain the call, then choose Save label.

Blind labeling. The judge’s verdict and reasoning stay hidden until you save, so your label is not swayed by it. After you save, the same screen shows Judge verdict next to Your label.

Changing a label. Choose Change label, then Update label. The earlier label is kept as history, not overwritten, and the change shows up in the audit export.

Second labelers. When two or more members have labeled before, about 10% of random cases also go to a second member, so Morse can measure how well your experts agree with each other.

Disagreements. If two experts differ, or an expert differs from the judge, the case appears under Disagreements. Open it and choose Save resolution. The resolution becomes the label used for agreement, and the case leaves the list.

Thumbs up or down on a single score is a separate, quicker signal. It is never counted as an expert label and never feeds agreement. Any workspace member can label, and every label records who made it. Viewers can’t label. In the demo workspace every labeling control is read-only (Read-only in demo.).

Reference verdicts and audit export

Promote a label to a reference verdict. Promoting an expert label copies it onto the test case as that rubric’s reference verdict, with a pointer back to the label. If the trace has no case yet, Promote creates one. The reference is a copy: editing the label afterwards doesn’t change it unless you promote again. Promotion is available through the API (POST /api/v1/dashboard/evals/labels/{id}/promote).

Audit export. Organization admins can download every label as JSON or CSV with GET /api/v1/dashboard/evals/label-audit-export. Each row has the labeler’s email, when the label was made, the source, the judge’s verdict and judge version at the time, and which label replaced it if it was changed. Replaced and resolution labels are included. Other members get a 403.

How to use it

• Open a trace and click Save as test case to snapshot its input into a reusable case — a reference/expected output is optional, and most scorers run without one.
• Create a new dataset by name, then expand any card to publish a version snapshot once its case list is ready to lock in for CI gating.
• Watch the coverage-gap panel on a dataset you’re building for CI — a thin dimension there is a blind spot a gate built on that dataset won’t catch.
• Review the Auto-Queue dataset periodically; it fills automatically from auto-sampling rules and has its own daily cap.

Before you have data

With no cases yet, the table reads “No regression cases yet” and points you at the trace-level flow: “Open any trace and click ‘Save as test case’ to turn production behavior into regression coverage.”

With no datasets yet, the grid reads “No datasets yet. Create one above to group cases for a future eval run.”

Related

• Regression — replay a dataset and compare two runs statistically.
• CI Gate — bind a dataset to pass/fail thresholds that gate a merge on recorded-output scores.
• Scoring — the auto-sampling rules that fill the Auto-Queue dataset, and judge reliability from your expert labels.