Quality

Quality verdict, risk drivers, and the one next action, in one screen. This is where you check whether your agents are behaving before you go looking for why.

Read the quality verdict (0:40, narrated)

What you see

The page is read-only: every card here links out to where you’d act.

Verdict strip

A single strip states the quality verdict — healthy, at risk, or blocked — with the weakest-signal scorer and its pass rate named inline.

Two numbers sit to the right so you never have to click through to judge confidence: total scored traces, and the smallest scorer sample size (N) behind the verdict.

Ask Morse in the page header starts with Ask about this page: a question about this page — how many runs were checked, and what the sampled checks do and don’t tell you. See Ask Morse.

Advisory scorers (for example retrieval relevance) are shown but not counted in the verdict; the strip names them, so a red advisory driver below a healthy verdict is never a contradiction. Before any traces are scored, the strip says so plainly instead of guessing.

Beside the verdict, a basis line says what it rests on, for example: “Scoring coverage: 120 of 400 runs were scored by a check (30%). This is sampled answer checks, not whether customer jobs were completed.” Quality is healthy means the sampled checks pass; it is not a business outcome.

Maturity-stage banner

A dashed-border banner appears only while your account is short of full maturity, and shows exactly one next action for your current stage — never a checklist of everything at once.

The stages run in order: no traces → traces with no scores → scores with no curated cases → cases with no replay runs → runs with no comparisons → policies with thin coverage → mature.

Quality Drivers

Quality Drivers ranks scorers by risk — most failures first, ties broken by lowest pass rate — so the worst offender is always the top row. Advisory scorers carry an Advisory · not in verdict tag. Each row opens its failed traces, or its warned traces when it has warnings but no failures.

Under each scorer, a coverage chain shows how many runs made it into its pass rate: “Of 400 runs in the window, 120 were sampled, 116 completed and 110 counted in the pass rate.” Completed leaves out errored and skipped results; counted also leaves out n/a.

Below the list, one line gives the average pass rate and its denominator: the average is taken over the scorers that have results, and the line says how many of your configured scorers that is. Scorers with no results are listed with the reason (no scored traces yet, or turned off in Score Settings).

Retrieval quality

For agents that retrieve documents, a Retrieval quality card shows a sampled quality signal. On a sample of traces, an LLM judge reads each retriever span’s query and returned chunks (up to 10, 2 KB each) and labels every chunk relevant or not, plus whether the chunks together are enough to answer the query. Morse computes the numbers from those labels:

Precision@k — the share of returned chunks that were relevant.
MRR — 1 divided by the rank of the first relevant chunk, so a relevant chunk at the top scores 1.0 and none scores 0.
Context sufficient — the share of judged spans where the chunks were enough to answer.

The card always states its coverage, for example “12 of 40 retriever spans judged”, and how many spans returned no documents. A retriever span with no documents is counted but never judged. The signal is advisory: it appears beside the quality verdict and does not change it, and it is never combined with cost. Open a judged retriever span in a trace to see each chunk’s rank, score and relevance. Recall is not measured, because that needs a known-correct answer set.

The card needs SDK output that includes documents — see Retriever documents.

Category accuracy

For a classifier agent with a category taxonomy, a Category accuracy panel shows how well its picks match your experts’ labels. Pick the Agent at the top right. Macro F1 treats each category the same, and Micro F1 treats each prediction the same. The table lists Precision, Recall, F1 and Support (how many labeled cases the category has) per category.

A category with fewer than 5 labeled cases reads Too few to judge and gets no score. If some traces’ predictions couldn’t be read from the metadata key or JSON pointer, a line such as “2 traces skipped: predictions couldn’t be read” says so, and they are left out of the numbers. The figures come from random-case labels and are computed from those labels, with no LLM involved. Until then the panel reads Label 30 random cases to see category accuracy, with a link to start labeling.

Next-best actions

Regression Readiness says whether you have two completed runs to compare and, when you do, the smallest pass-rate drop a comparison could catch at your case count — for example, “With 8 cases per run, a comparison can only flag a drop of about 70 points or more.” Add cases to detect smaller drops. The replay timeline lives on Regression Lab.

An Auto-Queue Review preview shows the first few system-sampled cases waiting for review, with a link to the full inbox on Cases.

Outcomes and judge budget

An Outcomes card shows the production success rate from labeled runs — beside the sampled quality verdict, never fused into it. A Judge Budget widget shows this month’s LLM-judge spend against your limit. Scorer configuration lives on Score Settings, and the CI gate on its own tab.

How to use it

Read the verdict strip first — it tells you healthy/at-risk/blocked and how much data backs that call.
Follow the maturity banner if it’s showing; it’s the single most useful next step for an account still ramping up.
Use Quality Drivers to find which scorer is failing the most, then jump into Regression or Cases from the linked action.
Before relying on the CI Gate, remember it re-scores recorded outputs, not fresh agent runs.

Before you have data

With no scored traces yet, the verdict strip reads “Not enough data yet” and explains that the verdict appears automatically once evals run against production traffic — no fabricated status.

Below the fold, an empty state reads “No eval scores yet — Scores will appear here automatically once traces are ingested.”

Cases — curate the datasets that feed every scorer and CI policy.
Regression — replay runs and compare two of them statistically.
CI Gate — turn a policy into a pass/fail check on recorded-output scores.
Scoring — configure which scorers run and how often.