Skip to Content
Getting StartedCommon tasksCan I trust Morse's labels?

Can I trust Morse’s labels?

Check two things. First, where each run’s label came from: labels you reported, a webhook or a rule are different from labels Morse inferred. Second, for inferred labels, measure how often Morse’s judge agrees with you on a random sample of runs, on the Verify screen. Until your checks agree with Morse at least 70% of the time, inferred labels stay in the background and are not used as outcomes.

Steps

  1. See where labels come from. Open Settings > Outcomes. Outcome coverage splits each agent’s runs into Reported (from your code, a webhook or a record), Rule, Inferred, Eval and Unlabeled. Each run counts once, under the label that currently wins. Reported labels are the ones Morse counts as confirmed. Rule and Inferred are estimates.
  2. Decide whether you need inferred labels. If most runs are Reported, you may not. If many are Unlabeled and you cannot report outcomes, an owner or admin can turn on Inferred outcomes (shadow) in the workspace settings. Morse then answers two questions on each finished run, and the answers stay in shadow: they do not affect verdicts, alerts or cost per confirmed outcome.
  3. Open Verify. Each item is one run with the question and rubric beside it. Answer Y, N or U (unsure) with the keyboard. You cannot choose or reorder items: Morse draws them at random from groups of the judge’s answers, and hides the judge’s answer until you submit, so it cannot sway you.
  4. Read the progress strip. It shows Runs checked (toward 30), Agreed “no” (toward 8), Agreement (at least) (toward 70%), and how many are still waiting. A “skipped” count per group shows how many checks you could not answer.
  5. Count it, or leave it. On the agent’s page, the Inferred outcomes panel (it also has a Verify button) says what is still needed. When the checks are enough, a card reads Ready to count inferred outcomes, and counting it is your decision, with a button such as Count “Need met” as success. Counted labels stay marked inferred and are never treated as confirmed outcomes.
  6. Read the correction, if shown. When there are enough checks, Morse can show an adjusted success rate that accounts for the judge’s mistakes. If not, it says how many more random spot-checks it needs.

What the numbers mean

  • Agreement (at least) is the low end of how often you agree with the judge: with this many checks, your checks agree with Morse at least this often (95% confidence). It is deliberately cautious so a lucky streak on a few checks cannot pass, and it doesn’t say the judge is calibrated. The threshold is 70%: Verify then reads “Your checks agree with Morse at least 70% of the time”.
  • Agreed “no” matters because a judge that says “yes” to everything agrees with you whenever the run really did succeed. Agreeing on “no” answers shows it can catch failures. At least 2 of the 8 must come from borderline “no” answers.
  • Skipped checks are counted per group. If more than 15% of borderline checks are skipped, evaluation is blocked, because the answered checks may not be representative.
  • A label from Morse’s judge is an estimate. It is never promoted to a confirmed outcome by repetition.

When there is not enough data

  • Verify stays empty until the sampler has drawn runs for a confirmed question. Confirm the agent’s question and let finished runs accumulate.
  • Checks only count for the question, judge and rubric that produced them. Editing the question or rubric asks for confirmation first, because existing checks would stop counting.
Last updated on
What does a good answer cost?Is my agent getting better?