Skip to Content
Getting StartedCommon tasksSet up outcomes

Set up outcomes

An outcome is whether a run’s work held up: the ticket stayed resolved, the pull request stayed merged. Without outcomes Morse can still trace and price every run, but it cannot say how often your agent succeeds or what a good answer costs. Open Settings > Outcomes to see how many runs have one and the single next step Morse suggests. Pick the source that fits how you already work.

Morse TVListen · 3 short clips · 2:13
Outcomes in three clips
  1. 1What counts as an outcome?
    • An outcome: did the run's work hold up?
    • No outcome means unlabeled, not failed
    • Settings > Outcomes shows your next step
    Open Settings › Outcomes in Morse → (opens in a new tab)
    Transcript

    An outcome is whether a run's work held up. The ticket stayed resolved. The pull request stayed merged. A run with no outcome is unlabeled, not failed. Morse still traces and prices it, but cannot say how often your agent succeeds. Open Settings, then Outcomes, to see your one next step.

  2. 2How do I report an outcome from code?How to
    1. Wrap the agent's work in a run: morse_ai.run("support-agent")
    2. Check the result with something your code can really verify
    3. Report it: r.set_outcome(success=..., outcome="...")
    4. Can't tell? Leave the outcome out; the run stays unlabeled
    5. Watch the labeled share rise in Settings > Outcomes
    Check coverage in Settings › Outcomes → (opens in a new tab)
    with morse_ai.run("support-agent") as r:
    answer = handle(ticket)
    replied = bool(answer) and ticket_has_reply(ticket)  # a check your code can make
    r.set_outcome(success=replied, outcome="replied" if replied else "no_reply")
    Transcript

    Here is how to report an outcome from code. Wrap the agent's work in a run. Check the result with something your code can really verify, then call set outcome with success and a short outcome name. This label is the strongest, and nothing later overwrites it, so never hard-code success as true. If your code cannot tell, leave it out. The code is on the card, ready to copy.

  3. 3How do I say what success means in the app?Screencast

    Watch walkthrough · 1:18 · captions on · best in full screen

    Open Agents in Morse → (opens in a new tab)
    Transcript

    To say what success means, open any agent from the Agents page. At the top of the agent's page is What success means for this agent. It takes about a minute, needs no code, and members and admins can confirm it. It has four steps: define success, connect outcomes, see the first result, and decide. In your own workspace, you pick what a good result looks like. For example, a support ticket that is resolved and stays resolved, or a pull request that is merged and kept. Then you choose how many runs need an outcome before Morse judges the agent at all. To see where you stand, open Settings, then Outcomes. Outcome coverage shows how many runs have an outcome, and where each one came from: reported from your code, a rule, inferred, an eval, or still unlabeled. The line at the top names your one next step. Unlabeled runs are not failures; they just have no outcome yet.

Steps

  1. Agree what success means. On the agent’s page, What success means for this agent records what counts as a success, without code. Whoever owns the agent can do this and send you the setup link; the link opens a setup page that ticks off each step below as data arrives. See Agent detail.

  2. See where you stand. Open Settings > Outcomes. The line at the top names the one next step. Outcome coverage shows, per agent, how many runs are labeled and by what: Reported, Rule, Inferred, Eval or Unlabeled. Unlabeled means no outcome yet, not failed.

  3. Report it from your code (fastest). When your code can check the result, say so inside the run. Derive the outcome from that check:

    with morse_ai.run("support-agent") as r:
    answer = handle(ticket)
    replied = bool(answer) and ticket_has_reply(ticket)  # a check your code can actually make
    r.set_outcome(success=replied, outcome="replied" if replied else "no_reply")

    This is the strongest label: it wins over every other source, and nothing that arrives later (an eval, a classifier, a webhook) can overwrite it. That is why you should never write success=True as a constant: copied into a real agent, it marks every run a success and Morse cannot correct it. If your code cannot tell, leave the outcome out. The run stays unlabeled, which is not the same as failed, and one of the sources below can label it later.

  4. Tag the business record, then report later. If the real result arrives later (a ticket is reopened, an order is refunded), tag each run with the record it handled, then send the result by that key:

    with morse_ai.run("support-agent", record_keys={"zendesk_ticket": ticket.id}) as r:
    ...

    The record_keys argument ships with SDK 0.7.0, which is not released yet. Until then, send the same keys over REST or OpenTelemetry, as shown in Record keys and versions.

    Then post {"key_type": "zendesk_ticket", "key_value": "4812", "success": false, "outcome": "reopened"} to the record outcomes API. Morse labels every run that carries the key. Keys also make cost per confirmed outcome count records, not runs.

  5. Coding agent? Connect GitHub. In the GitHub panel on Settings > Outcomes, choose Connect GitHub, pick the repositories to read, and tag runs with github_pr = owner/repo#number. A merged pull request becomes a provisional success, and a verified revert inside the agent’s window turns it into a failure.

  6. Use a rule for repeatable checks. In the Outcome rules panel, choose a template (coding or support), the record key type and event source, then what to write when the rule passes and when it fails. A rule that maps to nothing never writes an outcome.

  7. Check the result. Back in Outcome coverage, the labeled share rises as outcomes arrive. It reads like “Outcome coverage: 840 of 1,200 runs have an outcome (70%).” Recent ones show as provisional until their window passes, then mature.

What the numbers mean

  • Reported counts labels from your code, a person, a webhook or a record. Only these count as confirmed in cost per confirmed outcome.
  • Rule and Inferred are estimates. They are shown, not counted as confirmed.
  • Which label wins when sources disagree: your code first, then a confirmed classifier, a webhook, a rule, a record, an inferred label, then a sampled quality check. A rule never replaces a record label.
  • Morse does not predict when outcomes will arrive. It shows what you can do now.

When there is not enough data

  • No runs yet: coverage reads “No runs to measure yet”. Tracing keeps working while you set outcomes up.
  • A record key that matches nothing yet is kept and applied when a matching run arrives. If your organization has never sent that key type, the response says so, which usually means a typo.
  • Prefer not to report outcomes at all? See Can I trust Morse’s labels? for inferred outcomes and how to check them.
Last updated on
Is my agent getting better?Data Model