Skip to Content
Getting StartedCommon tasksIs my agent getting better?

Is my agent getting better?

Open Weekly Summary for a ranked read of every active agent against its own labeled history, then open one agent to see its success rate over time and the verdict on its latest change. Morse reports movement, and when the data cannot tell a real change from noise it says “too early” and shows what would settle it. It does not claim a change caused the movement.

Steps

  1. Start with the brief. Open Weekly Summary from the sidebar. Active agents are listed most urgent first. Each is compared with its own labeled history, and spend sits in its own column, so a cheaper week is never read as a better one.
  2. Open one agent. Select it. Success Rate shows the change against the previous window, and the Success Rate Over Time and Cost Per Day charts show the trend for the range you pick.
  3. Read the agent’s History. If you shipped a new agent version, git revision, model or prompt, History compares the runs before and after, one measure at a time: cost per run, typical response time (p50), slow-end response time (p95), error rate and success rate. Each reads improved, worsened, unchanged or inconclusive (too early).
  4. Check cost per confirmed outcome. On Cost Intelligence, Workflows, see whether each good result costs less than before. On Prompts, compare prompt versions by the same measure.
  5. Check quality separately. On Quality, the quality verdict and timeline come from sampled scorers and deploy markers. They are a separate signal and are not blended into the success rate.
  6. Open the runs behind a verdict. Each change verdict links to the before and after runs it was computed from.

What the numbers mean

  • Success rate is over labeled runs only. A jump in labeled runs (for example after you started reporting outcomes) can move the rate on its own, so check the run count beside it.
  • Improved or worsened means the plausible range for that measure sits clearly on one side of “no change”. Unchanged means any movement is smaller than a small margin (5% for cost, 3 points for success rate, for example). Inconclusive means too early.
  • Too early comes with a bound. For example “cost per run did not rise more than 12%”. That tells you how bad the change can be, even when it cannot yet tell you whether it helped.
  • Measures are never combined. A change can lower cost and raise errors. Morse shows both.
  • Patterns, not causes. Prompt edits, upstream model updates, traffic shifts and a busy week can all move a measure at once. Morse lists the overlapping changes it can see.

When there is not enough data

  • A measure needs about 100 runs on each side and 72 hours after the change before it gives a verdict, and success rate needs enough labeled runs on both sides. Until then it reads “collecting” with the progress so far.
  • No version or revision sent means nothing to detect. Send an agent version or git revision and Morse can find changes.
  • Small agents will mostly get bounds rather than verdicts. That is the honest answer, not a failure of the tool.
Last updated on
Can I trust Morse's labels?Set up outcomes