Reliability

Alerts, service targets (SLOs) and incidents used to be three disconnected screens you had to correlate yourself. Reliability puts them behind one URL — a posture strip up top, then Failures, Alerts, Service targets and Incidents tabs — so a service target shows you the early warning watching it, and an incident links back to the alert it was filed from. Failures opens first: what is failing, ranked by what it cost.

Reliability and repeated failures (0:42, narrated)

What you see

Reliability posture

A strip of three stat blocks under the tabs: Alerts firing (with how many fired in the last 24 hours against the usual daily count), Service targets breached, and Open incidents, each colored teal when clear and amber/red when not. Clicking a block opens that block’s tab and moves you to it.

The service-target block’s words come from Morse’s API, with the basis beside the label. When every target holds it reads, for example, “Healthy: 4 of 4 service targets met the target set for each in the latest check.” Otherwise it names how many are breached or at risk, with the next action — for example “Open Service targets and fix the at-risk ones before they breach.” With none defined it reads No service targets defined.

Alerts

A table of alert rules — agent, type, threshold, last fired, an Investigator status badge, an active toggle, edit, and delete (which asks before it removes the rule) — plus a Create Alert dialog. On a phone each rule is a card with the same controls.

Nine alert types are available: failure rate, inactivity, cost threshold, latency threshold, eval score, cost per run, daily spend, service target early warning, and metric anomaly — the create form’s fields (threshold label, extra config) change per type.

Two more types, budget threshold and budget forecast, are tied to a budget rather than an agent. They are evaluated hourly instead of on the 5-minute sweep. In the Create Alert dialog, choosing one of them replaces the agent picker with a budget picker (if you have no budgets yet, it links to the Budgets tab on the Cost page). A budget threshold alert also takes the percentages to alert at, such as 50, 80, 100. If the budget already has that kind of alert, the dialog tells you so instead of creating a second one.

A daily spend alert can watch your whole organization: leave the agent picker on Whole organization.

A new tool or MCP server alert takes no threshold. It fires once when an agent that is past its first 7 days calls a tool or MCP server it has not used before. See Tools first seen.

Budget threshold fires when a budget’s spend reaches a percentage of its amount — 50%, 80% and 100% by default. Each threshold fires once per period; editing the amount or toggling the budget does not re-arm one that already fired.
Budget forecast fires at most once per period, when a budget becomes projected over. It does not fire before the forecast is ready or once the budget is already over.

These alerts, and a daily spend alert scoped to the whole org, have no owning agent. They go to every org member who has notifications turned on for that severity and channel; if no member does, Morse emails the org owner.

Anomaly mode

For latency threshold, failure rate and eval score, the dialog has an Alert when choice: Static threshold (the default, a fixed line you pick) or Anomaly. An anomaly alert has no threshold. It fires when the agent is doing materially worse than it usually does, judged against that agent’s own last 7 days.

How it decides, in plain terms:

Now is the last 60 minutes: the time 95% of responses finished within (at least 20 runs), the failure rate (at least 20 runs with a known outcome; runs whose outcome is unknown are not counted as failures), or the average score for that agent and scorer (at least 10 scores).
Usual is the same measure for each hour of the previous 7 days. Morse takes the middle value and how much the hours normally vary. One bad hour in that history does not stretch “usual”.
It fires only when “now” is clearly outside the usual range and the change is large enough to matter: latency at least 25% above usual, or a failure rate or score at least 5 points (0.05) worse. Getting faster or scoring higher never fires.

Sensitivity sets how far outside usual counts as clear: Low (only large departures, fewest alerts), Medium (the default) or High (smaller departures too, more alerts).

Baseline forming. An anomaly alert needs at least 24 hours of activity (hours with at least 5 runs or scores) from the last 7 days before it can say what is usual. Until then the dialog shows “Baseline forming (11/24 hours)” and the alert never fires. That is deliberate: a quiet history is not evidence that something is wrong.

When you pick an agent, a preview card shows what the rule would say right now, for example “Usual 1.8s ±300ms · now 2.1s · would not fire”. If the last hour had too few runs to judge, it says so instead of reporting a verdict.

An anomaly alert waits 240 minutes (4 hours) between repeats by default, so one bad spell is one alert; you can change it. Its notifications say what was observed against what is usual, for example “95% of responses took up to 4.2s, against a usual 1.8s ±0.3s (last 60 min, 42 runs)”. They describe a measurement, not a cause. Whether Investigator runs on the alert follows the same rule-level and project-level setting as any other alert.

Eval score alerts, static and anomaly alike, now look only at scores for the alert’s own agent. Before, an eval alert on one agent also counted other agents’ scores for the same scorer, and scores with no agent attached. An existing eval score alert that fired because of another agent’s scores will stop doing so.

Alert titles are written in words, for example Failure rate, Daily spend or Service target early warning — never the raw rule type.

Clicking an agent name opens that alert’s detail page.

Service targets

A service target (an SLO) is a level someone on your team set for an agent — for example, “95% of responses finished within 3.0 s over 30 days”. The Service targets tab lists them in one table (cards on a phone): Service target, Status, Current, Target · window, Failure allowance left and Early warning.

Failure allowance left is how much of the target’s error budget remains in the window, for example “62% failure allowance left”.
Early warning is the alert watching the target. It fires when the failure allowance is being used up faster than the window allows, so the next slide reaches someone before the target is breached. A target with none reads No early warning · Add one →: a breach notifies no one.

Each row’s status comes from Morse’s API as label → basis → reason → next action:

Healthy — “Healthy: 96.1% of runs with a reported outcome succeeded against a 95% target over the last 30 days.”
At risk — still meeting the target, but less than 20% of the failure allowance is left; the next action is to look at what changed recently before the allowance runs out.
Breached — outside the target; the next action is to open the runs behind it.
Not enough data — nothing was measured in the window, so it is neither met nor missed.

Response-time targets read as a sentence: 95% of responses finished within {X} (or 99%).

Target types are success rate, response time (95th or 99th percentile), availability, cost per run, or a custom metric. The window is a preset dropdown — 7, 14, 30, or 90 days — not free text.

An Evaluate button re-runs all service-target evaluations on demand, and New service target opens the create form, which can define a brand-new custom metric inline without leaving the flow.

Incidents

A filterable (all / open / investigating / resolved) list of incident rows, each with a severity badge, a status badge, and a relative timestamp; a New Incident button opens an inline create form. You can also open an incident straight from an alert’s firing, on the alert page.

Clicking a row opens that incident’s timeline.

How to use it

Scan the posture strip first — it tells you which tab, if any, needs attention.
Switch tabs to work the specific surface: create or toggle an alert, define a service target (and its custom metric, if needed), or file and filter incidents.
Follow an early-warning link from a service target, or an alert link from an incident’s timeline entry, to move between the three without losing context.
Each tab’s empty state cross-links to the other two, so picking the wrong tab for what you’re trying to do isn’t a dead end.

The early warning uses Google’s SRE-workbook single-window burn-rate formula — current_error_rate / error_budget — evaluated once per service target per check, not the dual-window (fast+slow) technique. It fires at a set multiple of the pace the failure allowance is used.

Before you have data

No alerts configured: “No alerts configured”, with a prompt pointing to Service targets and Incidents.
No service targets: “No service targets yet. Use New service target to set the first reliability target.” A target with no qualifying runs in its window reads Not enough data.
No incidents recorded: “No incidents recorded” (or “No incidents match this filter” when a status filter is active), with a prompt pointing to Alerts and Service targets.

To get your first alert or service target here, follow the quickstart.

Repeated failures

When the same failure keeps happening, Morse groups it instead of listing every run. A group is the failed runs that share a span type, an exception type, and a message with the volatile parts (ids, numbers, emails, URLs, quoted values) stripped out. Each group shows grouped on: <type> · <normalized message> · <span>, so you can see exactly why runs landed together.

The panel appears in the weekly summary and on each agent’s page. For every group it shows how many runs failed in the window, the spend on those runs, the agents affected, and when it was first and last seen. Groups are ranked by what the failures cost, not by raw count.

New means the group was first seen inside the window.
Seen again after deploy means occurrences stopped for a day or more, a deploy was detected, and the same failure came back. It describes timing. It does not say the deploy caused it.
A failed run with no error at all (a run you marked success=false, or that ended on a refusal, a length cut-off, or a content filter) is grouped by agent, the step it ended on, and its status. Runs with no outcome label are never counted as failed.
View traces opens the Traces list filtered to that group (/traces?error_group=<id>).
Ignore hides a group and keeps it out of the attention list. Merge into folds a group into another when you consider them the same failure; Split undoes a merge. You need write access for all three.

Only the normalized message and its hash are stored as group text. One real example message is kept encrypted, and groups disappear when the underlying spans age out of your retention window.

To group failures by exception type, send it with the error. The Python and TypeScript SDKs do this for you on a span that raises; OTLP exporters send the standard exception.type and exception.message span event (record_exception in every OpenTelemetry SDK).

Alert — one alert’s rule, history, and investigation hypothesis.
Incident — one incident’s full timeline.
Traces — the runs an alert, service target, or incident is watching.