Score Settings
Configure which scorers run, how often, write your own LLM-as-judge scorer, and author deterministic rule checks — no code deploy required.
📷 Screenshot: Score Settings with Rule checks list and a sample-rate card — pending Task 2.
What you see
A Settings view switch splits the page in two, and it opens on Basics:
- Basics — “How often each scorer runs, and your own scorers. Most teams only need this.” Scorer sample rates and custom scorers.
- Advanced — “Rule checks, the failure taxonomy and auto-sampling.” Rule checks, category taxonomies and auto-sampling rules.
Shape-aware default scorers
Morse chooses applicable built-in scorers from the data actually present in each trace, then applies your plan and enabled/sample-rate settings. LLM input can trigger hallucination checks; LLM output can trigger the Keyword check (a fixed list of harmful phrases, not a classifier), response coherence, and the Pattern-based PII check (regex patterns for emails, phone numbers, card numbers, keys and similar). Context segments, tool spans, a system-prompt segment, or multiple LLM spans enable the corresponding faithfulness, tool-call validity, instruction-following, or trajectory checks on eligible plans. A retriever span enables the advisory retrieval-relevance check (see Retrieval quality); it makes up to three judge calls per trace and never changes the quality verdict. A trace with no applicable scorer completes without an invented score. Active custom judges require LLM input and output too. A free, deterministic check, System prompt repeated in output, applies when a trace has both a system prompt and LLM output.
Scorer sample rates
One card per registered scorer, each with an enabled toggle and a 0–100% sample-rate slider in 5% increments.
Sampling is deterministic by trace_id — the same trace always lands on the same side of the
sample line, and a disabled or sampled-down scorer only skips scoring, it never affects trace
ingestion.
Active rule_* scorers also appear here (and again under Rule checks) so you can throttle or disable
them without editing the check document.
Custom scorers
A list of your org’s LLM-as-judge scorers with create, edit, and delete. Each scorer is defined by
a name, a prompt template, a judge model (gpt-4o-mini or gpt-4o), an output format (pass/fail
or a 0–1 score), and — for the 0–1 format — pass/warn thresholds.
Custom scorers run alongside the built-in scorers on every applicable trace and feed the same quality verdict shown on Quality.
Write the prompt as a pass/fail rule. The judge must answer in JSON:
{"pass": true|false, "reasoning": "..."} (or {"score": 0.8, "reasoning": "..."} for the 0–1
format).
What the judge sees decides how much of the trace reaches the judge:
- Conversation: the first user message and the agent’s replies. This is the default, and how every scorer created before evidence mode still runs.
- Evidence: every LLM call in order, every tool call with its arguments and result, retrieved
context, the trace metadata keys you list under Metadata keys (up to 20, read from the
trace’s user metadata), and the final response. On a replay of a saved test case, the case’s
expected output is added as a reference answer. Each section is capped in size and marked
[truncated N characters]when cut.
Use Evidence when the facts that decide pass or fail live outside the first message, for example a price returned by a tool or the investor profile sent as metadata. Evidence prompts are larger, so each judge call costs more. In both modes, emails, card numbers, keys and similar patterns are redacted before the judge sees the text. Changing the mode or the metadata keys starts a new scorer version.
Start from a template offers ready-made rubrics. The first pack is for investment-adviser
agents: Unsupported claims, Numbers match the data, Evidence is current, Policy and
suitability, and Appropriate uncertainty. Each is a separate pass/fail check in Evidence
mode, reading investor_profile and as_of_date from metadata, so you can see which failure is
rising instead of one blended score. A template opens the normal dialog prefilled; edit anything
before saving.
These checks judge only the evidence captured in the trace. They are not validated suitability or investment advice. Until a judge has enough expert labels, its row says Not yet calibrated against human reviewers.
Judge reliability
Each judge row has a Judge reliability badge that shows how often the judge agrees with your own experts. Below 30 labels it reads Not enough labels (12 of 30): the judge has 12 of the 30 random-case labels it needs, no figure is shown, and the Not yet calibrated line stays.
At 30 labels or more the line is removed and the badge names a band, for example Moderate
agreement · κ 0.71 (n 48, 0.55–0.84). κ is the agreement score, n is how many labels it rests
on, and the range is its 95% interval. The band comes from the low end of that range, so it never
claims more than the labels support:
- Strong agreement: the low end of the range is 0.6 or more.
- Moderate agreement: the low end is 0.4 or more.
- Agreement not yet established: κ is 0.4 or more but the low end is below 0.4. A caution reads “Range too wide to rely on. More expert labels will narrow it.”
- Weak agreement: κ is below 0.4. The badge is amber with a ”!”, and a caution reads “Don’t rely on this judge alone.”
The cut-offs follow the common Landis and Koch scale, and the details sheet states them.
Select the badge to open the details: a confusion matrix of expert against judge (Pass and Fail), the experts’ own fail rate, a By how cases were picked table, and a Between experts panel. A link below opens the disagreements still to resolve.
Why the headline uses random cases only. Flagged cases are mostly failures and representative cases are mostly passes, so averaging them would flatter or punish the judge. Random cases are the fair sample. Flagged and representative cases appear as separate rows marked Not in headline.
How to read the numbers.
- Agreement is the share of cases where the expert and the judge gave the same pass or fail. Cases the expert marked Unsure are left out.
- κ (kappa) corrects agreement for chance. Two reviewers who both say “pass” nearly every time agree a lot without telling you much, and κ accounts for that. Higher is better; 0 is no better than guessing.
- The range is a 95% interval. A wide range means more labels would help, so treat the number loosely.
- If every expert label gives the same verdict, κ can’t be worked out. The details say Can’t compute: every label agrees, and show agreement without a κ.
30 labels minimum. Below 30 random-case labels, Morse shows how many you have, not a number that a few labels could swing.
A new judge version starts over. Changing a judge’s prompt, mode or metadata keys creates a new version (see above), and labels made against the old version don’t count for the new one. The badge returns to 0 of 30. The old version’s figures stay readable.
Reliability sits beside a judge. It is never folded into the judge’s score, the pass rate or the quality verdict, and Morse doesn’t change thresholds for you based on it. Label cases in the review inbox.
Category taxonomies
For a classifier agent, one that sorts each case into categories such as news event types, a judge is the wrong instrument. Define a Category taxonomy instead and measure against labels your experts trust. Under Category taxonomies, choose to create one:
- Pick the Agent and give the taxonomy a Name.
- Under Categories, list 2 to 50 categories, one per line.
- Under Each case has, choose One category or Several categories.
- Under Where the agent’s pick lives, choose Metadata key (a key on the trace’s metadata that
holds the predicted category, such as
category) or JSON pointer (a path into the final response, such as/result/categories).
Only admins can create, edit or delete taxonomies. Experts then pick the right categories for each case in the same labeling screen, and the results show as Category accuracy on Quality.
Rule checks
Deterministic checks (JSON Schema, regex, or field assertions) evaluated on every in-scope run — not sampled judge signals, and not fused into one score with them. Include in the quality verdict only when you opt in: the Include in quality verdict switch is off by default.
Each row shows the rule name, version badge, enable toggle, sample-rate slider, and a status line: Checked on every in-scope run when enabled at 100%, a sampled percentage when throttled, or Disabled — skipped on every trace. Advisory rules are marked advisory; rules you opted in say they feed the quality verdict.
Create from Blank or a starter template (support, coding, generic, investment adviser). The investment-adviser rule flags promised-return language and a missing risk disclaimer. The editor is a JSON document with inline validation, Test on sample, and a read-only Backtest against recent traces of one agent. Create / edit / delete / bind controls are demo-guarded.
On a trace, rule scores appear in the Eval Scores panel as code scorers with pass / fail / n/a
labels; expand a row for reasoning and version. Per-check pass/fail/unknown results with reason
codes show in the rule editor’s test output.
Auto-sampling
A master toggle, a score-trigger threshold (queue a case whenever a scorer returns warn/fail below it), four deterministic rule triggers, a fingerprint sampler, and a daily cap.
The four rule triggers: R1 trace failed, R2 a tool/function span failed, R3 a latency outlier against the agent’s rolling p95 (needs 100+ traces), and R4 context utilization above 85%.
The fingerprint sampler queues traces whose tool-call + outcome fingerprint hasn’t been seen before, broadening behavioral coverage rather than re-sampling the same shape of failure.
A daily cap limits how many cases get auto-queued per UTC day; once hit, the page shows a banner and new cases resume at midnight UTC.
How to use it
- Turn a scorer’s sample rate down (not off) to cut LLM-judge cost on a high-volume trace type while keeping some signal.
- Write a custom scorer for anything the built-in scorers don’t cover — tone, format compliance, a domain-specific rule — directly from the dialog, no deploy.
- Add a rule check for contracts you can express as schema, regex, or field asserts (refund caps, ticket-id format, PR summary shape). Keep Include in quality verdict off until you trust the check; use Test on sample and Backtest before enabling on production traffic.
- Tune auto-sampling rule triggers to match what “worth reviewing” means for your agents; the fingerprint sampler is the one to enable if you want broader coverage rather than more of the same failure.
- Raise the daily cap if you’re hitting it and still missing cases you’d want reviewed.
Before you have data
Custom scorers show an empty state — “No custom judges yet. Start from a template below or create one.” — with the template list open below it. Demo mode includes one Evidence-mode judge, Refund matches fare rules, with two failing scores; create, edit and delete stay blocked.
Rule checks show “No rule checks yet.” with “Add a deterministic check that runs on every
in-scope trace.” Demo mode ships support and coding starter rules plus score rows (including one
fail and one unknown / n/a) under the demo org only — create/edit/delete stay blocked.