Checks

Seven checks decide an agent’s level. Each runs per agent, on assessed runs only, and has one of five states. When a check is not met, below its minimum sample or not measured, the app shows one next step and links to its section here.

Thresholds below are the defaults, and each check in the app shows the values in force. The app lets an owner or admin change one of them, the PII leak threshold, on the Privacy tab; the others are not editable in the app. See Thresholds for every value.

Real traces in the assessed environment

Level: L1. Window: the assessed window (14 days by default).

Met when the agent has at least one assessed run in the window. Because an agent is only assessed when it has runs in the window, every assessed agent meets this check.

If it is not met: send runs from this agent to the assessed environment. Install the SDK, set your API key and run the agent once. See the Quickstart.

pip install morse-ai

Redaction beyond defaults

Level: L2. Window: the assessed window.

Met when any of these is true for the workspace:

• a redaction pattern beyond Morse’s defaults is turned on;
• a custom redaction pattern is added; or
• at least one of the agent’s runs in the window arrived with values already masked by the SDK.

If it is not met: an owner or admin turns on a built-in pattern that is off by default (today that is IP addresses), or adds a custom pattern, in Settings › Workspace (/settings/workspace), under Redaction: built-in patterns and Redaction: custom patterns. Or turn on masking in the SDK. See Data handling and Redaction evidence.

PII-leak scoring

Level: L2. Window: the assessed window. Minimum sample: 20 scored runs.

Met when the Pattern-based PII check scorer (pii_leakage_v1) has scored at least 20 of the agent’s assessed runs in the window. Runs the scorer marked “n/a” or “error” do not count.

• No scored runs: Not measured.
• 1 to 19 scored runs: Below minimum sample, for example “n = 12 of 20”.

A PII leak, here, is PII in the agent’s output that the requester did not give it in the conversation. This check only asks whether leaks are being measured. Leaks that are found open a PII leak exposure or PII leak regression finding.

If it is not met: in Quality › Score settings (/quality/score-settings), make sure Pattern-based PII check has a sample rate above 0%. It is on for every run by default; a sample rate of 0% turns it off. Then send more runs. See Score Settings.

Guardrail verdicts on runs

Level: L3. Window: the assessed window. Minimum sample: 20 runs.

Met when at least 95% of the agent’s assessed runs carry a working guardrail verdict: allow, flag, redact or block. Verdicts of error or unreachable mean the run was not checked, so they do not count.

• No runs: Not measured.
• 1 to 19 runs: Below minimum sample.

Morse records the verdicts of the guardrails you already run. It does not run a guardrail for you and is not in the request path.

If it is not met: run a guardrail on every request and send its verdict. Morse reads verdicts from LiteLLM, OpenInference and OpenAI Agents guardrail spans. See Guardrail verdicts, including how to report a guardrail you call yourself.

Tool allowlist

Level: L3. Window: none; it reads the agent’s current allowlist.

Met when the agent’s tool allowlist has at least one entry.

Not applicable when no tool or MCP server has been seen for the agent. The check reads “Not applicable: this agent has no tools”, and the agent can still reach L3.

The allowlist records which tools the agent is meant to call. Morse watches calls; it is not in the request path. Calls to a tool that is not on the list are listed on Agent access, not yet flagged as findings.

If it is not met: an owner or admin sets the allowlist for the agent on the Agent access tab.

Current release tested

Level: L4. Window: the agent’s current release, not a number of days.

Met when the current release has either:

• An imported red-team run whose commit matches the release, with the agent set on the import, results in at least 5 OWASP LLM categories and a pass rate of at least 90%; or
• 3 complete fresh CI runs whose commit matches the release, on the seed suite or the agent’s Security regressions dataset, with every case passing in every run. One passing run is not enough: the app shows “1 of 3 runs so far”.

Other states:

• Not measured when the agent has no release: its runs carry no git revision of at least 7 characters.
• Not met with “No test run on this release yet” when nothing has run on the release.
• Not met when a run on the release failed. That is a measured gap, so the agent counts toward the workspace level.
• An import with no agent set does not count for any agent, because one import must not satisfy L4 for every agent that shares the commit. The next step asks you to set the agent on the import.

On Free, only the seed suite’s own cases are graded on the one agent it is installed for.

If it has no release: send the git revision with your runs. The Python SDK reads it from the first of these that is set: MORSE_GIT_REVISION, GITHUB_SHA, CI_COMMIT_SHA, RAILWAY_GIT_COMMIT_SHA, VERCEL_GIT_COMMIT_SHA, RENDER_GIT_COMMIT, SOURCE_VERSION, then git rev-parse HEAD. In a Docker image without a .git directory, bake it in at build time:

ARG GIT_SHA
ENV MORSE_GIT_REVISION=$GIT_SHA
# docker build --build-arg GIT_SHA=$(git rev-parse HEAD) .

See Git revision.

If it has a release: import a red-team run for it (see Imported test runs), or run the seed suite in your CI 3 times on the release with every case passing. A fresh-outputs CI run needs an API key with the ci scope:

curl -sS -X POST "$MORSE_API_URL/api/v1/evals/ci/run" \
  -H "Authorization: Bearer $MORSE_CI_API_KEY" -H "Content-Type: application/json" \
  -d '{"policy_id": "<policy id>", "dataset_id": "<seed suite dataset id>",
       "mode": "fresh", "commit_sha": "'"$GITHUB_SHA"'"}'
# Then: GET /api/v1/evals/ci/run/{id}/cases?page=N, run your agent on each input with
# user_metadata {"morse.ci.run_id": <ci_run_id>, "morse.ci.case_id": <case id>},
# and poll GET /api/v1/evals/ci/run/{id} until the run is completed.
# Run it three times on the same commit: every case must pass in every run.

The app shows this snippet with your seed suite’s dataset id filled in once the suite is installed. See CI Gate for the fresh-outputs flow.

Open findings owned and fixes verified

Level: L5. Window: the fix lookback, 90 days by default.

Met when both are true for the agent’s security findings:

• every high-severity finding opened in the last 90 days is a verified fix (closed as resolved by a person) or has been set aside; and
• every open finding has an owner and a follow-up date that has not passed.

Only a person-verified fix counts as verified. A high-severity finding that an owner or admin dismissed as not relevant, or accepted as intended, is set aside: it is left out of the check, so it neither turns it green nor holds it red. The same finding closed by anyone else counts as not verified and keeps the check unmet, so a member with write access cannot clear it.

With no findings at all this check is met, which is why levels are cumulative: an agent reaches L5 only after it meets L1 to L4.

The machine never closes a finding. See how a finding closes.

If it is not met: close each high-severity finding as resolved once the fix is verified (an owner or admin can instead dismiss it, or accept a tool change as intended), and give every open finding an owner and a follow-up date.

Thresholds

The values the checks and findings use, with their defaults. Each is a workspace setting. In the app, an owner or admin can change only the PII leak threshold, on the Privacy tab under Thresholds.

Related

• Overview and levels: how the checks add up to a level.
• Findings: the findings that cap a level and feed L5.
• Areas: where each check shows in the app.