Skip to Content
Product GuideQualityCI Gate

CI Gate

A CI gate scores an eval dataset against your thresholds and returns a verdict. It runs in one of two modes:

  • Replay re-scores the outputs already recorded in the dataset’s cases. It catches scorer, judge and threshold changes, not changes to your agent’s code.
  • Fresh outputs has your CI job run your agent on the dataset’s inputs. Morse scores the new outputs, so it catches agent-code and prompt regressions too.

It has no dependency on the GitHub API — your CI step calls a REST endpoint and polls it, and your branch protection acts on the result.

What you see

How the gate works

An intro line explains the two modes and states plainly that Morse never touches your repo. The GitHub check result is pass or fail only; your branch protection is what enforces it.

Set up CI opens a panel where you pick a mode, a policy and a snippet format (GitHub Actions or curl), then copy the snippet. It calls the CI-scoped POST /evals/ci/run endpoint with an API key carrying the ci scope.

The CI endpoints are async: POST /evals/ci/run returns 202 immediately and your step polls GET /evals/ci/run/{id} until the run is completed.

Fresh outputs runs

A fresh run works in four steps:

  1. Create the run: POST /evals/ci/run with mode: "fresh". Optional settings are deadline_minutes (default 30, max 360), min_coverage (default 0.9) and max_cost_usd. A fresh run takes at most 1,000 cases.
  2. Page through the inputs: GET /evals/ci/run/{id}/cases?page=N, 100 per page.
  3. Run your agent on each input, and set two keys in the trace’s user_metadata: morse.ci.run_id (the ci_run_id from step 1) and morse.ci.case_id.
  4. Poll GET /evals/ci/run/{id}. Every linked trace is scored, with no sampling, within the run’s budget.

The run closes when every case has an output and is scored, or at the deadline. Its verdict is Passed, Failed or Incomplete. A run is Incomplete — neither a pass nor a fail — when fewer than min_coverage of the inputs came back, or when the run budget ran out. A run is compared only with an earlier run of the same mode; until one exists the page says No fresh baseline yet.

Traces carrying morse.ci.run_id are refused a link when the case id is missing, unknown or repeated, when they arrive after the deadline, or when the run has closed. Each run lists how many were refused for each reason, with the fix. A trace without morse.ci.run_id is an ordinary trace and never joins a run.

Recent CI runs

The Recent CI runs panel lists the last 50 runs, newest first: commit, mode, how many inputs were received and scored, the verdict and the reason. A run still waiting or scoring refreshes every 10 seconds. Opening a run shows its refused outputs and their fixes, its coverage and budget settings, and, for replay runs, how many judges were n/a because they need fresh outputs.

Policies

A readiness summary banner states how many of your policies are gate-ready versus how many “gate nothing meaningfully yet.” A policy is flagged weak when it has no scorer thresholds set, or when its dataset has fewer than 5 cases — either condition means the gate will pass regardless of agent behavior.

Readiness carries its basis: “Recorded outputs are re-scored; readiness is not test power. A gate that is set up can still miss a regression if its cases are few or unrepresentative.”

The policies table lists each policy’s name, a copyable policy ID, its bound dataset (with case count), scorer-threshold count, fail-on-error setting, readiness and latest run. New policy opens a dialog to create a policy from a name and a dataset; thresholds are configured after creation.

How to use it

  • Create a policy against a dataset, then set its per-scorer thresholds so it can actually fail a run.
  • Open Set up CI, choose Fresh outputs to test your agent’s code or Replay to test scorer and threshold changes, and copy the snippet into your pipeline.
  • Wire your branch protection rule to the resulting check — Morse’s role stops at the verdict.
  • If a fresh run ends Incomplete, open it in Recent CI runs: the refused outputs and their fixes say what to change.

Before you have data

With no policies yet, the page reads “No eval policies yet” with a Create your first policy button in place of the table. With no runs yet, Recent CI runs says runs appear once your pipeline starts one.

  • Cases — the datasets a policy binds to and gates on.
  • Regression — where replay runs establish the scores a policy checks.
  • Quality — CI Gate readiness surfaces here as an always-visible next action.
Last updated on
RegressionScore Settings