Skip to Content
SDKInfrastructure spans

Infrastructure spans

When your agent runs inside a Morse trace, the SDK records the database queries, outbound HTTP requests and queue publishes it makes as child spans of that trace. You change no code. The point is correlation: when an agent run is slow or fails, the call underneath it is on the same timeline.

This is trace-anchored only. Nothing is recorded unless a Morse trace is open, and Morse does not monitor your infrastructure on its own.

⚠️

Requires SDK 0.7.0 (not yet released; 0.6.0 does not have this). The default-on behaviour, the MORSE_CAPTURE_* switches and queue spans below ship in 0.7.0. On the released 0.6.0, database capture needs MORSE_CAPTURE_DB=1.

From SDK 0.7.0 this is on by default. Earlier versions needed MORSE_CAPTURE_DB=1. To keep the old behaviour, set MORSE_CAPTURE_INFRA=0 — see Turning it off.

What is captured

KindPythonTypeScriptSpan
DBsqlite3, psycopg2, psycopg, Djangopg, mysql2, better-sqlite3db span named for the SQL verb
HTTPhttpx, requestsfetch (undici) and node:http / node:httpshttp span named METHOD host
QueueCelery, SQS (boto3)BullMQ, SQS (@aws-sdk/client-sqs)queue span named publish <destination>

An HTTP span also records the response status. A 5xx, 408 or 429 marks the span failed; an ordinary 4xx does not, because that is your request being wrong rather than the dependency failing.

A failed db, http or queue span stays failed on its own row and still shows up in repeated failures, but it never marks the whole trace failed. Your agent decides that: a caught database error or a retried 503 does not turn a successful run into a failed one.

Queue spans carry the OpenTelemetry messaging attributes messaging.system, messaging.operation.type and messaging.destination.name in the span metadata. On publish the SDK adds a W3C traceparent to the message (Celery headers, SQS message attributes, BullMQ telemetry.metadata) so a consumer can join the same trace — see Consumers.

LLM provider calls are not recorded as HTTP spans. The framework adapters already record them as llm spans, so an http span as well would count them twice. The Morse ingest host is excluded too.

What is never recorded

  • Bound SQL parameters. Only the statement text is kept, never the values passed with it.
  • Values written into the SQL string. If your code builds WHERE email = 'a@b.com' AND id = 42 by string formatting, the stored statement is WHERE email = ? AND id = ?. String, number, hex and dollar-quoted literals and SQL comments are removed before the span leaves your process.
  • URL paths and query strings. The HTTP span name is the method and the host. GET api.example.com, not GET api.example.com/users/42?token=….
  • Headers and bodies, in both directions.
  • Message bodies, task arguments and job data.
  • The text of an error. A failed infra span records the error class (and the driver’s error code for databases). Driver, URL and broker messages routinely contain row values, paths and credentials, so the message is dropped.

The SDK’s test suite seeds known canary values into all of the above and fails the build if any of them appears in an exported span.

⚠️

Scrubbing is a safety net, not a licence to format values into SQL. Use bound parameters. Two limits: double-quoted text is treated as an identifier, so a database that reads "abc" as a string literal (MySQL without ANSI mode) is not scrubbed there; and a secret that is part of an identifier, such as a table named after a customer, is not a literal and is kept.

How many spans one trace keeps

A trace keeps at most 200 db spans, 100 http spans and 100 queue publish spans. Calls past the limit are not recorded individually. Instead the trace gets one summary span per kind, named like +300 more db calls (capped at 200), with the count in infra.suppressed_count. An ORM looping 500 queries inside a tool shows as 200 spans plus one line saying how many more there were.

Turning it off

VariableEffect
MORSE_CAPTURE_INFRA=0Off for database, HTTP and queue
MORSE_CAPTURE_DB=0Off for database only
MORSE_CAPTURE_HTTP=0Off for HTTP only
MORSE_CAPTURE_QUEUE=0Off for queue publishes only
MORSE_CAPTURE_HTTP_EXCLUDE_HOSTS=a.com,b.comSkip these hosts (and their subdomains)

0, false, no and off all count as off. A specific variable wins over MORSE_CAPTURE_INFRA, so MORSE_CAPTURE_INFRA=0 with MORSE_CAPTURE_DB=1 still captures the database. The variables are read once, when the SDK initialises.

Expect more spans per trace after upgrading. If a dashboard of yours counts spans per trace, that count rises by the number of calls your agents make to a database, an HTTP service or a queue.

Consumers

Producer spans are on by default. Continuing the producer’s trace on the consumer side changes how your worker traces are shaped, so it is opt-in: set MORSE_CAPTURE_QUEUE_CONSUMER=1.

  • Celery (Python): automatic. The worker joins the producer’s trace and records a process <queue> span.
  • SQS (Python): wrap the handler: with morse_ai.queue_capture.continue_from_sqs_message(msg, queue_url=url): ...
  • BullMQ and SQS (TypeScript): neither library has a stable worker hook, so wrap the handler: continueFromBullMqJob(job, "worker-name", () => handle(job)), continueFromSqsMessage(msg, "worker-name", () => ...).

If the message has no traceparent, or it is malformed, the worker starts a fresh trace. It never raises.

Limits you should know

  • TypeScript database and queue capture patch at require() time. An ESM-only app needs OpenTelemetry’s import hook for the pg, mysql2, better-sqlite3, BullMQ and SQS patches to apply. HTTP capture uses diagnostics_channel and works in ESM and CJS.
  • Python wraps connect() for sqlite3, psycopg2 and psycopg, so a connection opened before morse_ai.init() runs is not covered. Django is covered either way.
  • Other clients are not covered: aiohttp, urllib3 used directly, asyncpg, Kafka and Redis Streams.
Last updated on