Skip to main content

Monitoring

The platform uses Sentry for alerting on two categories of failure: data quality check failures and Dagster run crashes. Both are registered automatically via build_core_defs() — no per-OEM configuration is needed.

Sensors​

SensorTriggerSentry level
asset_check_sentry_sensorFailed asset check evaluation (WARN or ERROR)warning / error
run_failure_sentry_sensorDagster run transitions to FAILEDerror

Both sensors start automatically (DefaultSensorStatus.RUNNING) after every deploy.

asset_check_sentry_sensor​

Polls the Dagster event log every 60 seconds for new ASSET_CHECK_EVALUATION events. Each code location's sensor instance is scoped to its own dagster/code_location tag, so a check failure in one OEM is only reported once — not duplicated across every running code location. Uses a cursor (storage ID) so only evaluations recorded since the last tick are forwarded — not historical ones. Batches up to 100 events per tick and paginates if more are pending.

run_failure_sentry_sensor​

Event-driven — Dagster calls it immediately when a run reaches FAILED status. No polling lag. Captures the job name, error message, OEM, and partition date.

sentry_freshness_sensor​

Polls for freshness state changes on the code location's assets, with ticks at least 120 seconds apart. An asset is late when its freshness state is WARN or FAIL. An alert is one Sentry event about a late asset. The sensor sends an alert to the issue fingerprinted freshness:<asset_key> on each of these transitions:

  • from PASS or UNKNOWN to WARN or FAIL;
  • from WARN to FAIL.

The sensor resolves that issue on a recovery, which is one of these transitions:

  • from WARN or FAIL to PASS;
  • from FAIL to WARN.

An alert opens the asset's issue when that issue does not exist yet or is resolved.

An asset's catch-up window is its policy's lower_bound_delta for a cron policy, and its fail_window for a time-window policy.

An ancestor of an asset, in the same code location, is failing at one of the asset's state changes when all of these hold:

  • it has a freshness policy;
  • it is in FAIL at that change, or its next change in the same batch of state changes is into FAIL;
  • it entered FAIL within the asset's catch-up window of that change;
  • an alert about it has been sent, either an untagged alert that opened its issue or its follow-up alert.

The failing_ancestor tag and the follow-up alert are described below. A topmost failing ancestor is a failing ancestor that has no failing ancestor of its own. When several qualify, the sensor names the one whose asset key sorts first. Within a batch, the sensor processes the changes of an asset's ancestors before the asset's own changes.

When an alert opens the asset's issue while an ancestor of the asset is failing, the alert has the tag failing_ancestor=<asset_key>, naming the topmost failing ancestor. The tag does not change which issue the alert goes to.

Notifications come from an alert rule configured in Sentry, called the Sentry alert rule below. The Sentry alert rule does not notify on events that have the tag, so it sends one notification for a failing ancestor instead of one per downstream asset. An alert sent to an open issue never has the tag, since the Sentry alert rule does not notify on an event for an open issue, with or without the tag.

Every freshness event lists the asset's downstream assets that have a freshness policy, whatever their freshness state. The Sentry alert rule does not notify on the tagged alerts of a failing ancestor's downstream assets, so its notifications show those assets only through this list on the ancestor's event. The event holds the list in three places:

LocationContentShown in the notification
Exception value26 downstream assets with freshness policies: 8 enriched, 5 inventory_stats, … (counts by asset group, largest first)Yes, up to 256 characters
downstream_count tagThe number of assets in the listWhen the Sentry alert rule includes the tag
downstream_assets extraEvery downstream asset key, one per lineNo; open the Sentry issue

When a tagged alert opens the asset's issue, the Sentry alert rule does not notify on it. The sensor then sends at most one more alert for the asset:

  • The sensor waits on every ancestor that was failing at the tagged alert. While the follow-up alert is pending, the sensor also waits on every ancestor that, at any tick, meets all of these:

    • it has a freshness policy;
    • it is in FAIL;
    • an alert about it has been sent;
    • it entered FAIL no earlier than the asset's catch-up window before the tagged alert.

    Once those ancestors, and every asset between them and the asset, have been out of FAIL for the asset's catch-up window, the sensor sends a follow-up alert without the tag if the asset is still late. The follow-up alert goes to the issue freshness:<asset_key>:followup. If an earlier follow-up alert left that issue open, the sensor resolves the issue first.

  • If the sensor stops tracking the asset while it is late (the asset leaves the code location or loses its freshness policy), the sensor sends the follow-up alert at once.

  • If the asset stops being late before either of these, the sensor sends no alert.

The sensor resolves the follow-up issue once the asset is no longer late or no longer tracked. When handling a follow-up alert raises an error, the sensor reports the error to Sentry and retries on the next tick. The sensor still advances its cursor past the batch.

The sensor cursor holds:

  • the storage id of the last state change it read;
  • the assets watched for a follow-up alert, each with the ancestors it waits on and the time of its tagged alert;
  • the assets whose freshness:<asset_key> issue is open;
  • the late assets that have had an alert sent.

Environment Variable​

Set SENTRY_DSN in each OEM project's .env file (or as a secret in Dagster Cloud):

SENTRY_DSN=https://<key>@<org>.ingest.sentry.io/<project-id>

If SENTRY_DSN is not set, asset_check_sentry_sensor returns a SkipReason and no events are sent. run_failure_sentry_sensor exits silently.

Sentry Event Structure​

Asset check failure​

FieldValue
messageDagster asset check failure: <check_name> on <asset_key> (<severity>)
levelwarning (WARN) or error (ERROR)

Tags (filterable in Sentry):

TagExample
oemaudi
check_namediscrepancies
severityWARN
partition_date2026-05-01

Extra (visible in issue detail):

FieldDescription
asset_keyFull asset key path
run_idDagster run ID — paste into the Dagster UI to inspect the run
partition_datePartition date, if applicable
metadataCheck-specific metadata (e.g. count, discrepancies)

Check-specific metadata​

CheckMetadata fields
discrepanciesrecords — up to 50 sample conflicting records
integritycount, missing_source_count, unexpected_row_count
unresolved_refscount, sample_refs

Run failure​

FieldValue
messageDagster run failure: <asset_path> (<error_type>) (with step info — step_key rewritten as a slash-separated asset path) or Dagster run failure: <code_location>/<job_name> (<error_class>) (fallback when no step ran)
levelerror

Tags: job_name, oem, partition_date (when present)

Extra: run_id, job_name, error (step error message), partition_date

Sentry Grouping​

Sentry groups events into issues by message. The same check failing on the same asset every day accumulates as one issue with an incrementing occurrence count. Tags (not extra) are used in custom fingerprint rules — check_name, oem, and severity are in tags for this reason.

Local Testing​

To verify events reach Sentry without a real pipeline run:

# Asset check failure
$env:SENTRY_DSN = "https://..."
uv run --directory projects/ai_audi python ../../scripts/test_sentry_e2e.py

# Run failure
uv run --directory projects/ai_audi python ../../scripts/test_run_failure_e2e.py

Both scripts call the sensor helpers directly against an ephemeral Dagster instance — no dg dev required.