Monitoring
The platform uses Sentry for alerting on two categories of failure: data quality check failures and Dagster run crashes. Both are registered automatically via build_core_defs() — no per-OEM configuration is needed.
Sensors
| Sensor | Trigger | Sentry level |
|---|---|---|
asset_check_sentry_sensor | Failed asset check evaluation (WARN or ERROR) | warning / error |
run_failure_sentry_sensor | Dagster run transitions to FAILED | error |
Both sensors start automatically (DefaultSensorStatus.RUNNING) after every deploy.
asset_check_sentry_sensor
Polls the Dagster event log every 60 seconds for new ASSET_CHECK_EVALUATION events. Each code location's sensor instance is scoped to its own dagster/code_location tag, so a check failure in one OEM is only reported once — not duplicated across every running code location. Uses a cursor (storage ID) so only evaluations recorded since the last tick are forwarded — not historical ones. Batches up to 100 events per tick and paginates if more are pending.
run_failure_sentry_sensor
Event-driven — Dagster calls it immediately when a run reaches FAILED status. No polling lag. Captures the job name, error message, OEM, and partition date.
sentry_freshness_sensor
Polls for freshness state changes on the code location's assets, with ticks at least 120
seconds apart. An asset is late when its freshness state is WARN or FAIL. An alert is
one Sentry event about a late asset. The sensor sends an alert to the issue fingerprinted
freshness:<asset_key> on each of these transitions:
- from
PASSorUNKNOWNtoWARNorFAIL; - from
WARNtoFAIL.
The sensor resolves that issue on a recovery, which is one of these transitions:
- from
WARNorFAILtoPASS; - from
FAILtoWARN.
An alert opens the asset's issue when that issue does not exist yet or is resolved.
An asset's catch-up window is its policy's lower_bound_delta for a cron policy, and its
fail_window for a time-window policy.
An ancestor of an asset, in the same code location, is failing at one of the asset's state changes when all of these hold:
- it has a freshness policy;
- it is in
FAILat that change, or its next change in the same batch of state changes is intoFAIL; - it entered
FAILwithin the asset's catch-up window of that change; - an alert about it has been sent, either an untagged alert that opened its issue or its follow-up alert.
The failing_ancestor tag and the follow-up alert are described below. A topmost failing
ancestor is a failing ancestor that has no failing ancestor of its own. When several
qualify, the sensor names the one whose asset key sorts first. Within a batch, the sensor
processes the changes of an asset's ancestors before the asset's own changes.
When an alert opens the asset's issue while an ancestor of the asset is failing, the alert
has the tag failing_ancestor=<asset_key>, naming the topmost failing ancestor. The tag
does not change which issue the alert goes to.
Notifications come from an alert rule configured in Sentry, called the Sentry alert rule below. The Sentry alert rule does not notify on events that have the tag, so it sends one notification for a failing ancestor instead of one per downstream asset. An alert sent to an open issue never has the tag, since the Sentry alert rule does not notify on an event for an open issue, with or without the tag.
Every freshness event lists the asset's downstream assets that have a freshness policy, whatever their freshness state. The Sentry alert rule does not notify on the tagged alerts of a failing ancestor's downstream assets, so its notifications show those assets only through this list on the ancestor's event. The event holds the list in three places:
| Location | Content | Shown in the notification |
|---|---|---|
| Exception value | 26 downstream assets with freshness policies: 8 enriched, 5 inventory_stats, … (counts by asset group, largest first) | Yes, up to 256 characters |
downstream_count tag | The number of assets in the list | When the Sentry alert rule includes the tag |
downstream_assets extra | Every downstream asset key, one per line | No; open the Sentry issue |
When a tagged alert opens the asset's issue, the Sentry alert rule does not notify on it. The sensor then sends at most one more alert for the asset:
-
The sensor waits on every ancestor that was failing at the tagged alert. While the follow-up alert is pending, the sensor also waits on every ancestor that, at any tick, meets all of these:
- it has a freshness policy;
- it is in
FAIL; - an alert about it has been sent;
- it entered
FAILno earlier than the asset's catch-up window before the tagged alert.
Once those ancestors, and every asset between them and the asset, have been out of
FAILfor the asset's catch-up window, the sensor sends a follow-up alert without the tag if the asset is still late. The follow-up alert goes to the issuefreshness:<asset_key>:followup. If an earlier follow-up alert left that issue open, the sensor resolves the issue first. -
If the sensor stops tracking the asset while it is late (the asset leaves the code location or loses its freshness policy), the sensor sends the follow-up alert at once.
-
If the asset stops being late before either of these, the sensor sends no alert.
The sensor resolves the follow-up issue once the asset is no longer late or no longer tracked. When handling a follow-up alert raises an error, the sensor reports the error to Sentry and retries on the next tick. The sensor still advances its cursor past the batch.
The sensor cursor holds:
- the storage id of the last state change it read;
- the assets watched for a follow-up alert, each with the ancestors it waits on and the time of its tagged alert;
- the assets whose
freshness:<asset_key>issue is open; - the late assets that have had an alert sent.
Environment Variable
Set SENTRY_DSN in each OEM project's .env file (or as a secret in Dagster Cloud):
SENTRY_DSN=https://<key>@<org>.ingest.sentry.io/<project-id>
If SENTRY_DSN is not set, asset_check_sentry_sensor returns a SkipReason and no events are sent. run_failure_sentry_sensor exits silently.
Sentry Event Structure
Asset check failure
| Field | Value |
|---|---|
message | Dagster asset check failure: <check_name> on <asset_key> (<severity>) |
level | warning (WARN) or error (ERROR) |
Tags (filterable in Sentry):
| Tag | Example |
|---|---|
oem | audi |
check_name | discrepancies |
severity | WARN |
partition_date | 2026-05-01 |
Extra (visible in issue detail):
| Field | Description |
|---|---|
asset_key | Full asset key path |
run_id | Dagster run ID — paste into the Dagster UI to inspect the run |
partition_date | Partition date, if applicable |
metadata | Check-specific metadata (e.g. count, discrepancies) |
Check-specific metadata
| Check | Metadata fields |
|---|---|
discrepancies | records — up to 50 sample conflicting records |
integrity | count, missing_source_count, unexpected_row_count |
unresolved_refs | count, sample_refs |
Run failure
| Field | Value |
|---|---|
message | Dagster run failure: <asset_path> (<error_type>) (with step info — step_key rewritten as a slash-separated asset path) or Dagster run failure: <code_location>/<job_name> (<error_class>) (fallback when no step ran) |
level | error |
Tags: job_name, oem, partition_date (when present)
Extra: run_id, job_name, error (step error message), partition_date
Sentry Grouping
Sentry groups events into issues by message. The same check failing on the same asset every day accumulates as one issue with an incrementing occurrence count. Tags (not extra) are used in custom fingerprint rules — check_name, oem, and severity are in tags for this reason.
Local Testing
To verify events reach Sentry without a real pipeline run:
# Asset check failure
$env:SENTRY_DSN = "https://..."
uv run --directory projects/ai_audi python ../../scripts/test_sentry_e2e.py
# Run failure
uv run --directory projects/ai_audi python ../../scripts/test_run_failure_e2e.py
Both scripts call the sensor helpers directly against an ephemeral Dagster instance — no dg dev required.