Ingestion Design
The ingestion layer extracts raw data from OEM-specific external APIs, validates it, and progressively refines it into consolidated entity records partitioned by day. Each OEM runs its own independent Dagster deployment. Shared infrastructure lives in a common library so OEM pipelines don't share state or affect each other.
Pipeline Structure
Every ingestion pipeline follows the same three-stage structure:
| Stage | Responsibility |
|---|---|
| Raw | Fetch data from the source API and store the response exactly as returned |
| Transformed | Extract entity records from the raw response, deduplicate by key group, and validate |
| Consolidated | Merge transformed records across all sources for an entity type |
The optional composite tier merges a market's public inventory sources into a daily snapshot. It reads the transformed tier. See CompositeInventoryComponent.
Each stage has asset checks that run automatically after materialization.
Where a correction belongs
A tier's output is the contract its readers rely on. A wrong, duplicated or unbounded value is corrected in the tier that produces it, so every reader gets the same value:
| Problem | Corrected in |
|---|---|
| A source names or encodes a value differently from the other sources | The source's alias subclass or extractor, in the transformed tier |
| One source returns several rows for one key | The transformed tier's deduplication |
| Sources disagree on a value for one entity key, such as two texts for one option code | The consolidated tier: the entity's source groups and tiebreak |
| A computed column needs a floor, a cap or a minimum, such as the minimum divisor of a rate | The defs.yaml of the component that computes the column |
The derived tier, featurization and candidate selection read one row per entity key and use each computed column as its producer wrote it. A duplicate row or an implausible value there is a defect in the producing tier, and the fix goes in that tier.
Dependency Ordering
The asset DAG has two independent ordering dimensions that do not form cycles:
-
Source dependencies operate in the raw and transformed tiers. One raw source may depend on another (e.g. fetching search results requires dealer IDs from a prior fetch). These dependencies are fully resolved by the end of the transformed tier.
-
Entity dependencies begin at the consolidated tier, where
foreign_keysallow one entity's consolidation to reference another consolidated entity's same-partition data. For example, consolidated inventory can reference consolidated models to attach a resolved model ID to each row. Since all source-level dependencies are already resolved, entity ordering is independent of source ordering.
This separation guarantees no circular dependencies: sources never depend on entities, and entities never depend on sources.
Asset Key Conventions
Dagster asset keys form a path hierarchy using / as the separator:
| Stage | Pattern | Example |
|---|---|---|
| Source | <oem>/<market>/sources/<source>_<resource> | audi/us/sources/scs_search |
| Raw | <oem>/<market>/raw/<source>_<resource> | audi/us/raw/scs_search |
| Transformed | <oem>/<market>/transformed/<name>_<entity> | audi/us/transformed/onegraph_inventory |
| Composite | <oem>/<market>/composite/public_inventory | mercedes/mbusa/composite/public_inventory |
| Consolidated | <oem>/<market>/consolidated/<entity> | audi/us/consolidated/inventory |
Asset Checks
| Stage | Check name | What it verifies | Blocking |
|---|---|---|---|
| Raw | fetch_attempts | All HTTP requests made by the fetcher returned 2xx; per-status failure patterns recorded in metadata | Yes |
| Transformed | discrepancies | No conflicting rows for the same entity ID; same-source field disagreements grouped into discrepancy patterns | No |
| Consolidated | integrity | Every row has a non-null entity ID | Yes |
| Consolidated | unique_ids | No entity ID appears in more than one row; the repeated IDs' count and examples recorded in metadata | Yes |
| Consolidated | unresolved_refs | Cross-entity reference links resolved successfully | Configurable |
| Transformed (paired inventory sources) | cross_source_coverage | Membership overlap between two inventory feeds stays within an expected range; per-side coverage and a breakdown of keys missing from the other feed recorded in metadata | Configurable |
A non-blocking check surfaces an issue in the Dagster UI without aborting the run. The fetch_attempts check additionally reports tracking=instrumented or tracking=uninstrumented in its metadata so operators can tell whether the source module is recording per-request outcomes (see Component Reference).
Of the checks in the table, fetch_attempts, integrity and unique_ids always block. unresolved_refs blocks when the entity's unresolved_refs_blocking is true. cross_source_coverage blocks when its blocking flag is set, and then emits at severity ERROR. A failing blocking check skips every asset downstream of the checked assets in the same run. See ConsolidatedComponent → Cross-source coverage for configuration and for the runs that blocking does not gate.
Failing checks are triaged per the rules in Verification → Alert triage — potential raw-collection issues take priority over everything else, because raw gaps are permanent.
Response Integrity
A truncated response can look complete at the status line: the request succeeded, the status is 200, and only the payload is short. Left unchecked it is written to the raw tier as a partial JSON body and fails days later as a JSONDecodeError inside an OEM extractor.
Truncation splits into two cases, handled at two different layers.
Framing truncation. A body shorter than its declared Content-Length never becomes a response object. httpx raises RemoteProtocolError; libcurl raises CURLE_PARTIAL_FILE (18) or CURLE_RECV_ERROR (56). default_request_error_handler retries all three. Application code does not check Content-Length because the transports raise before that check can run. packages/ai_http/tests/test_errors.py covers this with a socket server that declares a longer body than it sends.
Payload truncation. An origin or CDN can truncate JSON upstream, then frame the partial bytes as a complete HTTP message. Content-Length matches the truncated body, so the transport cannot detect the failure. Every successful response passes through raise_for_incomplete_body inside HttpSession.request_events and the standalone http_get_with_retry / http_post_with_retry helpers. The check reads a bounded window at the start and end of the body. A body whose first significant character is { or [ must end with the matching } or ].
The check runs before _validate_and_get_bytes, so validators never see an incomplete body. GM's AEC check and Genesis's dealer response check parse the body with ijson. Without the guard, a cut body makes the validator raise ijson.IncompleteJSONError, which the GM check's except ValueError does not catch.
A body that fails the check raises TruncatedResponseError, a subclass of TransientHTTPError. The request is retried until its retry budget is exhausted, then recorded as a failed attempt in the raw asset's fetch_attempts check. Custom RequestErrorHandlers should branch on the exception type. A truncated body keeps the 2xx status it arrived with, so a status-only handler cannot identify it.
Only 2xx bodies are checked. On a failure status, the status itself is the error. A short error page must not hide that status, and a terminal status must not be retried.
Two limits are worth knowing:
- The terminator check is necessary but incomplete. A stream cut exactly on an inner object boundary (
{"items":[{...},{...}) can still end with a plausible terminator. An exact check requires a full parse, which allocates several times the body. The check accepts bodies whose first or last significant character is outside the scan window. - One rejected page does not reject the partition. The truncated document is not stored. With
abort_on_failure=False, the default for the paginated helpers, the remaining pages are still written. A successful materialization with a failedfetch_attemptscheck can mean the partition is missing one page.
Malformed payloads. A body can pass the terminator check and still not parse. A complete document followed by a repeat of its own last bytes ends with the same terminator as an intact one. A 2xx body that does not parse raises MalformedJSONResponseError. Like TruncatedResponseError, it subclasses TransientHTTPError, so the request is retried. HttpSession.request_events does not raise it, because detecting it requires a full parse of every response. A source's response validator raises it when that validator already parses the body.
Source modules that call httpx directly bypass the guard. New sources should use the session helpers.
Asset Groups and Tags
Source observability assets are grouped sources. Raw fetch assets are grouped raw. Transformed assets are grouped transformed. Composite assets are grouped composite. Consolidated assets are grouped consolidated.
Groups are visible in the Dagster UI's asset catalog and can be used to filter and select assets in bulk.
Every ingestion asset also has an oem tag for per-OEM filtering:
oem: audi
Partitioning
Ingestion assets use a DailyPartitionsDefinition with end_offset=1, so today is always a valid
partition and an asset can be materialized on the day it is due.
An asset with an end_date uses end_offset=0. The end date already sets the last partition, so
the offset does not change the partition keys. It does change the partition definition's
description. The Dagster UI compares that description when it decides which upstreams to check
before a launch: it warns about missing upstream partitions only for upstreams whose description
matches the launched asset's. An upstream with an end date therefore has a different description
and is not checked, and the downstream's partitions after that date launch without an "Upstream
data missing" warning. Downstream mappings still need allow_nonexistent_upstream_partitions,
which lets those partitions execute.
Assets collected through a dealer's OMS session partition by dealer and day, and a day's partition
keeps every collection made that day. A materialization appends rows with its run id as
collection_id. It never overwrites the partition. Readers take the latest collection per
partition, and the ordering service reads by collection_id
(ADR-016).
Request Throttling
Fan-out HTTP fetches are paced by ai_http.Throttle, which is shared across every worker thread and retry in a fan-out. It enforces steady-state spacing (min_interval seconds between requests across the pool) and a shared 429 penalty window (penalize() pauses the whole pool).
By default the throttle keeps its state in-process, so it coordinates threads within a single Dagster run. When multiple concurrent runs call the same origin or share one API budget, a per-run throttle cannot see the combined request rate. Throttle accepts a pluggable backend: the default LocalBackend or a DynamoThrottleBackend that stores the pacing state in a shared DynamoDB item. Every process using the same key shares that state.
Distributed throttle
Construct a cross-process throttle with ai_http.distributed_throttle(key, min_interval, *, local_min_interval=None, meter=None). The key is a stable string identifying the shared budget:
- Per-credential limits → one key per credential. Zyte's rate limit is per API key, so
zyte_throttle(prefix)builds the key from a prefix (e.g."BMW_STOLO"→"zyte/bmw_stolo").ZyteSession/ZyteSessionFactorytake that samekey_prefixand use it for both the throttle and the<PREFIX>_ZYTE_API_KEYcredential lookup, so a call site's pacing and credential stay paired. Call sites sharing a prefix share one pool; distinct keys are paced independently. Only the prefix is used — the secret value never reaches the throttle store. - Per-origin limits where only concurrent runs of the same source contend →
"<oem>/<source>", e.g."gm/inventory".
Coordination is a single conditional UpdateItem per granted slot. No process holds a lock, so a crashed process cannot block later requests. Timestamps are wall-clock epoch seconds. NTP skew between tasks is far below any min_interval in use, and the conditional update compares stored state, so skew affects only exact spacing. Items include an expires_at TTL so idle keys are removed.
Enabling it. The backend is used only when DISTRIBUTED_THROTTLE_TABLE is set to the DynamoDB table name (distributed-throttle in prod). When unset, distributed_throttle() returns an in-process Throttle at local_min_interval. Leaving it unset disables the backend for local dev, unit tests, and deployments that have not opted in. DISTRIBUTED_THROTTLE_REGION is optional; unset lets boto3 resolve the region from AWS_REGION or task metadata. DYNAMODB_LOCAL_ENDPOINT is a local override. The run task's IAM role needs dynamodb:UpdateItem and dynamodb:GetItem on the table.
Contention vs. outage. Every grant and every failed conditional update writes to one partition key, so a large fan-out can make DynamoDB throttle those writes. The backend treats a DynamoDB throttling response as a backoff signal. It does not count that response toward the outage fallback.
Fail-open. DynamoDB availability is not a correctness dependency. On three consecutive non-throttling, non-conditional failures, the backend logs once and paces in-process at max(min_interval, local_min_interval) for a bounded window. After that window, it probes DynamoDB again. A DynamoDB outage costs cross-run coordination for that window but does not fail the run.
Metering. An injected RequestMeter records this process's request count, not the pool-wide aggregate. Read each run's meter summary to reconstruct each process's contribution.