Component Reference
Advanced configuration for ai_dagster component classes. See Code Structure for the component overview and basic YAML format.
Raw Tier Schema
Every raw asset stores exactly five columns (produced by make_raw_row() in ai_core/schema.py):
| Column | Type | Content |
|---|---|---|
_partition_date | date | Partition date the fetch ran for |
_fetched_at | datetime | UTC timestamp of the fetch |
_request_url | str | URL called |
_request_params | str | JSON-encoded query params |
_response_body | str | Raw JSON response string |
TransformedEntityComponent reads _response_body and passes it to the extractor in the source registry.
ai_http / ai_core Utilities
packages/ai_http/src/ai_http/ and packages/ai_core/src/ai_core/patterns.py provide reusable helpers and dataclasses for source modules. Source modules should use these helpers for retry, pagination, and request-attempt tracking.
http_get_events(url, ...)/http_post_events(url, ...)— make one request and yield oneHttpRequestEventfor each attempt. A successful request yields aRESPONSEevent. A failed request yieldsEXCEPTIONevents, withwill_retry=Truefor attempts that will be retried andwill_retry=Falsefor the terminal attempt. Passcontextto add labels to emittedrequest_paramswithout sending them as query params or request body fields.events_to_raw_fetch_result(events, *, url, partition_key, ...)— converts request events into aRawFetchResult. Each response event becomes one raw row. Each exception event becomes a failedFetchAttempt, withattempt_count,will_retry, andterminal_exceptionin its context.paginated_events(url, *, first_page_params, extract_total, page_params, page_size, session_factory, ...)— fetches all pages from one paginated endpoint. Page 0 is fetched first.extract_totalreads its body and determines the remaining page count. Follow-up pages can complete in any order whenmax_workers > 1.parallel_paginated_events(items, *, page_params, extract_total, page_size, session_factory, ...)— fetches page 0 for each item, then schedules that item's follow-up pages on the same worker pool.context_for,page_context_for, andurl_forattach item-specific labels, page-specific labels, and item-specific URLs.json_path_total(path, *, default=0)— builds anextract_totalfunction that reads the first integer at anijsonpath, such asmetadata.totalCount, without parsing the full response body.paginated_get_raw_fetch_result(...)/paginated_post_raw_fetch_result(...)— fetch paginated HTTP responses and return an IPC-backedRawFetchResult.drain_fetch_results(chunks)— concatenates a generator ofRawFetchResultchunks into one lazyRawFetchResult. Chunks with no columns are dropped. Use it when a fetcher retrieves per-resource chunks but the component should return one result.HttpxSessionFactory,AkamaiSessionFactory, andZyteSessionFactory— create one session per worker thread and close all created sessions when the factory exits. Usewith session_factory:or a helper that owns the factory context. If a caller may stop iterating early, close the generator withcontextlib.closing()or atry/finally.AkamaiSession—curl_cffisession for endpoints that need browser TLS impersonation.seed_urlperforms a warmup GET before the first real request.randomize_user_agent=Falsekeeps the User-Agent and client-hint headers supplied bycurl_cffifor a pinned browser impersonation.ZyteSession— sends target requests through Zyte API. Passkey_prefixto read<PREFIX>_ZYTE_API_KEYand build the matching per-key throttle.Throttleanddistributed_throttle(key, min_interval, ...)— pace worker pools and shared API budgets. See Request Throttling for key selection, DynamoDB configuration, and failure behavior.TransientHTTPError(message, *, status_code=None)/NonRetryableHTTPError(message, *, status_code=None)— HTTP exceptions that preserve a status code for request-attempt reporting.extract_status_code(exc, resp=None)— reads a status code fromTransientHTTPError,NonRetryableHTTPError,httpx.HTTPStatusError, or a fallback response.FetchAttempt(ok, status_code, context)— records one request outcome for thefetch_attemptsasset check.contextshould identify the request target, such as{"page": 4, "vehicle_id": "v123"}.RawFetchResult(df, attempts)— return value forfetch()andfetch_with_upstream()onRawSourceComponent.dfmay be apl.DataFrameor apl.LazyFrame.make_raw_row_frame(partition_date, request_url, request_params_list, response_body_list, *, fetched_at)— builds apl.DataFramematchingRAW_ROW_SCHEMAfrom columnar inputs. Use it when a fetcher already has lists of request params and response bodies.
raise_for_incomplete_body checks only 2xx JSON responses. HTTP transports already reject a body that is shorter than its declared Content-Length; this guard covers a complete HTTP response whose JSON body is cut upstream and framed as complete. The check reads a bounded window at the start and end of the body. It rejects bodies that start as a JSON object or array but do not end with the matching terminator. It can miss a body cut at an inner object boundary, and it accepts bodies whose first or last significant character is outside the scan window.
max_pages is a data-quality guard. When an API reports more pages than the configured limit, page_count raises RuntimeError. Narrow the query to collect a complete page set.
Concurrency helpers yield events in completion order. By default, a job that ends
with an exception yields its terminal EXCEPTION event and the remaining jobs
continue. abort_on_failure stops scheduling new work after the first
terminal failure. raise_on_failure also raises the terminal exception.
failure_ratio stops scheduling once the failed/completed ratio reaches the
configured value after min_samples jobs. A job that returns [] is skipped and
does not count as completed.
For memory and wall-time considerations, including measured wins from the current ingestion patterns and deferred enhancements, see Performance Notes.
Source Overrides
PPM_SOURCE_OVERRIDES is an optional env var pointing to a YAML file that substitutes local fixture data for live API calls. This allows full pipeline materializations without network access, and is the preferred approach for integration testing. See ai_core/source_overrides.py for the override resolution logic.
OEMComponent
ComponentBase reads op_tags, ecs_vcpu and ecs_memory_gb from defs.yaml.
OEMComponent adds start_date, and oem and market from the defs directory. A
component builds its assets from these through the following calls:
| Call | Returns |
|---|---|
component.daily_partitions_def() | daily partitions from start_date, including today |
component.op_tags_for(group) | op_tags with the ECS task tags of the OEM, market and group |
component.ecs_op_tags(group=..., oem=..., market=...) | the same tags for a ComponentBase that names its own OEM and market |
component.automation_condition() | AutomationCondition.eager() when automation is eager, else None, on an EagerAutomation component |
iceberg_asset_metadata(key) | the Iceberg IO manager metadata of a daily asset whose table schema follows its output |
RawSourceComponent
Fetch contract
fetch() and fetch_with_upstream() return a single RawFetchResult. The df field may be a pl.DataFrame or a pl.LazyFrame. Use drain_fetch_results, described above, when a fetcher produces data in natural per-resource batches.
fetch_attempts instrumentation
The raw-tier fetch_attempts asset check (severity ERROR, blocking) reports per-request outcomes in its metadata so HTTP failures show up in the Dagster UI without parsing logs. To feed it, the source module's fetch_* function should:
- Catch
TransientHTTPErrorandhttpx.HTTPErrorper request. - Build a
FetchAttemptfor each request. On success, setok=True. On failure, setok=False,status_code=extract_status_code(exc, resp)andexception=describe_exception(exc). Populatecontextwith whatever fields identify the request (e.g.{"target": vehicle_id, "page": page_idx}). The context of a failed attempt also needs"terminal_exception": Truewhen the request is not retried, or"will_retry": Truewhen it is. The check counts a failure towardterminal_failure_rate, and groups it intoterminal_failure_patternsorretried_failure_patterns, only by these two keys.events_to_raw_fetch_resultsets both. - Return
RawFetchResult(df=df, attempts=attempts)from the fetcher, and have the component'sfetch()return that value unchanged.
OEMs that have not been instrumented continue to work — the component wraps a bare pl.DataFrame return into a RawFetchResult automatically. The asset check reports tracking=uninstrumented in its metadata so operators can tell instrumented and uninstrumented sources apart.
The check's metadata also includes terminal_failure_patterns and retried_failure_patterns — failures grouped by (status_code, set of context keys) and capped at MAX_PATTERNS_IN_METADATA = 50 so the JSON payload stays small enough for the Dagster UI. An attempt's fields are its context plus two fields: exception, which holds FetchAttempt.exception as <class>: <message>, and exception_type, which holds the class name alone from FetchAttempt.exception_type. The metadata shows at most the first 200 characters of each string, and attempts are compared by those characters. Each pattern reports common_fields (the fields that are constant across the pattern) and variations (per-attempt values for the fields that differ). When every attempt in a pattern has the same exception, common_fields holds it once, for example "exception": "TooManyRedirects: Exceeded maximum allowed redirects.". A message often names the URL or a timing, so it differs between attempts and each entry of variations holds it; exception_type is then usually still in common_fields.
Concurrency
Set concurrency: N in YAML when a raw component makes many parallel requests (e.g. per-vehicle detail fetches). The base class uses this to cap in-flight requests.
Upstream Source Dependencies
When a raw fetch requires data from another raw asset (e.g. dealer IDs must be fetched first), override fetch_with_upstream() instead of fetch() in the RawSourceComponent subclass. The upstream asset key is declared via the upstream_dep attribute in YAML as a full asset key path (e.g., "audi/raw/scs_search"). The base class sets up the dependency automatically.
Declarative raw sources in ai_public
The raw assets of the ai_public code location use
ai_public.components.raw_source.PublicApiRawComponent. Its raw_source attribute
names, as "<module>:<attribute>", a raw source that a source module in
packages/ai_public_api exports. A raw source pairs the URL recorded as the source
asset's base_url with the function that fetches the raw rows. Upstream parsing,
fan-out and log lines belong to that function, not to a component.
A source with no configuration exports a constant:
# ai_public_api/kia/inventory.py
INVENTORY = UpstreamRawSource(url=INVENTORY_BASE_URL, fetch=fetch_inventory_for_vehicle_models)
A source that needs configuration, such as a brand, a locale or a dealer file, exports a
factory: a function whose keyword-only parameters are that configuration and which
returns a raw source. The params attribute holds its arguments. Loading validates them
against the factory's signature, so an unknown name, a missing argument or a value
outside a Literal or Field bound fails with the parameter it names:
# ai_public_api/stellantis/dealer_locator.py
def dealers(*, brand: StellantisBrand) -> RawSource: ...
type: ai_public.components.raw_source.PublicApiRawComponent
attributes:
raw_source: ai_public_api.stellantis.dealer_locator:dealers
params:
brand: jeep
source: dealer_locator
resource: jeep
A factory that declares a market parameter receives the market inferred from the
defs/<oem>/<market>/<tier>/ path, which params must not set. A path relative to the
defs.yaml is written "{{ defs_dir }}/<file>".
The raw source class must match upstream_dep: RawSource takes no upstream and a
fetch called as fetch(partition_key=..., log=...); UpstreamRawSource takes one asset
key and fetch(upstream, partition_key=..., log=...); MultiUpstreamRawSource takes a
list and fetch(upstreams, partition_key=..., log=...). Loading fails on a mismatch.
A fetch that returns a RawFetchResultWithMetadata has its metadata recorded on the
materialization. big_query_backfill names a (partition_key, client, log) function
that reads a partition before today from BigQuery, through the big_query_resource_key
resource, in place of the live fetch. It requires skip_past_partitions: false.
A subclass of PublicApiRawComponent adds only what the generic component cannot
express: NissanDealerIdsRawComponent and NaftaInventoryVinRawComponent add their
own asset checks. GmInventoryHistoricalRawComponent is the one raw component built on
another base, BQHistoricalRawComponent, which reads past partitions from BigQuery
through per-brand record hooks.
BigQuery-backed Sources
When a raw component reads from BigQuery instead of an HTTP API, it obtains a BigQueryClientResource (from ai_dagster.resources) under the key named by its big_query_resource_key attribute, which defaults to BIG_QUERY_RESOURCE_KEY (exported from ai_dagster.components, value "big_query_resource").
A resource key resolves once per code location. A location serving a single OEM takes the default; one serving several declares a resource per GCP project and each component names its own, since the client's project qualifies the table it reads ({project}.{dataset}.{table}).
The resource is declared as a component in defs/resources/defs.yaml, supplying the GCP project:
type: ai_dagster.components.BigQueryResourceComponent
attributes:
resource_key: big_query_resource
project: "{{ env('<OEM>_GCP_PROJECT', 'ai-app-<oem>') }}"
ai_public serves every migrated OEM, so its keys are OEM-scoped. GM's BQ-backed inventory
components set big_query_resource_key: gm_big_query_resource and the matching resource
declares resource_key: gm_big_query_resource against ai-app-gm. A second BQ-backed OEM
adds its own resource document rather than sharing GM's.
The client authenticates as described in GCP Access.
Tags
RawSourceComponent accepts a tags: dict[str, str] attribute. The component merges this with {"oem": oem} on every asset it produces. OEMs with multiple brands use this to attach a brand tag:
tags:
brand: jeep
Each brand gets its own raw/transformed components. The consolidated tier unions them by listing all brand-scoped transformed asset keys as separate source_groups entries.
Retries
By default, RawSourceComponent does not retry the raw asset's step. Its retry_policy
attribute retries the step when the fetch raises or the blocking fetch_attempts check
fails:
retry_policy:
max_retries: 1
delay_seconds: 1800
When a blocking check fails, Dagster fails the step without applying the step's retry
policy. The component therefore raises RetryRequested itself when fetch_attempts
fails and retries remain.
Each retry fetches the whole partition again. The last attempt records the failed check
and blocks downstream assets as it would without retries.
UserSessionSensorComponent
Launches a collection from a private API that requires a browser sign-in.
Such a source cannot be collected on a cron. The session comes from somebody signing in, so a task started at a fixed hour may have nothing to authenticate with and no way to obtain one. This sensor reverses the order: it checks for a session on a cadence, and requests a run once a day, at the first tick that finds a session it has not tried.
type: ai_dagster.components.UserSessionSensorComponent
attributes:
name: <oem>_<market>_<name>_session
targets:
- <oem>/<market>/raw/<name>_latest
timezone: America/New_York # optional; the day starts at midnight in this timezone
poll_seconds: 1800 # optional; half an hour by default
The user_cookies resource the code location registers is what decides. It
offers only complete sessions that Cookie Monster saw within the OEM's
max_session_age_hours. When it produces a session, the tick requests a run. When
it refuses, the tick skips and reports the refusal as its skip reason. A tick runs in the code
server, so waiting costs nothing.
Targets must be unpartitioned, and the sensor raises on a partitioned one. A run request that names no partition cannot materialize a partitioned asset, and a collection of a source's current state has no partition to name.
That calls for a pair of assets. The unpartitioned collection holds what the
source serves now. A daily-partitioned snapshot below it holds what the source
served on each day, so that reprocessing a day reads the same bytes it read the
first time. Give the snapshot AutomationCondition.eager().
A tick skips when the targets have materialized since midnight, or while a run that the sensor started for any of the targets is in progress. Otherwise it reads the freshest complete session and requests a run, unless it already requested one with that session today.
A run in progress has materialized nothing yet. Without the in-progress check, a sign-in recorded during a collection would start a second collection beside it.
The cursor records each request's time and a fingerprint of its session. The fingerprint is a hash of the cookies, because the Dagster UI shows a sensor's cursor.
A run with a session the source refuses writes nothing, so the collection stays due. When Cookie Monster records the next sign-in, the sensor requests another run with it. The sensor does not request a run with the refused session again that day. A run that fails for any other reason is retried the same way: with a new session, or with the same session on the next day.
Give the collection a FreshnessPolicy.time_window(fail_window=...) of several
days. A stretch with no collection at all is then what alerts. That matches a
source that runs only after somebody signs in.
An OMS that is read for more than one purpose has a sensor of its own (Allocation Windows). That sensor requests the OMS's collection by the rules above.
BaseTransformedComponent
BaseTransformedComponent and TransformedEntitySpec define the settings and methods
that every transformed asset shares. A component whose input is not a
raw-tier asset, such as files read from S3, subclasses BaseTransformedComponent and
parses its input in its own build_defs. TransformedEntityComponent and
MbusaInventoryTransformedComponent subclass both classes.
TransformedEntitySpec holds the settings of one entity output:
key_fieldsaggregationstiebreakacknowledged_discrepanciesrow_group_sizerow_count_checkrow_filterrow_checksunkeyed_check
A component with one output subclasses TransformedEntitySpec, so these fields are
top-level attributes in its defs.yaml. A component with several outputs holds one spec per output.
| Method | Returns |
|---|---|
spec.asset_metadata(key) | the Iceberg metadata: partition_expr, schema_update_mode, schema, key_fields, row_group_size |
spec.check_specs(key, partitions_def) | discrepancies, unkeyed_records when unkeyed_check is set, and one check per entry of row_checks |
spec.asset_checks(key, partitions_def) | the row_count_deviation check, or none when row_count_check is null |
spec.finalize(records, ...) | yields the unkeyed_records, discrepancies and row check results; returns the deduplicated entities with the partition column |
component.asset_tags(entity) | the tags of a transformed asset |
component.discrepancy_severity(load_context) | the severity of the discrepancies check |
finalize is a generator that returns the entities, so the asset body calls it with
yield from:
entities = yield from self.finalize(
records, asset_key=asset_key, severity=severity, partition_value=day
)
yield dg.Output(entities)
TransformedEntityComponent
Registry module
The extractor registry defaults to ai_<oem>.sources.registry. Where it lives outside the
OEM's own project, name it:
registry_module: ai_public.sources.kia.registry
See Code Structure → Source Registry.
Unkeyed records
Deduplication renders a null key component as the literal "NULL", so records stating none of
their key fields share one id and merge into a single row. unkeyed_check emits an inline
ERROR check, unkeyed_records, counting them before deduplication:
key_fields: [order_number]
unkeyed_check: true
Defaults to false. A record keyed on only part of a composite still identifies itself and
does not count.
Row checks
row_checks is a list of WARN checks on the output rows. Each check has a name and a
where field. where holds a Polars-SQL condition.
A check counts the rows that meet its condition. It passes when no row meets the condition. Its metadata has the count and the key field values of up to five of those rows.
The asset materializes whether its row checks pass or fail. Use a row check to detect data that stops meeting an assumption the pipeline relies on.
Row filter
row_filter is a Polars-SQL condition. The component evaluates it after deduplication and
after the unkeyed_records and discrepancies checks. It drops each output row that does
not meet the condition. Each row has _partition_date, the date of the partition being
materialized:
row_filter: "sales_model_year >= YEAR(_partition_date) - 2"
A row for which the condition is null is dropped.
Tiebreak
Selects a winning row when duplicate records conflict on non-aggregated columns. The
strategies are described on TiebreakColumn in
packages/ai_dagster/src/ai_dagster/components/transformed/base.py.
tiebreak:
- column: date_offer
strategy: max
- column: feature_refs
strategy: longest
Aggregations
Collects distinct values from duplicate rows into a list column:
aggregations:
- output_name: vsapi_classifications
collect_distinct_from: vsapi_classification
Subclass hooks
Source-specific behavior belongs on an OEM subclass, not on this component's YAML surface. Two hooks are available, in the order they run:
| Hook | Input | Use for |
|---|---|---|
_transform(raw) | raw rows, per source | Filtering raw rows before extraction |
_post_extract(lf) | frame matching the entity schema, all sources unioned | Correcting extracted records before dedup |
Both see only this asset's own raw sources. A correction that needs another asset belongs
downstream — in an enriched tier asset, or in the consolidated derived_fields.
_post_extract sees the same columns the asset outputs, and runs before _clean, so its
corrections are visible to the deduplication and the discrepancies check. A partition that
extracted no records is routed through it as a zero-row frame with the derived entity
schema, so a hook that adds or drops a column writes the same schema on both paths.
_clean collects its input several times (entity groups, tiebreak winners, acknowledged-field
counts, field diffs, final output), so anything the hook adds to the plan is re-executed per
collect. Keep per-row work on wide columns to a fixed number of passes, independent
of the number of configured rules. Stellantis uses the hook to strip mutually exclusive
option codes its inventory feeds report together — see public inventory source.
EnrichedInventoryComponent
Emits <oem>/<market>/enriched/<source> — the transformed inventory asset with every
element of a list column rebound to the package that installs it. Consolidated reads the
enriched asset by naming tier: enriched on the source group.
A vehicle's feature references arrive as (option_code, parent_package_code) pairs, and
the consolidated features entity is keyed on both: a member code appears once standalone
and once per package that contains it. A feed that reports bare option codes therefore
resolves every reference to the standalone row, losing the package that installs it.
type: ai_dagster.components.EnrichedInventoryComponent
attributes:
source: inventory_api_inventory
match_field: option_code
key_fields: [vin]
row_keys: [model_code, model_year, trim_identifier]
list_field: feature_refs # default
| Field | Description |
|---|---|
source | Transformed asset name; also the enriched asset's final key segment |
match_field | Struct field joined against the features entity's own column |
key_fields | Row identity, republished as metadata; must match the source's |
row_keys | Row-level columns scoping the features lookup to this vehicle's trim |
list_field | List-of-struct column to rewrite (default feature_refs) |
key_fields matters because the asset is a row-for-row pass-through: consolidated reads
key_fields metadata off whichever tier a source group names, and where it is absent
falls back to intersecting the entity's key with the source's columns — which can widen
the key and split one entity's history into several rows.
The component depends on <oem>/<market>/consolidated/features and resolves each
reference this way:
- A reference that already has a
parent_package_codeis preserved verbatim, so a feed that knows its own package membership stays authoritative. - Otherwise, if the vehicle has a code that appears as a
parent_package_codein features for itsrow_keys, the reference gets that package as its parent. - Otherwise the reference keeps its null parent and resolves to the standalone row.
Where several packages on the vehicle contain the same member code, the parent is chosen lexicographically. The ambiguity comes from the source data. The rebuilt list is deduplicated on the whole reference: assigning a parent maps a member's standalone and packaged references onto the same pair, and a reference resolves to one feature id however many times the source reported it.
CompositeInventoryComponent
See the CompositeInventoryComponent reference. The inferred sales it also writes are described in Inferred Sales.
ConsolidatedComponent
See the ConsolidatedComponent reference for full documentation, including:
- Source Groups —
history_days,observation_date_col,filter_window_boundary,locked_whenandcustomer_sold_when, gap fill - Merge Spec —
match_ontiers,coalescepolicies - Foreign Keys — single-value joins,
left_col>ref_colsyntax, list-mode dot-path joins
Cross-source coverage
The inventory entity spec accepts an optional coverage_check, which checks that
entities expected in two feeds appear in both. The check detects a feed gap, in which
a vehicle disappears from one source without any error.
The check is emitted as a multi_asset_check over the transformed assets of
the paired source groups (not the consolidated asset), so it runs as soon as those
transformed assets materialize. Each pair names two source groups by their
SourceGroup.name. Groups not named in any pair (e.g. a sales-history feed) are
ignored, which is how the check is scoped to feeds expected to overlap.
inventory:
source_groups:
- { name: dc_active, source: inventory_from_dealerconnect_bq_active, priority: 1 }
- { name: public, source: inventory_api_inventory, priority: 2 }
- { name: dc_sold, source: inventory_from_dealerconnect_bq_sold, priority: 3 }
coverage_check:
match_on: [ vin ] # join key identifying an entity across both feeds
blocking: true # ERROR + skip consolidated on failure; false = WARN only
breakdown_by: [ brand, model_code ] # group the "missing from other feed" counts
pairs:
- left: public
right: dc_active # dc_sold is omitted → not compared
left_coverage: { min: 0.85 } # frac of public VINs present in dc_active
right_coverage: { min: 0.0 } # frac of dc_active VINs present in public
right_filter: # restrict a side to its comparable subset
- { column: order_type, op: not_in, values: [in_transit, in_production] }
Coverage is directional: left_coverage = |left ∩ right| / |left|, and symmetrically
for the right. Each side's coverage must fall within its {min, max} band or the
check fails for that side's transformed asset. An empty side scores 0.0 coverage, so
a wholly missing feed fails. *_filter predicates compare
the column as a string; not_in/ne retain null rows, in/eq drop them.
Tuning the directions. The two directions catch different gaps and have different sensitivities:
left_coverage(public → private) catches a private active-feed gap — vehicles the public scrape still sees that the private feed dropped. It needs no filter.right_coverage(private → public) is confounded by units a private feed legitimately has but a public scrape never lists (in-transit / in-production, often 25–30% of allocation). Give it a meaningful floor only after aright_filterscopes the private side to retail-ready, on-lot units.
Blocking is a same-run gate. A failing blocking check raises
DagsterAssetCheckFailedError and skips assets downstream of the checked transformed
assets in the same run — so in a full pipeline materialization
(raw → transformed → consolidated) it short-circuits consolidated/inventory. It does
not gate a selective re-materialization of only consolidated/inventory, which
reads transformed from Iceberg without re-running the check. Treat blocking as a strong
guard on the daily pipeline, not an airtight cross-run invariant. Roll a new check out
with blocking: false to observe coverage in the UI, calibrate the bands, then flip it on.