Skip to main content

Component Reference

Advanced configuration for ai_dagster component classes. See Code Structure for the component overview and basic YAML format.

Raw Tier Schema​

Every raw asset stores exactly five columns (produced by make_raw_row() in ai_core/schema.py):

ColumnTypeContent
_partition_datedatePartition date the fetch ran for
_fetched_atdatetimeUTC timestamp of the fetch
_request_urlstrURL called
_request_paramsstrJSON-encoded query params
_response_bodystrRaw JSON response string

TransformedEntityComponent reads _response_body and passes it to the extractor in the source registry.

ai_http / ai_core Utilities​

packages/ai_http/src/ai_http/ and packages/ai_core/src/ai_core/patterns.py provide reusable helpers and dataclasses for source modules. Source modules should use these helpers for retry, pagination, and request-attempt tracking.

  • http_get_events(url, ...) / http_post_events(url, ...) — make one request and yield one HttpRequestEvent for each attempt. A successful request yields a RESPONSE event. A failed request yields EXCEPTION events, with will_retry=True for attempts that will be retried and will_retry=False for the terminal attempt. Pass context to add labels to emitted request_params without sending them as query params or request body fields.
  • events_to_raw_fetch_result(events, *, url, partition_key, ...) — converts request events into a RawFetchResult. Each response event becomes one raw row. Each exception event becomes a failed FetchAttempt, with attempt_count, will_retry, and terminal_exception in its context.
  • paginated_events(url, *, first_page_params, extract_total, page_params, page_size, session_factory, ...) — fetches all pages from one paginated endpoint. Page 0 is fetched first. extract_total reads its body and determines the remaining page count. Follow-up pages can complete in any order when max_workers > 1.
  • parallel_paginated_events(items, *, page_params, extract_total, page_size, session_factory, ...) — fetches page 0 for each item, then schedules that item's follow-up pages on the same worker pool. context_for, page_context_for, and url_for attach item-specific labels, page-specific labels, and item-specific URLs.
  • json_path_total(path, *, default=0) — builds an extract_total function that reads the first integer at an ijson path, such as metadata.totalCount, without parsing the full response body.
  • paginated_get_raw_fetch_result(...) / paginated_post_raw_fetch_result(...) — fetch paginated HTTP responses and return an IPC-backed RawFetchResult.
  • drain_fetch_results(chunks) — concatenates a generator of RawFetchResult chunks into one lazy RawFetchResult. Chunks with no columns are dropped. Use it when a fetcher retrieves per-resource chunks but the component should return one result.
  • HttpxSessionFactory, AkamaiSessionFactory, and ZyteSessionFactory — create one session per worker thread and close all created sessions when the factory exits. Use with session_factory: or a helper that owns the factory context. If a caller may stop iterating early, close the generator with contextlib.closing() or a try/finally.
  • AkamaiSession — curl_cffi session for endpoints that need browser TLS impersonation. seed_url performs a warmup GET before the first real request. randomize_user_agent=False keeps the User-Agent and client-hint headers supplied by curl_cffi for a pinned browser impersonation.
  • ZyteSession — sends target requests through Zyte API. Pass key_prefix to read <PREFIX>_ZYTE_API_KEY and build the matching per-key throttle.
  • Throttle and distributed_throttle(key, min_interval, ...) — pace worker pools and shared API budgets. See Request Throttling for key selection, DynamoDB configuration, and failure behavior.
  • TransientHTTPError(message, *, status_code=None) / NonRetryableHTTPError(message, *, status_code=None) — HTTP exceptions that preserve a status code for request-attempt reporting.
  • extract_status_code(exc, resp=None) — reads a status code from TransientHTTPError, NonRetryableHTTPError, httpx.HTTPStatusError, or a fallback response.
  • FetchAttempt(ok, status_code, context) — records one request outcome for the fetch_attempts asset check. context should identify the request target, such as {"page": 4, "vehicle_id": "v123"}.
  • RawFetchResult(df, attempts) — return value for fetch() and fetch_with_upstream() on RawSourceComponent. df may be a pl.DataFrame or a pl.LazyFrame.
  • make_raw_row_frame(partition_date, request_url, request_params_list, response_body_list, *, fetched_at) — builds a pl.DataFrame matching RAW_ROW_SCHEMA from columnar inputs. Use it when a fetcher already has lists of request params and response bodies.

raise_for_incomplete_body checks only 2xx JSON responses. HTTP transports already reject a body that is shorter than its declared Content-Length; this guard covers a complete HTTP response whose JSON body is cut upstream and framed as complete. The check reads a bounded window at the start and end of the body. It rejects bodies that start as a JSON object or array but do not end with the matching terminator. It can miss a body cut at an inner object boundary, and it accepts bodies whose first or last significant character is outside the scan window.

max_pages is a data-quality guard. When an API reports more pages than the configured limit, page_count raises RuntimeError. Narrow the query to collect a complete page set.

Concurrency helpers yield events in completion order. By default, a job that ends with an exception yields its terminal EXCEPTION event and the remaining jobs continue. abort_on_failure stops scheduling new work after the first terminal failure. raise_on_failure also raises the terminal exception. failure_ratio stops scheduling once the failed/completed ratio reaches the configured value after min_samples jobs. A job that returns [] is skipped and does not count as completed.

For memory and wall-time considerations, including measured wins from the current ingestion patterns and deferred enhancements, see Performance Notes.

Source Overrides​

PPM_SOURCE_OVERRIDES is an optional env var pointing to a YAML file that substitutes local fixture data for live API calls. This allows full pipeline materializations without network access, and is the preferred approach for integration testing. See ai_core/source_overrides.py for the override resolution logic.

OEMComponent​

ComponentBase reads op_tags, ecs_vcpu and ecs_memory_gb from defs.yaml. OEMComponent adds start_date, and oem and market from the defs directory. A component builds its assets from these through the following calls:

CallReturns
component.daily_partitions_def()daily partitions from start_date, including today
component.op_tags_for(group)op_tags with the ECS task tags of the OEM, market and group
component.ecs_op_tags(group=..., oem=..., market=...)the same tags for a ComponentBase that names its own OEM and market
component.automation_condition()AutomationCondition.eager() when automation is eager, else None, on an EagerAutomation component
iceberg_asset_metadata(key)the Iceberg IO manager metadata of a daily asset whose table schema follows its output

RawSourceComponent​

Fetch contract​

fetch() and fetch_with_upstream() return a single RawFetchResult. The df field may be a pl.DataFrame or a pl.LazyFrame. Use drain_fetch_results, described above, when a fetcher produces data in natural per-resource batches.

fetch_attempts instrumentation​

The raw-tier fetch_attempts asset check (severity ERROR, blocking) reports per-request outcomes in its metadata so HTTP failures show up in the Dagster UI without parsing logs. To feed it, the source module's fetch_* function should:

  1. Catch TransientHTTPError and httpx.HTTPError per request.
  2. Build a FetchAttempt for each request. On success, set ok=True. On failure, set ok=False, status_code=extract_status_code(exc, resp) and exception=describe_exception(exc). Populate context with whatever fields identify the request (e.g. {"target": vehicle_id, "page": page_idx}). The context of a failed attempt also needs "terminal_exception": True when the request is not retried, or "will_retry": True when it is. The check counts a failure toward terminal_failure_rate, and groups it into terminal_failure_patterns or retried_failure_patterns, only by these two keys. events_to_raw_fetch_result sets both.
  3. Return RawFetchResult(df=df, attempts=attempts) from the fetcher, and have the component's fetch() return that value unchanged.

OEMs that have not been instrumented continue to work — the component wraps a bare pl.DataFrame return into a RawFetchResult automatically. The asset check reports tracking=uninstrumented in its metadata so operators can tell instrumented and uninstrumented sources apart.

The check's metadata also includes terminal_failure_patterns and retried_failure_patterns — failures grouped by (status_code, set of context keys) and capped at MAX_PATTERNS_IN_METADATA = 50 so the JSON payload stays small enough for the Dagster UI. An attempt's fields are its context plus two fields: exception, which holds FetchAttempt.exception as <class>: <message>, and exception_type, which holds the class name alone from FetchAttempt.exception_type. The metadata shows at most the first 200 characters of each string, and attempts are compared by those characters. Each pattern reports common_fields (the fields that are constant across the pattern) and variations (per-attempt values for the fields that differ). When every attempt in a pattern has the same exception, common_fields holds it once, for example "exception": "TooManyRedirects: Exceeded maximum allowed redirects.". A message often names the URL or a timing, so it differs between attempts and each entry of variations holds it; exception_type is then usually still in common_fields.

Concurrency​

Set concurrency: N in YAML when a raw component makes many parallel requests (e.g. per-vehicle detail fetches). The base class uses this to cap in-flight requests.

Upstream Source Dependencies​

When a raw fetch requires data from another raw asset (e.g. dealer IDs must be fetched first), override fetch_with_upstream() instead of fetch() in the RawSourceComponent subclass. The upstream asset key is declared via the upstream_dep attribute in YAML as a full asset key path (e.g., "audi/raw/scs_search"). The base class sets up the dependency automatically.

Declarative raw sources in ai_public​

The raw assets of the ai_public code location use ai_public.components.raw_source.PublicApiRawComponent. Its raw_source attribute names, as "<module>:<attribute>", a raw source that a source module in packages/ai_public_api exports. A raw source pairs the URL recorded as the source asset's base_url with the function that fetches the raw rows. Upstream parsing, fan-out and log lines belong to that function, not to a component.

A source with no configuration exports a constant:

# ai_public_api/kia/inventory.py
INVENTORY = UpstreamRawSource(url=INVENTORY_BASE_URL, fetch=fetch_inventory_for_vehicle_models)

A source that needs configuration, such as a brand, a locale or a dealer file, exports a factory: a function whose keyword-only parameters are that configuration and which returns a raw source. The params attribute holds its arguments. Loading validates them against the factory's signature, so an unknown name, a missing argument or a value outside a Literal or Field bound fails with the parameter it names:

# ai_public_api/stellantis/dealer_locator.py
def dealers(*, brand: StellantisBrand) -> RawSource: ...
type: ai_public.components.raw_source.PublicApiRawComponent
attributes:
raw_source: ai_public_api.stellantis.dealer_locator:dealers
params:
brand: jeep
source: dealer_locator
resource: jeep

A factory that declares a market parameter receives the market inferred from the defs/<oem>/<market>/<tier>/ path, which params must not set. A path relative to the defs.yaml is written "{{ defs_dir }}/<file>".

The raw source class must match upstream_dep: RawSource takes no upstream and a fetch called as fetch(partition_key=..., log=...); UpstreamRawSource takes one asset key and fetch(upstream, partition_key=..., log=...); MultiUpstreamRawSource takes a list and fetch(upstreams, partition_key=..., log=...). Loading fails on a mismatch.

A fetch that returns a RawFetchResultWithMetadata has its metadata recorded on the materialization. big_query_backfill names a (partition_key, client, log) function that reads a partition before today from BigQuery, through the big_query_resource_key resource, in place of the live fetch. It requires skip_past_partitions: false.

A subclass of PublicApiRawComponent adds only what the generic component cannot express: NissanDealerIdsRawComponent and NaftaInventoryVinRawComponent add their own asset checks. GmInventoryHistoricalRawComponent is the one raw component built on another base, BQHistoricalRawComponent, which reads past partitions from BigQuery through per-brand record hooks.

BigQuery-backed Sources​

When a raw component reads from BigQuery instead of an HTTP API, it obtains a BigQueryClientResource (from ai_dagster.resources) under the key named by its big_query_resource_key attribute, which defaults to BIG_QUERY_RESOURCE_KEY (exported from ai_dagster.components, value "big_query_resource").

A resource key resolves once per code location. A location serving a single OEM takes the default; one serving several declares a resource per GCP project and each component names its own, since the client's project qualifies the table it reads ({project}.{dataset}.{table}).

The resource is declared as a component in defs/resources/defs.yaml, supplying the GCP project:

type: ai_dagster.components.BigQueryResourceComponent
attributes:
resource_key: big_query_resource
project: "{{ env('<OEM>_GCP_PROJECT', 'ai-app-<oem>') }}"

ai_public serves every migrated OEM, so its keys are OEM-scoped. GM's BQ-backed inventory components set big_query_resource_key: gm_big_query_resource and the matching resource declares resource_key: gm_big_query_resource against ai-app-gm. A second BQ-backed OEM adds its own resource document rather than sharing GM's.

The client authenticates as described in GCP Access.

Tags​

RawSourceComponent accepts a tags: dict[str, str] attribute. The component merges this with {"oem": oem} on every asset it produces. OEMs with multiple brands use this to attach a brand tag:

tags:
brand: jeep

Each brand gets its own raw/transformed components. The consolidated tier unions them by listing all brand-scoped transformed asset keys as separate source_groups entries.

Retries​

By default, RawSourceComponent does not retry the raw asset's step. Its retry_policy attribute retries the step when the fetch raises or the blocking fetch_attempts check fails:

retry_policy:
max_retries: 1
delay_seconds: 1800

When a blocking check fails, Dagster fails the step without applying the step's retry policy. The component therefore raises RetryRequested itself when fetch_attempts fails and retries remain. Each retry fetches the whole partition again. The last attempt records the failed check and blocks downstream assets as it would without retries.

UserSessionSensorComponent​

Launches a collection from a private API that requires a browser sign-in.

Such a source cannot be collected on a cron. The session comes from somebody signing in, so a task started at a fixed hour may have nothing to authenticate with and no way to obtain one. This sensor reverses the order: it checks for a session on a cadence, and requests a run once a day, at the first tick that finds a session it has not tried.

type: ai_dagster.components.UserSessionSensorComponent
attributes:
name: <oem>_<market>_<name>_session
targets:
- <oem>/<market>/raw/<name>_latest
timezone: America/New_York # optional; the day starts at midnight in this timezone
poll_seconds: 1800 # optional; half an hour by default

The user_cookies resource the code location registers is what decides. It offers only complete sessions that Cookie Monster saw within the OEM's max_session_age_hours. When it produces a session, the tick requests a run. When it refuses, the tick skips and reports the refusal as its skip reason. A tick runs in the code server, so waiting costs nothing.

Targets must be unpartitioned, and the sensor raises on a partitioned one. A run request that names no partition cannot materialize a partitioned asset, and a collection of a source's current state has no partition to name.

That calls for a pair of assets. The unpartitioned collection holds what the source serves now. A daily-partitioned snapshot below it holds what the source served on each day, so that reprocessing a day reads the same bytes it read the first time. Give the snapshot AutomationCondition.eager().

A tick skips when the targets have materialized since midnight, or while a run that the sensor started for any of the targets is in progress. Otherwise it reads the freshest complete session and requests a run, unless it already requested one with that session today.

A run in progress has materialized nothing yet. Without the in-progress check, a sign-in recorded during a collection would start a second collection beside it.

The cursor records each request's time and a fingerprint of its session. The fingerprint is a hash of the cookies, because the Dagster UI shows a sensor's cursor.

A run with a session the source refuses writes nothing, so the collection stays due. When Cookie Monster records the next sign-in, the sensor requests another run with it. The sensor does not request a run with the refused session again that day. A run that fails for any other reason is retried the same way: with a new session, or with the same session on the next day.

Give the collection a FreshnessPolicy.time_window(fail_window=...) of several days. A stretch with no collection at all is then what alerts. That matches a source that runs only after somebody signs in.

An OMS that is read for more than one purpose has a sensor of its own (Allocation Windows). That sensor requests the OMS's collection by the rules above.

BaseTransformedComponent​

BaseTransformedComponent and TransformedEntitySpec define the settings and methods that every transformed asset shares. A component whose input is not a raw-tier asset, such as files read from S3, subclasses BaseTransformedComponent and parses its input in its own build_defs. TransformedEntityComponent and MbusaInventoryTransformedComponent subclass both classes.

TransformedEntitySpec holds the settings of one entity output:

  • key_fields
  • aggregations
  • tiebreak
  • acknowledged_discrepancies
  • row_group_size
  • row_count_check
  • row_filter
  • row_checks
  • unkeyed_check

A component with one output subclasses TransformedEntitySpec, so these fields are top-level attributes in its defs.yaml. A component with several outputs holds one spec per output.

MethodReturns
spec.asset_metadata(key)the Iceberg metadata: partition_expr, schema_update_mode, schema, key_fields, row_group_size
spec.check_specs(key, partitions_def)discrepancies, unkeyed_records when unkeyed_check is set, and one check per entry of row_checks
spec.asset_checks(key, partitions_def)the row_count_deviation check, or none when row_count_check is null
spec.finalize(records, ...)yields the unkeyed_records, discrepancies and row check results; returns the deduplicated entities with the partition column
component.asset_tags(entity)the tags of a transformed asset
component.discrepancy_severity(load_context)the severity of the discrepancies check

finalize is a generator that returns the entities, so the asset body calls it with yield from:

entities = yield from self.finalize(
records, asset_key=asset_key, severity=severity, partition_value=day
)
yield dg.Output(entities)

TransformedEntityComponent​

Registry module​

The extractor registry defaults to ai_<oem>.sources.registry. Where it lives outside the OEM's own project, name it:

registry_module: ai_public.sources.kia.registry

See Code Structure → Source Registry.

Unkeyed records​

Deduplication renders a null key component as the literal "NULL", so records stating none of their key fields share one id and merge into a single row. unkeyed_check emits an inline ERROR check, unkeyed_records, counting them before deduplication:

key_fields: [order_number]
unkeyed_check: true

Defaults to false. A record keyed on only part of a composite still identifies itself and does not count.

Row checks​

row_checks is a list of WARN checks on the output rows. Each check has a name and a where field. where holds a Polars-SQL condition.

A check counts the rows that meet its condition. It passes when no row meets the condition. Its metadata has the count and the key field values of up to five of those rows.

The asset materializes whether its row checks pass or fail. Use a row check to detect data that stops meeting an assumption the pipeline relies on.

Row filter​

row_filter is a Polars-SQL condition. The component evaluates it after deduplication and after the unkeyed_records and discrepancies checks. It drops each output row that does not meet the condition. Each row has _partition_date, the date of the partition being materialized:

row_filter: "sales_model_year >= YEAR(_partition_date) - 2"

A row for which the condition is null is dropped.

Tiebreak​

Selects a winning row when duplicate records conflict on non-aggregated columns. The strategies are described on TiebreakColumn in packages/ai_dagster/src/ai_dagster/components/transformed/base.py.

tiebreak:
- column: date_offer
strategy: max
- column: feature_refs
strategy: longest

Aggregations​

Collects distinct values from duplicate rows into a list column:

aggregations:
- output_name: vsapi_classifications
collect_distinct_from: vsapi_classification

Subclass hooks​

Source-specific behavior belongs on an OEM subclass, not on this component's YAML surface. Two hooks are available, in the order they run:

HookInputUse for
_transform(raw)raw rows, per sourceFiltering raw rows before extraction
_post_extract(lf)frame matching the entity schema, all sources unionedCorrecting extracted records before dedup

Both see only this asset's own raw sources. A correction that needs another asset belongs downstream — in an enriched tier asset, or in the consolidated derived_fields.

_post_extract sees the same columns the asset outputs, and runs before _clean, so its corrections are visible to the deduplication and the discrepancies check. A partition that extracted no records is routed through it as a zero-row frame with the derived entity schema, so a hook that adds or drops a column writes the same schema on both paths.

_clean collects its input several times (entity groups, tiebreak winners, acknowledged-field counts, field diffs, final output), so anything the hook adds to the plan is re-executed per collect. Keep per-row work on wide columns to a fixed number of passes, independent of the number of configured rules. Stellantis uses the hook to strip mutually exclusive option codes its inventory feeds report together — see public inventory source.

EnrichedInventoryComponent​

Emits <oem>/<market>/enriched/<source> — the transformed inventory asset with every element of a list column rebound to the package that installs it. Consolidated reads the enriched asset by naming tier: enriched on the source group.

A vehicle's feature references arrive as (option_code, parent_package_code) pairs, and the consolidated features entity is keyed on both: a member code appears once standalone and once per package that contains it. A feed that reports bare option codes therefore resolves every reference to the standalone row, losing the package that installs it.

type: ai_dagster.components.EnrichedInventoryComponent
attributes:
source: inventory_api_inventory
match_field: option_code
key_fields: [vin]
row_keys: [model_code, model_year, trim_identifier]
list_field: feature_refs # default
FieldDescription
sourceTransformed asset name; also the enriched asset's final key segment
match_fieldStruct field joined against the features entity's own column
key_fieldsRow identity, republished as metadata; must match the source's
row_keysRow-level columns scoping the features lookup to this vehicle's trim
list_fieldList-of-struct column to rewrite (default feature_refs)

key_fields matters because the asset is a row-for-row pass-through: consolidated reads key_fields metadata off whichever tier a source group names, and where it is absent falls back to intersecting the entity's key with the source's columns — which can widen the key and split one entity's history into several rows.

The component depends on <oem>/<market>/consolidated/features and resolves each reference this way:

  • A reference that already has a parent_package_code is preserved verbatim, so a feed that knows its own package membership stays authoritative.
  • Otherwise, if the vehicle has a code that appears as a parent_package_code in features for its row_keys, the reference gets that package as its parent.
  • Otherwise the reference keeps its null parent and resolves to the standalone row.

Where several packages on the vehicle contain the same member code, the parent is chosen lexicographically. The ambiguity comes from the source data. The rebuilt list is deduplicated on the whole reference: assigning a parent maps a member's standalone and packaged references onto the same pair, and a reference resolves to one feature id however many times the source reported it.

CompositeInventoryComponent​

See the CompositeInventoryComponent reference. The inferred sales it also writes are described in Inferred Sales.

ConsolidatedComponent​

See the ConsolidatedComponent reference for full documentation, including:

  • Source Groups — history_days, observation_date_col, filter_window_boundary, locked_when and customer_sold_when, gap fill
  • Merge Spec — match_on tiers, coalesce policies
  • Foreign Keys — single-value joins, left_col>ref_col syntax, list-mode dot-path joins

Cross-source coverage​

The inventory entity spec accepts an optional coverage_check, which checks that entities expected in two feeds appear in both. The check detects a feed gap, in which a vehicle disappears from one source without any error.

The check is emitted as a multi_asset_check over the transformed assets of the paired source groups (not the consolidated asset), so it runs as soon as those transformed assets materialize. Each pair names two source groups by their SourceGroup.name. Groups not named in any pair (e.g. a sales-history feed) are ignored, which is how the check is scoped to feeds expected to overlap.

inventory:
source_groups:
- { name: dc_active, source: inventory_from_dealerconnect_bq_active, priority: 1 }
- { name: public, source: inventory_api_inventory, priority: 2 }
- { name: dc_sold, source: inventory_from_dealerconnect_bq_sold, priority: 3 }
coverage_check:
match_on: [ vin ] # join key identifying an entity across both feeds
blocking: true # ERROR + skip consolidated on failure; false = WARN only
breakdown_by: [ brand, model_code ] # group the "missing from other feed" counts
pairs:
- left: public
right: dc_active # dc_sold is omitted → not compared
left_coverage: { min: 0.85 } # frac of public VINs present in dc_active
right_coverage: { min: 0.0 } # frac of dc_active VINs present in public
right_filter: # restrict a side to its comparable subset
- { column: order_type, op: not_in, values: [in_transit, in_production] }

Coverage is directional: left_coverage = |left ∩ right| / |left|, and symmetrically for the right. Each side's coverage must fall within its {min, max} band or the check fails for that side's transformed asset. An empty side scores 0.0 coverage, so a wholly missing feed fails. *_filter predicates compare the column as a string; not_in/ne retain null rows, in/eq drop them.

Tuning the directions. The two directions catch different gaps and have different sensitivities:

  • left_coverage (public → private) catches a private active-feed gap — vehicles the public scrape still sees that the private feed dropped. It needs no filter.
  • right_coverage (private → public) is confounded by units a private feed legitimately has but a public scrape never lists (in-transit / in-production, often 25–30% of allocation). Give it a meaningful floor only after a right_filter scopes the private side to retail-ready, on-lot units.

Blocking is a same-run gate. A failing blocking check raises DagsterAssetCheckFailedError and skips assets downstream of the checked transformed assets in the same run — so in a full pipeline materialization (raw → transformed → consolidated) it short-circuits consolidated/inventory. It does not gate a selective re-materialization of only consolidated/inventory, which reads transformed from Iceberg without re-running the check. Treat blocking as a strong guard on the daily pipeline, not an airtight cross-run invariant. Roll a new check out with blocking: false to observe coverage in the UI, calibrate the bands, then flip it on.