ConsolidatedComponent

A single component per OEM that merges transformed records across all sources for all four entity types (dealers, models, features, inventory) and resolves cross-entity foreign-key references. Each entity type is configured under its own key within the component's attributes.

Top-level attributes

Attribute	Required	Description
`oem`	yes	OEM identifier (e.g. `"audi"`)
`market`	yes	Market identifier (e.g. `"us"`)
`start_date`	yes	Earliest partition date for the `DailyPartitionsDefinition`
`dealers`	yes	Per-entity spec for the dealers entity
`models`	yes	Per-entity spec for the models entity
`features`	yes	Per-entity spec for the features entity
`inventory`	yes	Per-entity spec for the inventory entity

Per-entity spec

Each entity block (dealers, models, features, inventory) shares the same structure:

Field	Description
`source_groups`	Priority-ranked list of source groups — see Source Groups
`merge`	Cross-source matching and field resolution — see Merge Spec
`key_fields`	Fields used to identify and deduplicate records across sources; also used to compute the entity ID hash
`id_policy`	`"hash_all"` (default) hashes all `key_fields` together; `"first_non_empty"` hashes only the first non-empty value — useful when sources populate mutually exclusive identifiers (e.g. VIN vs order number). The `inventory` entity defaults to `"first_non_empty"`
`foreign_keys`	Cross-entity reference lookups — see Foreign Keys
`field_casts`	Polars-SQL CAST target types for merged columns — see Field Casts below
`derived_fields`	Polars-SQL expressions computing additional columns — see Derived Fields below

The inventory and models blocks each accept a few additional fields, covered below.

Common Schema

Every consolidated entity must carry the columns its entity type declares in ai_core.entities.COMMON_ENTITY_SCHEMAS, at the declared dtype. The merged frame is compared against that schema before the write, and a mismatch raises — naming each missing field or wrong dtype — so a partition fails rather than writing a table that differs from its peers. Each source group maps its native column names onto the common ones with adapter.fields — see Source Groups.

Identity and key fields stay out of the common schema: consolidated keys range from a single code to three-part composites, so each entity keys on its own key_fields.

Canonical values

Where a common field names one concept, its value carries that concept alone — a source value that packs several is narrowed before it is written.

model_code is the model line only; model year and trim belong to model_year and trim_code:

Source value	Canonical `model_code`
BMW NA `27XD`	`XD`
Stellantis Public `CUJ202510JLJS74A`	`JLJS74`
Audi Public `FU2AAY0_2026`	`FU2AAY`

Option codes follow the same rule: a type-related prefix (Mercedes 2:040, Stellantis -XY) is not part of the option's identity.

Field Casts

field_casts maps a merged column to a Polars-SQL CAST target type (e.g. BIGINT), applied right after the entity ID hash is computed and before foreign-key resolution runs. Use it when a column's native type varies by source, or when a foreign key joins two entities on a field that different transformed sources represent differently. A plain Polars join fails with a dtype-mismatch error if the two sides disagree, and this normalizes the type once per entity instead of once per source group:

models:
  # ...
  key_fields: [model_code, model_year]
  field_casts:
    model_year: BIGINT
inventory:
  # ...
  foreign_keys:
    - entity_type: models
      target: model_id
      "on": [model_code, model_year]
  field_casts:
    model_year: BIGINT

Every entity that carries the field — as a key field or as part of an on join condition — needs the same cast. The mechanism normalizes one entity's own merged frame, not the field across entities. enrich_hash_key casts key fields to Utf8 before hashing, so casting a clean numeric string to an integer type does not change the entity ID.

Derived Fields

derived_fields, available on every entity block, computes columns from a Polars-SQL expression, evaluated after foreign keys resolve and drop_fields runs. Each entry has a name and an expr. Expressions apply sequentially against the full row (every column, including earlier derived fields, is in scope), so a later expression can reference a column an earlier one produced:

inventory:
  # ...
  derived_fields:
    - name: inferred_sold_date
      expr: >-
        CASE
          WHEN private_sold_date IS NOT NULL
            THEN private_sold_date
          ELSE GREATEST(public_last_seen, private_last_seen)
        END
    - name: days_on_lot
      expr: >-
        CASE
          WHEN inferred_sold_date IS NOT NULL
            THEN DATEDIFF('day', arrival_date, inferred_sold_date)
          ELSE NULL
        END
    - name: is_sold_in_window
      expr: >-
        inferred_sold_date IS NOT NULL
        AND inferred_sold_date >= DATE('{partition_date}') - INTERVAL 365 DAYS
        AND inferred_sold_date < DATE('{partition_date}')
    - name: is_in_inventory
      expr: inferred_sold_date IS NULL

Any expr may contain {partition_date}, replaced at materialization time with the current partition key (e.g. 2025-04-15) — essential for sold-window flags. Expressions without {partition_date} are unaffected. Field names in {...} are Python str.format() placeholders — escape literal braces as {{ or }}.

Base Model ID

models.base_model_id_fields computes a year-independent base_model_id — a hash of the listed key fields — enabling cross-model-year grouping in downstream components. Omitting it (the default) computes no base_model_id.

models:
  # ...
  base_model_id_fields: [model_code, trim_identifier]
  base_model_id_aliases:
    - match: {model_code: "X5B"}
      canonical: {model_code: "X5"}

Field	Description
`base_model_id_fields`	Key-field subset hashed into `base_model_id`. `None` (default) disables the feature
`base_model_id_aliases`	Remaps the hash input for rows whose identifier changed between years (e.g. a renamed model code), so they still hash to their canonical counterpart. Only the hash input is remapped — real column values are unchanged

A blocking base_model_id_unique_per_model_year check fails if two distinct rows hash to the same (base_model_id, model_year).

Example

type: ai_dagster.components.ConsolidatedComponent
attributes:
  oem: audi
  market: us
  start_date: "2025-01-01"
  dealers:
    source_groups:
      - name: pss
        source: pss_dealers
        priority: 1
        history_days: 365
    merge:
      match_on: [[mat_primary_code]]
    key_fields: [mat_primary_code]
  models:
    source_groups:
      - name: onegraph
        source: onegraph_models
        priority: 1
        history_days: 365
    merge:
      match_on: [[sales_model_year, model_code, trimline_id]]
    key_fields: [sales_model_year, model_code, trimline_id]
  features:
    source_groups:
      - name: catalog
        source: catalog_features
        priority: 1
        history_days: 365
    merge:
      match_on: [[model_catalog_id, pr3_id, parent_package_code]]
    key_fields: [model_catalog_id, pr3_id, parent_package_code]
    foreign_keys:
      - entity_type: models
        target: model_id
        "on": [model_catalog_id]
  inventory:
    source_groups:
      - name: scs
        source: scs_inventory
        priority: 1
        history_days: 365
    merge:
      match_on: [[vin]]
    key_fields: [vin]
    foreign_keys:
      - entity_type: dealers
        target: dealer_id
        "on": [dealer_mat_primary_code>mat_primary_code]
      - entity_type: models
        target: model_id
        "on": [model_catalog_id]
      - entity_type: features
        target: feature_ids
        "on": [model_catalog_id, feature_refs.pr3_id, feature_refs.parent_package_code]

Top-level attributes​

Per-entity spec​

Common Schema​

Canonical values​

Field Casts​

Derived Fields​

Base Model ID​

Example​