Code Structure
Package Layout
The repo is a uv workspace monorepo. Ingestion code is split across four layers:
packages/ai_core owns the OEM-agnostic, Dagster-free building blocks: entities, schema conversion, polars helpers, pydantic-to-frame record conversion and the Nessie catalog client. It imports dagster nowhere.
packages/ai_harmonize owns the Polars steps of the transformed, enriched and consolidated tiers:
clean, which reduces the transformed tier's duplicate rows to one row per entity and reports the discrepancies between them, andclean_output_schema, its output schema when a partition has no recordsconsolidate_entity, which consolidates one entity: it reduces each source group to one row per key over its window, merges the groups, and adds the entity's ids, references and derived fields, in the order its docstring statesconsolidate_inventoryandconsolidate_models, which run the steps ofconsolidate_entityand add the inventory and models entities' own steps: gap fill and the order lock for inventory, andbase_model_idandmodel_group_idfor models- the steps these functions run, which the composite tier and other callers also use:
aggregate_history,merge_with_tiers,fill_source_group_gaps,resolve_references,resolve_tiered_referencesandapply_sql_fields assign_parent_package_codes- the feature co-occurrence rules
Each function takes the values it reads as arguments. Where an argument is a list of items that each have several values, each item is a NamedTuple defined in the function's module, such as SqlField or Gap. The functions take no config models.
ai_harmonize/config/ holds the config models that a component and code outside Dagster, such as the ordering service, both parse from the same YAML: today the consolidated tier's specs and the models they are built from. They are plain pydantic models. A component declares a field of one of these types as Validated[T], which renders the field's templates and then validates the result as T. Each entity spec's consolidate method passes the spec's values to the entity's consolidation function, so the asset and the ordering service run the same steps on the same values. A config model that only a component parses stays in ai_dagster.
No other module of ai_harmonize imports ai_harmonize/config/, so the directory can move to its own package later. ai_harmonize depends on ai_core and imports dagster nowhere.
packages/ai_dagster owns the Dagster orchestration layer: the component base classes, the Iceberg IO manager, sensors, schedules, jobs, build_core_defs, the derived/ calculation tier and featurization. ai_dagster.components is the single import path and YAML type: prefix every project uses — flat, with no submodules. It depends on ai_core, ai_harmonize and ai_http. Nothing under any of them may import it.
packages/ai_http owns the HTTP transport helpers every source module fetches through — sessions, retry, pagination, throttling, and the two raw-tier result containers FetchAttempt and RawFetchResult. It depends on ai_core, never the reverse. Source modules import every HTTP name from ai_http.
Inside ai_dagster.derived.projection
The demand-projection tier splits by role rather than by stage alone:
| Module | Owns |
|---|---|
rules | the config models every rule type resolves from |
engine | the delta-fold machinery the remove and rule stages share |
codes | the two attribute_value delimiter layers, MULTI (,) and CROSS (|) |
rename, removes, proxy_estimates, assumed_rates, cross_model_estimates | one module per stage |
A delta frame holds rows whose rate columns are added to the base frame. Delta frames are folded one rule at a time, so each rule sees the previous one's effect. See Demand Projection for what the stages do and the order they run in.
projects/ai_<oem> (e.g. projects/ai_audi) owns everything OEM-specific: the source modules that call the APIs, the component subclasses that register them as assets, and the YAML config that instantiates everything.
Directory layout inside an OEM project
packages/ai_public_api/src/ai_public_api/audi/
├── onegraph.py ← API client + the OneGraph records
├── pss.py ← API client + PSSDealerEntity
├── catalog.py ← API client + CatalogFeatureEntity
└── common.py ← nested types and helpers the records share
projects/ai_audi/
└── src/ai_audi/
├── components/
│ └── raw.py ← RawSourceComponent subclasses (override fetch() and base_url)
├── sources/
│ ├── accessaudi.py ← BigQuery reader + AccessAudiDealerInventoryEntity
│ ├── vtp_inventory.py ← reshape for the archived inventory backfill
│ └── registry.py ← extractor registry: maps (source, resource, entity) → function
└── defs/
└── audi/<market>/
├── raw/ ← verbatim fetch components (one per source endpoint)
├── transformed/ ← entity extraction + deduplication
└── consolidated/ ← cross-source merge + entity refs
Component Classes
Components are importable from ai_dagster.components. Each class maps to one stage of the pipeline:
| Class | Assets emitted | Purpose |
|---|---|---|
RawSourceComponent | <oem>/<market>/sources/<source>_<resource> + <oem>/<market>/raw/<source>_<resource> | Emits the source observability node and the verbatim fetch asset; subclass to supply fetch() and base_url |
TransformedEntityComponent | <oem>/<market>/transformed/<name>_<entity> | Extracts a specific entity type from the verbatim raw asset via the OEM's extractor registry, then deduplicates rows by key group hashes and aggregates list fields. name is a free-form YAML field (e.g. a source name like onegraph, a pipeline name like nafta, or an endpoint like equipment_categories) |
CompositeInventoryComponent | <oem>/<market>/composite/public_inventory | Merges a market's public inventory sources into one row per vehicle listed on the partition date; see CompositeInventoryComponent |
CompositeInventoryComponent | <oem>/<market>/composite/public_inferred_sales | Dates each vehicle that left a market's composite inventory; see Inferred Sales |
ConsolidatedComponent | <oem>/<market>/consolidated/<entity> | Merges transformed records across all sources for an entity type; tracks per-source provenance via {name}_exists columns; resolves cross-entity references via foreign_keys |
RawSourceComponent is fully implemented — subclasses only override fetch() and base_url. TransformedEntityComponent is config-driven:
- the extractor function (looked up via the OEM registry) defines the output schema
key_fieldscontrol deduplicationaggregationsspecify list-collection columns
ConsolidatedComponent is a single component per OEM that merges transformed records across all sources for all four entity types and resolves cross-entity foreign-key references.
Source Modules
Source modules in ai_<oem>/sources/ are pure Python with no Dagster imports. They contain HTTP logic, data extraction, and the Pydantic entity subclasses for that source's API structure. This keeps them testable in isolation without a Dagster context.
Each source module owns the record for its API: canonical Python-named fields, with validation_alias / AliasPath declaring that API's naming. Parse functions call Record.model_validate(raw_dict).
# ai_public_api/audi/pss.py — one record per API, aliases declared on it
class PSSDealerEntity(BaseModel):
model_config = ConfigDict(populate_by_name=True)
pss_dealer_id: str = Field("", validation_alias="dealerId")
display_name: str | None = Field(
None, validation_alias=AliasPath("additionalData", "displayName")
)
...
def parse_dealer_entities(dealers: list[dict[str, Any]]) -> list[PSSDealerEntity]:
return [PSSDealerEntity.model_validate(d) for d in dealers]
One record per API, not one per entity type: two APIs that both return dealers describe different field sets, and a record shared between them carries columns neither can fill. Where a project keeps a base in entities.py with a subclass per source, that split is confined to sources whose records genuinely agree.
Component subclasses implement two hooks that the base build_defs calls:
class ScsInventoryRawComponent(RawSourceComponent):
@property
def base_url(self) -> str:
return scs.SEARCH_URL # single source of truth — defined in the source module
def fetch(self) -> RawFetchResult:
return scs.fetch_vehicles()
fetch() returns either a single RawFetchResult (the common case) or an Iterator[RawFetchResult] for streaming fetchers. Each RawFetchResult bundles the raw pl.DataFrame and the FetchAttempt records that produced it. See Component Reference → HTTP Utilities for the dataclass definitions and instrumentation pattern.
The base class adds these to the asset graph automatically — no build_defs override needed.
Source Registry
Each OEM project exposes an extractor registry in ai_<oem>/sources/registry.py. It maps (source, resource, entity) tuples to extraction functions, and is the single lookup point TransformedEntityComponent uses at materialization time:
# ai_audi/sources/registry.py
REGISTRY: dict[tuple[str, str, str], Callable[[str, str], pl.DataFrame]] = {
("pss", "dealers", "dealers"): pss.extract_dealers,
("scs", "search", "inventory"): scs.extract_inventory,
("onegraph", "carlines", "models"): onegraph.extract_models,
("catalog", "for_model", "features"): catalog.extract_features,
}
TransformedEntityComponent derives the module path from the OEM it serves, so a registry
in its own OEM's project needs no configuration. A project hosting another OEM's transformed
tier names the module instead:
registry_module: ai_public.sources.kia.registry
Keep one registry module per OEM. REGISTRY is keyed on (source, resource, entity), so a
tuple like ("inventory", "us", "inventory") would not stay distinct across two OEMs sharing
one dict.
Each extractor function lives in the corresponding source module (sources/scs.py, sources/onegraph.py, etc.) and has the signature (body: str, params: str) -> pl.DataFrame. Both arguments are raw JSON strings — the _response_body and _request_params columns from the raw Iceberg table, passed without pre-parsing. The function is responsible for:
- calling
orjson.loadsinternally - building entity records via Pydantic
- serializing any dict/nested-model fields
- returning a typed
pl.DataFrame
Module-level schema constants (pydantic_to_polars_schema and json_serialized_fields of the base entity class) are computed at import time and reused across calls.
If a (source, resource, entity) tuple is not found in the registry, the asset raises NotImplementedError with instructions on where to add it.
YAML Configuration
Components are instantiated from YAML files in defs/. Dagster's load_from_defs_folder discovers them automatically — no changes to definitions.py are needed when adding a new component.
Each file declares one component:
# defs/raw/defs.yaml (one multi-document file per tier; components separated by ---)
type: ai_audi.components.ScsInventoryRawComponent
attributes:
oem: audi
source: scs # API name
resource: search # endpoint/query within that API
market: us
auth: public
protocol: json_rest
description: "Audi US vehicle inventory search (SCS)"
start_date: "2024-01-01"
# defs/transformed/defs.yaml
# The raw/<source>_<resource> asset is auto-wired as a data input; no deps entry needed.
type: ai_dagster.components.TransformedEntityComponent
attributes:
oem: audi
market: us
name: onegraph # free-form label — forms transformed/onegraph_inventory
entity: inventory
start_date: "2024-01-01"
sources:
- [onegraph, search] # (source, resource) tuples looked up in the OEM registry
# defs/audi/us/consolidated/defs.yaml
type: ai_dagster.components.ConsolidatedComponent
attributes:
oem: audi
start_date: "2024-01-01"
dealers:
source_groups:
- name: pss
source: pss_dealers
priority: 1
merge:
match_on: [[mat_primary_code]]
key_fields: [mat_primary_code]
models:
source_groups:
- name: onegraph
source: onegraph_models
priority: 1
merge:
match_on: [[sales_model_year, model_code, trimline_id]]
key_fields: [sales_model_year, model_code, trimline_id]
features:
source_groups:
- name: catalog
source: catalog_features
priority: 1
merge:
match_on: [[model_catalog_id, pr3_id, parent_package_code]]
key_fields: [model_catalog_id, pr3_id, parent_package_code]
foreign_keys:
- entity_type: models
target: model_id
"on": [model_catalog_id]
inventory:
source_groups:
- name: scs
source: scs_inventory
priority: 1
merge:
match_on: [[vin]]
key_fields: [vin]
foreign_keys:
- entity_type: dealers
target: dealer_id
"on": [dealer_mat_primary_code>mat_primary_code]
- entity_type: models
target: model_id
"on": [model_catalog_id]
The type field is a fully-qualified Python class path. Switching from a base class to an OEM subclass is a one-line change in the YAML.
See the ConsolidatedComponent reference for full configuration options including source groups, gap fill, merge spec, and foreign keys.
Attribute validation
The ai_dagster component classes enforce required attributes via Pydantic field_validator definitions. If a required field is missing or empty, load_from_defs_folder raises a ValidationError. See the validators on the component classes in packages/ai_dagster/src/ai_dagster/components/ for the current set. Every OEM project's test suite validates its YAML configs by calling load_from_defs_folder — see the testing conventions in .claude/rules/testing.md.
Adding a New OEM
The platform uses a Dagster hybrid ECS deployment — code locations run as Fargate tasks in our AWS VPC, not on Dagster's managed serverless infrastructure. Adding a new OEM requires both Python scaffolding and AWS infrastructure changes in the same PR.
Run /scaffold-oem and Claude will handle the full process. For the complete checklist
of every required change — Python project, Dockerfile, Terraform (S3, ECR), CI workflow,
dagster_cloud.yaml — see Scaffolding a New OEM.
The Python implementation steps are:
- Scaffold via
/scaffold-oem(generates project skeleton,build.yaml,Dockerfile, PantsBUILDfiles, registers the resolve inpants.toml— theshared_source_targetsbackend picks up the new resolve automatically) - Create source modules in
src/ai_<oem>/sources/— pure Python, no Dagster imports - Create
src/ai_<oem>/sources/registry.pymapping(source, resource, entity)tuples to extractor functions - Create component subclasses in
src/ai_<oem>/components/raw.py - Add YAML in
src/ai_<oem>/defs/<oem>/<market>/raw/defs.yaml,defs/<oem>/<market>/transformed/defs.yaml, anddefs/<oem>/<market>/consolidated/defs.yaml - Run
pants generate-lockfiles --resolve=ai_<oem>anduv sync --directory projects/ai_<oem>