DealerConnect Source

The dealerconnect source reads pre-parsed PDF blob data from BigQuery tables populated by the ppm-ordering-services collection pipeline. It is specific to the Stellantis project (projects/ai_stellantis).

BigQuery Tables

Table pattern	Description	Partition
inventory_{dealerId}	In-stock vehicles with POC blobs	DAY (fetch_date)
sales_{dealerId}	Sold vehicles with POC blobs	None (filtered by fetch_date + sale_date)
code_guides	Trim/equipment spec blobs	None (filtered by fetch_date + blob_id dedup)
feature_catalog	Features available per model/trim (structured rows, not blobs)	None (filtered by exact DATE(fetch_date))

Dealer tables are discovered dynamically at runtime by listing all tables in the configured dataset that match the relevant prefix (inventory_ or sales_). There is no static dealer ID list — new dealers are picked up automatically as they are onboarded to the collection pipeline.

Required Environment Variables

Variable	Description	Example
STELLANTIS_GCP_PROJECT	GCP project ID containing the DealerConnect dataset (defaults to ai-app-stellantis)	my-gcp-project
GCP_CREDENTIALS	Service account JSON key content (production only; omit to use ADC locally)	{"type":"service_account",...}

The BigQuery dataset name defaults to dealerconnect and is set as a component YAML attribute (dataset:) on each DealerConnectRawSourceComponent subclass — it is not an environment variable.

Authentication

In local development, authenticate with Application Default Credentials (ADC):

gcloud auth application-default login

In production, set GCP_CREDENTIALS to the JSON content of a service account key (not a file path). The BigQueryResource registered in definitions.py reads this env var via gcp_credentials=dg.EnvVar("GCP_CREDENTIALS").

The service account requires the following BigQuery IAM roles on the dataset:

• roles/bigquery.dataViewer — read table data
• roles/bigquery.metadataViewer — list tables (required for dealer discovery)
• roles/bigquery.jobUser — run queries

Dagster Resource Key

The BigQuery client is injected via the standard BIG_QUERY_RESOURCE_KEY resource (value: "big_query_resource") exported from ai_core.components.bigquery. All DealerConnectRawSourceComponent subclasses declare this key in _required_resource_keys() and obtain the client via get_bq_resource(context) (exported from ai_core.components.bigquery).

Raw Asset Contract

Each raw asset row has the standard RAW_ROW_SCHEMA columns. The _response_body column contains a JSON-serialized BQ row dict with all original columns from the source table, including poc_raw_blob (for inventory/sales) or pdf_raw_blob (for code guides), which may be null if the collection pipeline failed to parse that PDF.

Feature Catalog

feature_catalog is a structured (non-blob) table: one row per (brand_code, model_code, model_year, package_code, feature_code), describing a feature available on a model/trim. It feeds both the features and models entities.

Wiring: sources/feature_catalog.py (BQ fetch + extractors), StellantisFeatureCatalogRawComponent (raw), StellantisFeatureCatalogTransformedComponent (transformed, two assets: feature_catalog_features + feature_catalog_models).

Raw query

BQHistoricalRawComponent._fetch_from_bq queries WHERE DATE(fetch_date) = @partition_date. The table holds many intra-day captures across multiple dealers, so duplicate rows per feature are kept verbatim in raw; the transformed tier's _clean collapses them to one row per entity key. The nested package_contents array is preserved in the raw _response_body.

Field mapping (transformed)

Entity field	Source
option_code	feature_code (package row) / package_contents[].code (member rows), prefix-stripped
feature_prefix	leading * (seat) or - (interior color) stripped off the code
lower_level_package	package_code
trim_identifier	package_code[-1]
model_code, model_year	direct
brand	brand_code via dealer_locator.BRAND_CODES (J→jeep, C→chrysler, …)
short_description	feature_name / content name
msrp	msrp (dropped on split seat/color rows — a combo price is not a per-half price)
category_* / subcategory_*	backfilled by joining equipment_categories (not in this table)

Code normalization

feature_code comes in two shapes: 3-char codes (e.g. ACG) and 7-char concatenated seat/trim codes (*SEAT/-COLOR, e.g. *E7/-X9). The concatenated form is split on / into one feature per half, and the leading */- prefix is stripped into feature_prefix (bare code in option_code) — matching how every other source (equipment_categories, configure, catalog_option, inventory) represents seat and interior-color options, so the codes join cross-source.

package_contents explosion

Each (non-split) row is emitted as a package feature (option_code = feature_code, content = the list of member codes). Every package_contents item is additionally emitted as its own member feature row (option_code = content.code) linking back via parent_option_codes. Member rows inherit the package's is_configurable value.

is_configurable rule

Deterministic (no LLM): a feature is configurable iff feature_group != "Package Content" and feature_code is not a 3-char admin/fleet code starting with "X9" or "5U" (e.g. X9A, 5U9). The length guard keeps the -X9 interior-color half of a concatenated seat/trim code configurable. Computed here for internal use (package-content inheritance) but not surfaced to consolidated — code_guides is the authority there (priority 3); configure fills the brands it does not cover.

Operational Notes

• NULL blobs are retained in the raw asset so collection failures are auditable. The transform-tier extractors skip NULL blobs gracefully and log a warning per skip.
• Missing dealer tables are warned and skipped — a partition can materialize successfully even if some dealers have no data for that date.
• Zero tables found is logged at info level and results in an empty DataFrame — this is valid for early backfill dates before any dealers were onboarded.