Skip to main content

Source Groups

Each entry in source_groups names one priority-ranked group of source assets. Records are merged in priority order — lower value wins on conflict (1 = highest priority).

source_groups:
- name: primary # label used in {name}_exists / {name}_last_seen columns
source: pss_dealers # last segment of the source asset key
priority: 1 # lower = higher precedence
history_days: 365 # days of partitions to scan (default 365)

Use sources: [key1, key2] instead of source: when multiple assets share the same schema and should be concatenated before the merge step.

tier​

Default: transformed, so a group resolves to <oem>/<market>/transformed/<source>.

Set tier: enriched when the source has been pre-processed by another asset before consolidation — a feed backfilled from its siblings, say — and so is keyed <oem>/<market>/enriched/<source>:

source_groups:
- name: code_guides
source: code_guides_features
tier: enriched
priority: 3

enriched is optional and sits between transformed and consolidated. The source segment does not have to be unique across tiers, but it must be unique across the source groups of one entity — it names the merge input.

observation_date_col​

By default, first_seen and last_seen are derived from the Dagster _partition_date. When a source carries its own authoritative observation timestamp, set observation_date_col to that column name and both date fields will be derived from it instead:

source_groups:
- name: dc_active
source: inventory_from_dealerconnect_bq_active
priority: 1
observation_date_col: last_seen_dc_active

allow_nonexistent_upstream_partitions​

Passed to Dagster's TimeWindowPartitionMapping. When true (default), the consolidated asset can materialize even if some upstream transformed partitions within the history window do not exist yet — Dagster treats missing partitions as empty rather than raising a dependency error. Set to false to require that every partition in the window is present before the downstream run executes.

filter_window_boundary​

Default: true on inventory source groups, false on dealers, models, and features.

Drops records whose first_seen equals the window start date (partition_date - history_days) — the oldest partition this group's window can see. Such a record was already present when the window opened, so its first_seen is the truncation date, not a real first observation.

Whether that warrants dropping the record depends on what reads first_seen. On inventory it is the arrival date days-on-lot subtracts from, so an artifactual value produces a wrong duration and the vehicle is dropped. On a catalog entity the date is informational while the record is not — dropping it removes an entity that inventory and features still reference.

Set it explicitly to override the entity default. The usual case is false on an inventory group whose observation_date_col supplies real observation dates, where the window edge means nothing:

source_groups:
- name: dc_active
source: inventory_from_dealerconnect_bq_active
priority: 1
observation_date_col: last_seen_dc_active
filter_window_boundary: false

locked_when and customer_sold_when​

Inventory only, and set together. Each is a Polars-SQL condition on one row of the source, which is one day's capture of an order:

  • locked_when is true once the OEM no longer accepts changes to the order. A null result counts as unlocked.
  • customer_sold_when is true while the order is sold to a customer.
source_groups:
- name: bmwusa_private
source: bmwusa_private_inventory
priority: 1
locked_when: TRY_CAST(event_id AS INT) >= 115
customer_sold_when: COALESCE(valid_customer_order, FALSE)

Every consolidated inventory has three columns that come from these conditions:

ColumnValue
locked_dateThe earliest _partition_date in the history window on which locked_when is true
customer_sold_dateThe earliest _partition_date in the history window on which customer_sold_when is true. Null when the latest capture with a non-null result of customer_sold_when has the result false
is_custom_orderTrue when the order was sold to a customer on or before the day it locked, or was sold and has not locked. False when it was never sold, or was sold after it locked

When several source groups set the conditions, each date is the earliest across them. All three columns are null on a vehicle that no row of such a source group describes. When no source group of the entity sets the conditions, they are null on every vehicle.

The dates are the earliest captures the source has. An order that locked or was sold before the source first captured it gets the date of its first capture.

The first locked capture sets locked_date for good. An order that later returns to an unlocked status keeps its locked_date.

A custom order is one whose build the customer could still choose. A stock order sold after its build locked is not a custom order, even though the OEM marks it as sold to a customer.

A spec with fields projects only the columns it lists. The columns that the conditions read must be in fields or in the group's extra_fields.

gap_fill​

Optional gap-fill for inventory source groups. When set, two boolean indicator columns (last_in_gap, first_in_gap) are added to the output and affected dates are estimated from the typical days-on-lot for each vehicle segment.

Two cases handled​

last_in_gap — last_seen falls within a gap: the vehicle was sold during the outage so its true sale date is unknown. last_seen is replaced with first_seen + agg(days_on_lot) over clean records in the same segment, clamped so the estimate never moves last_seen earlier than observed and never projects past the gap end date (the vehicle must have sold before data collection resumed).

first_in_gap — first_seen is the day after a gap ends and the vehicle is not still on lot: the vehicle first appeared in the feed when collection resumed, meaning it was delivered during the gap. first_seen is replaced with last_seen - agg(days_on_lot), clamped so the estimate never moves first_seen later than observed and never projects before the gap start date (the vehicle must have been delivered after the outage began).

The aggregation excludes last_in_gap records (unknown sale date), first_in_gap records (unknown delivery date), and still-on-lot vehicles (incomplete duration). Records with no clean reference rows in their group are left with their original dates; the indicator columns still mark them.

Configuration​

source_groups:
- name: primary
source: nafta_inventory
priority: 1
gap_fill:
gaps:
- start_date: "2025-03-01" # inclusive
end_date: "2025-03-14" # inclusive
group_by: [model, trim] # vehicle segments for days-on-lot aggregation
aggregation: mean # mean or median (default: mean)
FieldDescription
gapsOne or more inclusive date ranges where data collection stopped. At least one gap is required.
group_byColumns defining vehicle segments for the days-on-lot aggregation (e.g. model, trim). At least one column required.
aggregation"mean" (default) or "median" applied to (last_seen - first_seen) over clean records. median is more robust to outliers.
group_filterOptional restriction of gap detection to specific group_by value combinations (see below).

group_filter​

By default gap detection and filling applies to all group_by combinations. group_filter restricts it to specific combinations — useful when a gap only affected certain vehicle segments. Rows outside the filter are treated as clean records and may contribute to the reference aggregation.

Each inner list must have the same length as group_by and specifies exact column values to match:

gap_fill:
gaps:
- start_date: "2025-03-01"
end_date: "2025-03-14"
group_by: [model, trim]
group_filter:
- ["GLE", "AMG Line"] # only this model+trim combination is gap-filled
- ["GLC", "AMG Line"]