Skip to main content

CompositeInventoryComponent

The composite tier merges the public inventory sources of one market into one row per vehicle per day. Its asset is <oem>/<market>/composite/public_inventory, and it reads the transformed tier. The component also writes the market's inferred sales; see Inferred Sales.

A composite partition is a daily snapshot of the market's inventory. It contains each vehicle that at least one source listed on the partition date. A vehicle that every source stopped listing is absent from that partition onward. The consolidated tier keeps such a vehicle for its whole history_days window instead.

Why the tier exists​

Public inventory feeds are incomplete in different ways. A feed can leave a vehicle out for a day and list it again the next day, and the dealer-website sources observe the same vehicles independently of the OEM feed. Merging the feeds gives a more complete view of the market than any single feed.

The consolidated tier already merges sources, but it runs inside each OEM's project and merges public and private feeds together, so its output is not public. The composite tier uses only public sources, so its output can be read by any consumer of the public data.

How a partition is built​

For partition date D:

  1. The VINs that at least one source group listed on D are collected. Only these VINs are read in the next step.
  2. Each source group reads its transformed asset over [D - history_days, D]. For each VIN, first_seen and last_seen are the dates of the earliest and latest partitions in that window that contain the VIN. The group keeps the VIN's row from its last_seen partition unchanged. A field that is null in that row is not filled from an earlier row.
  3. The groups are merged on vin with the consolidated tier's merge; see Merge Spec. Where several groups have a value for a column, the group listed first in the configuration wins, unless the column is in use_latest. A null is filled from the next group that has a value.
  4. first_seen, sources_present and sources_reporting_sold are resolved across the groups, and the per-group columns they are resolved from are dropped.
  5. Each group that sets is_on_lot, reported_arrival_date, dealer_arrival_date or custom_order reads every row of its window for the VINs of step 1, not only the latest one. The arrival dates and is_custom_order are resolved from those rows of every group; see Arrival date and Custom orders.

A source that did not list a vehicle on D still contributes its most recent row from the window, provided another source listed the vehicle on D.

To keep each source's value of a column, rename the column in each group's adapter, for example price to tvm_price, so that the groups do not share it. To merge one column from several groups into one, rename it to the same name in each group's adapter, for example price to dealer_price. The merged column takes the first non-null value in configuration order.

history_days defaults to 365. With that window, first_seen for a vehicle in stock for less than a year reaches back to its first listing. The window does not change which vehicles a partition contains.

Arrival date​

The composite writes three candidate arrival dates and resolves arrival_date from them. Each source group sets the options its source supports. The columns each option reads must be in fields or the group's extra_fields, under the names the group's adapter gives them.

ColumnGroup optionValue
inferred_arrival_dateis_on_lotThe earliest partition, across the groups, whose row satisfies is_on_lot
reported_arrival_datereported_arrival_dateThe earliest arrival date the sources report
dealer_arrival_datedealer_arrival_dateThe earliest stock date the dealer websites report

is_on_lot is a Polars-SQL boolean expression that is true when a row shows the vehicle on the dealer's lot, for example NOT in_transit, and false when the row shows it off the lot. A row whose expression is null shows neither. reported_arrival_date and dealer_arrival_date are Polars-SQL date expressions. arrival_date_strategy resolves each date from the rows of every group that sets its option:

StrategyResolved date
earliestThe earliest of the groups' dates

When the on-lot date is not an arrival​

The inferred arrival is the first day a group shows the vehicle on the lot. That is the arrival only if the group was already watching that lot before the vehicle came. When a dealer website is added to the scrape on 1 March, every vehicle already on that lot shows as on the lot for the first time on 1 March, though none of them arrived that day.

collection_scope names the column that identifies the lot a row comes from, such as a site id or a dealer code. The first day a value of that column appears in the window is the day collection of that lot began. A vehicle on the lot that day may have arrived earlier, so that day gives it no inferred arrival. A vehicle first on the lot on a later day did arrive then. A row whose value is null gives no inferred arrival either.

A group without collection_scope treats its first day in the window as the day collection of every lot began: the window start, or the day the source began collecting when that is later. A lot the group starts collecting later is then not recognized, and its existing stock reads as arrivals on that day.

Each row is checked against its own value, so a vehicle that moves to another dealer or site is checked against the lot that showed it on the lot.

inferred_arrival_date is null when no on-lot row on the earliest on-lot partition across the groups is an arrival, even when another group gives a later on-lot date.

Reported and dealer dates the on-lot status contradicts​

A reported or dealer date on or before a partition in which any group's row shows the vehicle off the lot is null. is_on_lot must be false in that row, not null. Only partitions before the vehicle's earliest on-lot date across the groups count, so a vehicle that a group shows in transit again after it arrived keeps its dates. A vehicle that the groups only ever showed in transit has no reported or dealer arrival before its last in-transit partition.

A reported or dealer date later than the vehicle's earliest on-lot date is wrong, because the vehicle was already on the lot. Where the earliest on-lot date is inferred_arrival_date, the reported or dealer date is capped to it. Where the earliest on-lot date is not an arrival, the reported or dealer date is null.

Resolving arrival_date​

arrival_date_priority lists the three candidate dates in order of preference, by default [inferred, reported, dealer]. arrival_date is the first of the listed dates that is not null. A candidate date left out of the list never sets arrival_date. arrival_date_source names the candidate date that set it, and is null when arrival_date is null. A reported or dealer date that was capped holds inferred_arrival_date, so when it sets arrival_date, arrival_date_source is inferred.

Reported sale​

A source group may set reported_sold, a Polars-SQL boolean expression that is true when a row reports the vehicle sold. sources_reporting_sold names the groups that listed the vehicle on the partition date and report it sold in that day's row. A group's earlier row is not read for the flag. is_reported_sold is true when sources_reporting_sold is not empty, and false otherwise, including in a market where no group sets reported_sold.

Custom orders​

A source group may set custom_order, a Polars-SQL boolean expression that is true when a row identifies the vehicle as a custom order. is_custom_order is true when any row of any group's window satisfies it, so a vehicle stays a custom order after the source stops marking it. It is false otherwise, including in a market where no group sets custom_order.

Output columns​

ColumnMeaning
_partition_dateThe partition date D
sources_presentNames of the source groups that listed the vehicle on D, in configuration order
sources_reporting_soldNames of the groups in sources_present whose row on D reports the vehicle sold; see Reported sale
<group>_last_seenOne column per source group: the last date in the group's window on which it listed the vehicle. Null when the group did not list the vehicle within its window
first_seenThe earliest date in the window on which any group listed the vehicle
inferred_arrival_date, reported_arrival_date, dealer_arrival_dateThe candidate arrival dates; see Arrival date
arrival_dateThe first candidate date in arrival_date_priority that is not null
arrival_date_sourceinferred, reported or dealer: the candidate that set arrival_date
is_reported_soldWhether sources_reporting_sold is not empty
is_custom_orderWhether the vehicle is a custom order; see Custom orders
source columnsEach column the source groups read, with their adapters applied, resolved as in step 3

The output has no column named after a source group, and no entity ID. A source group may not write a column with one of these names after its adapter; the component raises when one does. At least one group must write dealer_code.

Configuration​

The component takes oem and market from the directory of its file, defs/<oem>/<market>/composite/defs.yaml.

type: ai_dagster.components.CompositeInventoryComponent
attributes:
start_date: "2026-09-03"
fields: [vin, model_year, model_code, msrp]
use_latest: [msrp]
look_back_days: 14
look_back_threshold: 0.7
min_look_back_present_days: 2
look_ahead_days: 7
min_look_ahead_observed_days: 4
source_groups:
- name: nafta
source: nafta_inventory
extra_fields: [dealer_code]
- name: tvm
source: tvm_inventory
extra_fields: [price]
adapter:
rename: { price: tvm_price }
FieldDefaultDescription
start_daterequiredFirst partition. A source that starts later adds, from its first partition, the vehicles only it lists, so the daily counts of vehicles and inferred sales step up on that date. A group with collection_scope gives those vehicles no inferred arrival on that date; see When the on-lot date is not an arrival
fieldsrequiredColumns read from every source group
source_groupsrequiredListed in merge priority order. Where two groups hold a value for one field, the earlier group's value wins, unless the field is in use_latest. Group names must be unique
use_latestemptyFields that take the value from the group that most recently listed the vehicle with a non-null value for the field. Merge priority breaks a tie
arrival_date_strategyearliestHow each candidate arrival date is resolved from the groups' dates
arrival_date_priority[inferred, reported, dealer]The candidate arrival dates in order of preference for arrival_date. Each candidate date may appear only once
look_back_days, look_back_threshold, min_look_back_present_days, look_ahead_days, min_look_ahead_observed_daysrequiredSettings of <oem>/<market>/composite/public_inferred_sales, which every composite writes; see Inferred Sales
gaps, presence_sourcesempty, unsetInferred-sales settings; see Inferred Sales
ecs_vcpu, ecs_memory_gbunsetFargate vCPU and memory (GB) of the asset's step

Each source group takes these fields:

FieldDefaultDescription
namerequiredNames the group in sources_present and <group>_last_seen
sourcerequiredAsset name of the source, keyed <oem>/<market>/transformed/<source>
extra_fieldsemptyColumns read from this source in addition to fields
adapteremptyrename and Polars-SQL fields applied before history aggregation and merge, as in a consolidated source group
history_days365Number of days before D that the group reads
is_on_lotunsetPolars-SQL boolean expression that is true when a row shows the vehicle on the lot and false when it shows it off the lot. Unset, the group gives no on-lot date
collection_scopeunsetColumn that identifies the lot a row comes from, such as a site id or a dealer code. On the first day a value appears, collection of that lot began, so that day gives no inferred arrival; see When the on-lot date is not an arrival. Requires is_on_lot
reported_arrival_dateunsetPolars-SQL date expression giving the arrival date the source reports
dealer_arrival_dateunsetPolars-SQL date expression giving a dealer website's stock date
reported_soldunsetPolars-SQL boolean expression that is true when a row reports the vehicle sold
custom_orderunsetPolars-SQL boolean expression that is true when a row identifies the vehicle as a custom order

The following belong to the consolidated tier, and the component does none of them:

  • resolve foreign keys;
  • add the order-lock columns;
  • require the InventoryEntity common columns.