CompositeInventoryComponent
The composite tier merges the public inventory sources of one market into one row per
vehicle per day. Its asset is <oem>/<market>/composite/public_inventory, and it reads the
transformed tier. The component also writes the market's inferred sales; see
Inferred Sales.
A composite partition is a daily snapshot of the market's inventory. It contains each
vehicle that at least one source listed on the partition date. A vehicle that every source
stopped listing is absent from that partition onward. The consolidated tier keeps such a
vehicle for its whole history_days window instead.
Why the tier exists
Public inventory feeds are incomplete in different ways. A feed can leave a vehicle out for a day and list it again the next day, and the dealer-website sources observe the same vehicles independently of the OEM feed. Merging the feeds gives a more complete view of the market than any single feed.
The consolidated tier already merges sources, but it runs inside each OEM's project and merges public and private feeds together, so its output is not public. The composite tier uses only public sources, so its output can be read by any consumer of the public data.
How a partition is built
For partition date D:
- The VINs that at least one source group listed on
Dare collected. Only these VINs are read in the next step. - Each source group reads its transformed asset over
[D - history_days, D]. For each VIN,first_seenandlast_seenare the dates of the earliest and latest partitions in that window that contain the VIN. The group keeps the VIN's row from itslast_seenpartition unchanged. A field that is null in that row is not filled from an earlier row. - The groups are merged on
vinwith the consolidated tier's merge; see Merge Spec. Where several groups have a value for a column, the group listed first in the configuration wins, unless the column is inuse_latest. A null is filled from the next group that has a value. first_seen,sources_presentandsources_reporting_soldare resolved across the groups, and the per-group columns they are resolved from are dropped.- Each group that sets
is_on_lot,reported_arrival_date,dealer_arrival_dateorcustom_orderreads every row of its window for the VINs of step 1, not only the latest one. The arrival dates andis_custom_orderare resolved from those rows of every group; see Arrival date and Custom orders.
A source that did not list a vehicle on D still contributes its most recent row from the
window, provided another source listed the vehicle on D.
To keep each source's value of a column, rename the column in each group's adapter,
for example price to tvm_price, so that the groups do not share it. To merge one
column from several groups into one, rename it to the same name in each group's
adapter, for example price to dealer_price. The merged column takes the first
non-null value in configuration order.
history_days defaults to 365. With that window, first_seen for a vehicle in stock for
less than a year reaches back to its first listing. The window does not change which
vehicles a partition contains.
Arrival date
The composite writes three candidate arrival dates and resolves arrival_date from them.
Each source group sets the options its source supports. The columns each option reads must
be in fields or the group's extra_fields, under the names the group's adapter gives
them.
| Column | Group option | Value |
|---|---|---|
inferred_arrival_date | is_on_lot | The earliest partition, across the groups, whose row satisfies is_on_lot |
reported_arrival_date | reported_arrival_date | The earliest arrival date the sources report |
dealer_arrival_date | dealer_arrival_date | The earliest stock date the dealer websites report |
is_on_lot is a Polars-SQL boolean expression that is true when a row shows the vehicle
on the dealer's lot, for example NOT in_transit, and false when the row shows it off the
lot. A row whose expression is null shows neither. reported_arrival_date and
dealer_arrival_date are Polars-SQL date expressions. arrival_date_strategy resolves
each date from the rows of every group that sets its option:
| Strategy | Resolved date |
|---|---|
earliest | The earliest of the groups' dates |
When the on-lot date is not an arrival
The inferred arrival is the first day a group shows the vehicle on the lot. That is the arrival only if the group was already watching that lot before the vehicle came. When a dealer website is added to the scrape on 1 March, every vehicle already on that lot shows as on the lot for the first time on 1 March, though none of them arrived that day.
collection_scope names the column that identifies the lot a row comes from, such as a
site id or a dealer code. The first day a value of that column appears in the window is
the day collection of that lot began. A vehicle on the lot that day may have arrived
earlier, so that day gives it no inferred arrival. A vehicle first on the lot on a later
day did arrive then. A row whose value is null gives no inferred arrival either.
A group without collection_scope treats its first day in the window as the day
collection of every lot began: the window start, or the day the source began collecting
when that is later. A lot the group starts collecting later is then not recognized, and its
existing stock reads as arrivals on that day.
Each row is checked against its own value, so a vehicle that moves to another dealer or site is checked against the lot that showed it on the lot.
inferred_arrival_date is null when no on-lot row on the earliest on-lot partition across
the groups is an arrival, even when another group gives a later on-lot date.
Reported and dealer dates the on-lot status contradicts
A reported or dealer date on or before a partition in which any group's row shows the
vehicle off the lot is null. is_on_lot must be false in that row, not null. Only
partitions before the vehicle's earliest on-lot date across the groups count, so a vehicle
that a group shows in transit again after it arrived keeps its dates. A vehicle that the
groups only ever showed in transit has no reported or dealer arrival before its last
in-transit partition.
A reported or dealer date later than the vehicle's earliest on-lot date is wrong, because
the vehicle was already on the lot. Where the earliest on-lot date is
inferred_arrival_date, the reported or dealer date is capped to it. Where the earliest
on-lot date is not an arrival, the reported or dealer date is null.
Resolving arrival_date
arrival_date_priority lists the three candidate dates in order of preference, by default
[inferred, reported, dealer]. arrival_date is the first of the listed dates that is
not null. A candidate date left out of the list never sets arrival_date.
arrival_date_source names the candidate date that set it, and is null when
arrival_date is null. A reported or dealer date that was capped holds
inferred_arrival_date, so when it sets arrival_date, arrival_date_source is
inferred.
Reported sale
A source group may set reported_sold, a Polars-SQL boolean expression that is true when
a row reports the vehicle sold. sources_reporting_sold names the groups that listed the
vehicle on the partition date and report it sold in that day's row. A group's earlier row
is not read for the flag. is_reported_sold is true when sources_reporting_sold is not
empty, and false otherwise, including in a market where no group sets reported_sold.
Custom orders
A source group may set custom_order, a Polars-SQL boolean expression that is true when a
row identifies the vehicle as a custom order. is_custom_order is true when any row of any
group's window satisfies it, so a vehicle stays a custom order after the source stops
marking it. It is false otherwise, including in a market where no group sets
custom_order.
Output columns
| Column | Meaning |
|---|---|
_partition_date | The partition date D |
sources_present | Names of the source groups that listed the vehicle on D, in configuration order |
sources_reporting_sold | Names of the groups in sources_present whose row on D reports the vehicle sold; see Reported sale |
<group>_last_seen | One column per source group: the last date in the group's window on which it listed the vehicle. Null when the group did not list the vehicle within its window |
first_seen | The earliest date in the window on which any group listed the vehicle |
inferred_arrival_date, reported_arrival_date, dealer_arrival_date | The candidate arrival dates; see Arrival date |
arrival_date | The first candidate date in arrival_date_priority that is not null |
arrival_date_source | inferred, reported or dealer: the candidate that set arrival_date |
is_reported_sold | Whether sources_reporting_sold is not empty |
is_custom_order | Whether the vehicle is a custom order; see Custom orders |
| source columns | Each column the source groups read, with their adapters applied, resolved as in step 3 |
The output has no column named after a source group, and no entity ID. A source group may
not write a column with one of these names after its adapter; the component raises when one
does. At least one group must write dealer_code.
Configuration
The component takes oem and market from the directory of its file,
defs/<oem>/<market>/composite/defs.yaml.
type: ai_dagster.components.CompositeInventoryComponent
attributes:
start_date: "2026-09-03"
fields: [vin, model_year, model_code, msrp]
use_latest: [msrp]
look_back_days: 14
look_back_threshold: 0.7
min_look_back_present_days: 2
look_ahead_days: 7
min_look_ahead_observed_days: 4
source_groups:
- name: nafta
source: nafta_inventory
extra_fields: [dealer_code]
- name: tvm
source: tvm_inventory
extra_fields: [price]
adapter:
rename: { price: tvm_price }
| Field | Default | Description |
|---|---|---|
start_date | required | First partition. A source that starts later adds, from its first partition, the vehicles only it lists, so the daily counts of vehicles and inferred sales step up on that date. A group with collection_scope gives those vehicles no inferred arrival on that date; see When the on-lot date is not an arrival |
fields | required | Columns read from every source group |
source_groups | required | Listed in merge priority order. Where two groups hold a value for one field, the earlier group's value wins, unless the field is in use_latest. Group names must be unique |
use_latest | empty | Fields that take the value from the group that most recently listed the vehicle with a non-null value for the field. Merge priority breaks a tie |
arrival_date_strategy | earliest | How each candidate arrival date is resolved from the groups' dates |
arrival_date_priority | [inferred, reported, dealer] | The candidate arrival dates in order of preference for arrival_date. Each candidate date may appear only once |
look_back_days, look_back_threshold, min_look_back_present_days, look_ahead_days, min_look_ahead_observed_days | required | Settings of <oem>/<market>/composite/public_inferred_sales, which every composite writes; see Inferred Sales |
gaps, presence_sources | empty, unset | Inferred-sales settings; see Inferred Sales |
ecs_vcpu, ecs_memory_gb | unset | Fargate vCPU and memory (GB) of the asset's step |
Each source group takes these fields:
| Field | Default | Description |
|---|---|---|
name | required | Names the group in sources_present and <group>_last_seen |
source | required | Asset name of the source, keyed <oem>/<market>/transformed/<source> |
extra_fields | empty | Columns read from this source in addition to fields |
adapter | empty | rename and Polars-SQL fields applied before history aggregation and merge, as in a consolidated source group |
history_days | 365 | Number of days before D that the group reads |
is_on_lot | unset | Polars-SQL boolean expression that is true when a row shows the vehicle on the lot and false when it shows it off the lot. Unset, the group gives no on-lot date |
collection_scope | unset | Column that identifies the lot a row comes from, such as a site id or a dealer code. On the first day a value appears, collection of that lot began, so that day gives no inferred arrival; see When the on-lot date is not an arrival. Requires is_on_lot |
reported_arrival_date | unset | Polars-SQL date expression giving the arrival date the source reports |
dealer_arrival_date | unset | Polars-SQL date expression giving a dealer website's stock date |
reported_sold | unset | Polars-SQL boolean expression that is true when a row reports the vehicle sold |
custom_order | unset | Polars-SQL boolean expression that is true when a row identifies the vehicle as a custom order |
The following belong to the consolidated tier, and the component does none of them:
- resolve foreign keys;
- add the order-lock columns;
- require the
InventoryEntitycommon columns.