Reporting Layer — Architecture Decision Records
These records cover the reporting layer. Terms are defined under Terminology on the design page. The ordering platform keeps its own series at Ordering decisions, numbered separately.
| ADR | Decision |
|---|---|
| 001 | Shared components, configured per OEM |
| 002 | Vehicle performance metrics read consolidated; build deltas come from the private sources |
| 003 | Each OEM declares how an order matches an inventory record |
| 004 | Build delta calculation is shared; classification thresholds are configurable |
| 005 | Volume deviation is reported separately from build deltas |
| 006 | Metrics group by whether an order reached the OMS |
| 007 | Per-run and vehicle performance metrics have different cadences |
| 008 | Each OEM marks its own control vehicles |
| 009 | Cutover follows a comparison against the legacy output |
| 010 | Order sets are evaluated against reference order sets formed from consolidated inventory |
| 011 | Reference code counts are materialized daily by their own component; reference order sets are not stored |
| 012 | Statistics, set generators and evaluations are subclasses selected by type |
| 013 | Take-rate surprise compares order sets with demand take rates |
| 014 | An order set evaluation reads every input at the run's partition_date |
| 015 | An evaluation can select its attribute types by attribute weight |
| 016 | Configuration rarity scores each unit's weighted configuration against demand take rates |
| 017 | Set size z-score compares an order set's size with the dealer's reference windows |
ADR-001: Shared components, configured per OEM
Status: Proposed (2026-09-21)
Context
Reporting today is two legacy Dataform projects, one for BMW and one for Mercedes, sharing no code. They compute the same concepts by different means, so a change to a definition has to be made twice. Stellantis and Audi have no formal reporting pipeline and are queried ad hoc.
Decision
Reporting components are generic and live in packages/ai_dagster/src/ai_dagster/components/. Each
OEM project configures them under
projects/ai_<oem>/src/ai_<oem>/defs/<oem>/<market>/reporting/defs.yaml.
The boundary follows ConsolidatedComponent. Each metric and the calculation behind it is defined
in the component, once, for every OEM. Outputs conform to a shared schema so they are comparable
across OEMs. Configuration supplies what differs per OEM: which sources to read, which columns match
an order to a vehicle, which features are mutually exclusive, and how each metric classifies a build
delta.
An OEM whose source data needs its own handling may get code for that, as Audi's staged orders likely will.
Consequences
A metric is defined once. Adding an OEM is mostly a defs.yaml.
ADR-002: Vehicle performance metrics read consolidated; build deltas come from the private sources
Status: Proposed (2026-09-21)
Context
Reporting answers two questions: what did an ordering run produce, and how did the resulting vehicle
perform. The ordering event stream answers the first, and OrderingEventsComponent already writes
it to Iceberg for four OEMs.
Consolidated inventory merges public and private sources and computes days_on_lot,
inferred_sold_date and is_active, which is what vehicle performance metrics need.
A build delta needs something consolidated cannot give it. Consolidated resolves a vehicle's features by source priority, so where a public source outranks a private one, the merged features come from the public listing rather than from the OMS.
Decision
Per-run metrics read the ordering event stream. Vehicle performance metrics read consolidated inventory. Build deltas are computed from the private OMS sources, which list a vehicle's features as the OMS records them.
Consequences
Cross-dealer comparison becomes possible without a separate layer, because consolidated already holds the public sources.
Build deltas read each OEM's active private inventory directly, such as Stellantis dc_active and
Mercedes netstar. A build changes only before production. So build deltas need neither the sold
records nor a merge of a market's sources.
ADR-003: Each OEM declares how an order matches an inventory record
Status: Proposed (2026-09-21)
Context
An order has to be matched to the vehicle it became before that vehicle can be assessed. No single identifier serves every OEM.
Decision
Each OEM declares the columns that match an order to an inventory record. Orders matching no record are counted, and that count has an asset check.
This excludes cancelled staged orders, which are expected not to match anything.
Consequences
An order that never matches is counted rather than dropped silently. bmw-po-reporting added an
assertion for that failure after the fact.
ADR-004: Build delta calculation is shared; classification thresholds are configurable
Status: Proposed (2026-09-21)
Context
Multiple reporting metrics depend on assessing the differences between build versions on a given order. Each of these metrics may require its own classification or calculation depending on the type and magnitude of these differences.
Reported metrics based on differences between builds may include:
- OMS Curing: How much did the validation process change our recommended candidates?
- Dealer Acceptance KPIs: To what extent are dealers keeping or modifying our placed orders?
- Per-unit billing: Were our orders modified so much that they should no longer be invoiced in a per-unit billing scenario?
Types of differences are calculated across-the-board, but utilized separately per metric:
- Count of mutually exclusive features
replaced - Count of
addedorremovedfeatures - Types of features with the above changes
- Prices of features with the above changes
- Change in total MSRP for the whole build
Decision
Build deltas capture the raw differences and their magnitude, but stop short of classifying the change.
A given metric constructs its own classifications in defs.yaml.
Each OEM declares its mutually exclusive feature groups, so that a replacement inside a group counts once rather than as an addition and a removal. Changes outside any group count individually.
Consequences
There is no one-size-fits-all threshold for assessing whether a given set of build changes is sufficient for classifying as major or minor.
Adding a classification does not change how deltas are calculated.
Every consumer reads the same numbers for the same change, whichever two builds are compared.
ai_dagster.components.synthetic_candidates models the same constraint as MutuallyExclusiveGroup.
ADR-005: Volume deviation is reported separately from build deltas
Status: Proposed (2026-09-21)
Context
Volume optimization decides how many vehicles of a model to order for a dealer.
Two things are knowable about a model at a dealer: how many of our orders survived, and how much each surviving build changed. Neither implies the other.
A dealer's order volumes can change after we place them. During an allocation window, dealers may be able to cancel our orders and create their own. Further on in the production pipeline, changes can occur via rejection pools, dealer trades and supplementary allocations.
The reasoning behind a volume decision is not yet persisted. volume_optimizations stores counts
only, without the solver inputs, the per-model reasoning, or the models the optimizer considered and
allocated zero.
Decision
Volume deviation is its own dimension. It is counted on a daily basis per dealer, by model and model group.
Metrics covering additional volume optimizer data will follow when that data is available.
Consequences
Volume optimization yields an additional layer of dealer acceptance beyond build modifications.
Volume count is tracked across time. Volume acceptance is best assessed when an order's status locks in the order.
ADR-006: Metrics group by whether an order reached the OMS
Status: Proposed (2026-09-21)
Context
Of the four RunMode values, SIMULATED and DRY_RUN place no order and STAGE and EXECUTE
place one. UNPLACED_STATUSES draws the same line on the attempt side. Only a placed order reaches
a dealer and becomes a vehicle.
Decision
Per-run metrics cover every run mode. Vehicle performance metrics cover placed orders only. Run mode is a tag on the output rather than a split between outputs, so a mode can be reported on its own when that is wanted.
Consequences
Whether STAGE survives as a distinct run mode is being settled elsewhere. Reporting does not
depend on the answer.
Cancelled staged orders count toward volume deviation, but are excluded from vehicle performance metrics and asset check for unmatched orders.
ADR-007: Per-run and vehicle performance metrics have different cadences
Status: Proposed (2026-09-21)
Context
A run's own metrics are final once it finishes. A vehicle's performance changes daily, since days on lot advances and a sale can occur at any time.
Decision
Per-run metrics are computed from the partition the ordering-events sensor registers. Vehicle performance metrics are computed on a daily partition.
Consequences
The two use different partition definitions.
Per-run metrics are available as soon as a run finishes. Vehicle performance metrics settle over time.
ADR-008: Each OEM marks its own control vehicles
Status: Proposed (2026-09-21)
Context
Each OEM marks its control vehicles differently.
Decision
Each OEM marks its control vehicles, and reporting consumes that marking. Each vehicle's group follows from what happened to it: no order placed, order kept, or order changed.
Consequences
How a control set is constructed can change per OEM without touching reporting.
ADR-009: Cutover follows a comparison against the legacy output
Status: Proposed (2026-09-21)
Context
Reconciliation against the legacy system is a required capability. Running both systems in parallel delays decommissioning the legacy ones, so the comparison window should be kept short.
Decision
New output is compared against the legacy tables, read with BigQueryAssetBackfillComponent.
Cutover happens once the remaining differences are explained and the new output is judged at least
as accurate. Exact agreement is not required. The legacy ingestion is removed after cutover, and no
standing parity check is kept.
Consequences
Differences are found while both systems still run. No later output depends on a decommissioned system continuing to produce numbers.
ADR-010: Order sets are evaluated against reference order sets formed from consolidated inventory
Status: Proposed (2026-10-07)
Context
Candidate selection can return an order set in which every build is valid but the whole set is implausible, for example every vehicle in one paint color. The evaluation asks whether the order set is far outside what dealers order. It does not ask whether we outperform the dealer.
Possible evaluations include:
- concentration per attribute type;
- take-rate surprise;
- the probability that a dealer orders a candidate or an order set.
Each needs its own definition of what is typical.
Decision
An order set is compared with reference order sets. Each reference is configured as a population
of consolidated/inventory vehicles and a set generator that forms them into sets, for example
every dealer's arrivals within a 30-day window. Each evaluation that reads a reference names it.
Vehicles are compared by their (attribute_type, attribute_code) pairs, which describe both
recommended candidates and the vehicles in attributes/vehicles.
Orders we placed are left out of every reference, so that our own recommendations do not become part of what is typical.
The evaluation reads the recommendations of a run, not the OMS outcome. Its output is reported only; it does not stop a run.
Consequences
Runs of every mode are evaluated, including SIMULATED and DRY_RUN. Candidate selection can
therefore be evaluated before any order is placed. What counts as typical is configured in
defs.yaml, so changing it needs no code change. A reference can include only vehicles that
consolidated inventory holds, by default those seen in the last 365 days.
ADR-011: Reference code counts are computed in memory for each run; neither reference order sets nor their code counts are stored
Status: Proposed (2026-10-20)
Context
A reference depends on its configuration and the inventory partition, not on the run. The
evaluations depend only on how many vehicles of a set have each attribute code. Storing the
reference order sets themselves would write one row per vehicle and set, and a sliding_window
generator can put a vehicle in several sets.
The code counts could be materialized once a day in their own asset and read by every run of that
day. That asset would need its own backfill before older runs could be evaluated, a start_date
tied to reporting/placed_orders, and a partition for the run's partition_date before the run
could be evaluated. A run is evaluated when its partition is registered, so a missing daily
partition fails the evaluation. A day has few runs, and computing one day's code counts takes
seconds. An empirical evaluation compares an order set only with reference order sets of the same
match_keys values, so most sets of a daily asset would not be compared with any order set of a
run.
Decision
OrderSetEvaluationComponent lists the references and the evaluations. For each run, it reads
consolidated/inventory, attributes/vehicles and reporting/placed_orders at the run's
partition_date (ADR-014), forms the reference order sets in memory, and computes per
set:
- the number of the set's vehicles with each attribute code;
- per attribute type, the number of the set's vehicles with a code of that type, as
coded_units; - the set's number of vehicles, as
set_size.
A reference forms sets only for the values of the match keys that every evaluation reading it shares, and only for the values that the run's order sets have.
The reference order sets get the columns of model_columns, such as base_model_id, from
consolidated/models on the vehicle's model_id, as the order sets do. They do not read them from
inventory.
Neither the reference order sets nor their code counts are stored.
Consequences
- Adding an evaluation, a reference or an attribute type needs no backfill of another asset. Older
runs are evaluated with it by materializing their
order_set_evaluationspartitions again. - The reference of a run is computed again each time the run is evaluated. It can be rebuilt with
the same configuration from the
consolidated/inventory,attributes/vehiclesandreporting/placed_orderspartitions of the run'spartition_date, for as long as those are kept. order_set_evaluationsreads inventory, so its ECS task needs the memory of a reference's population.- The component checks the references and evaluations together when the code location loads. An evaluation that names an unknown reference fails the load.
- A statistic over a property of a vehicle other than its attribute codes, such as its price, needs the reference to compute that property.
- No time series of the reference is stored. Another consumer of the same reference, such as a model of the probability that a dealer orders a candidate, would compute it again or add an asset that stores it.
ADR-012: Statistics, set generators and evaluations are subclasses selected by type
Status: Proposed (2026-10-07)
Context
More evaluation types and more ways of forming reference order sets will be added. Each addition should be one class. A new evaluation type that reads an existing input should leave the components unchanged.
Decision
SetStatistic, SetGenerator and Evaluation are abstract base classes in
packages/ai_evaluation/src/ai_evaluation/order_sets/. Each implementation is a Pydantic model
with a type field. The implementations of each base class form a discriminated union, and the
type value in defs.yaml selects one. The scorers, normalizers and data loaders of
candidate_selection use the same pattern.
Each Evaluation declares the input it reads: the code counts of a reference, or take rates.
Evaluation.evaluate receives the order sets and its rows of that input. An empirical evaluation
combines a SetStatistic with size_ratio and tail. A model-based evaluation, such as take-rate
surprise, implements evaluate itself. Evaluation.problems reports what is wrong with the
schema of the input, and evaluate_order_sets raises for an evaluation with problems.
evaluate and evaluate_order_sets return a LazyFrame and collect nothing, so the query
planner sees all evaluations of a run as one plan. The order sets and each input are cached in that
plan, which the caller collects once. A check that needs the rows of an input, such as one code per
unit, is attached to the evaluation's output and raises when the plan is collected. The components
return their output lazily too, and the IO manager streams it to the table, so a failed check
writes nothing.
The evaluations live in their own package, ai_evaluation, rather than in ai_core, which holds
the shared entities and infrastructure. ai_evaluation depends on ai_core and imports no Dagster.
Its only other dependencies are polars and pydantic. OrderSetEvaluationComponent in ai_dagster
defines the asset and computes it with ai_evaluation.order_sets. Building order sets from the ordering
events is in ai_dagster.components.order_set_evaluation, because it depends on the schema of the
recommendation event.
Consequences
An unknown type makes the code location fail to load. An evaluation whose input lacks a column it
reads raises when a run is evaluated, and so does one whose input has no partition on the run's
partition_date (ADR-014).
An application outside Dagster, such as candidate selection, can compute an evaluation by
depending on ai_evaluation.
It could then evaluate the order sets of an ordering run before any order is placed, once it puts
its candidates in the long form that code_counts reads.
ADR-013: Take-rate surprise compares order sets with demand take rates
Status: Proposed (2026-10-12)
Context
One possible source of the expected mix for take-rate surprise is the dealer's arrivals of the
order set's base model, shrunk toward all dealers' arrivals of that base model.
output/demand_take_rates already holds each dealer's expected mix per model. The mix is the
sales take rates blended across the dealer, its neighborhood and the country. Candidate selection
scores candidates against these rates.
Decision
Take-rate surprise reads demand take rates. The expected code probabilities of an order set are the take rates of its units' dealer and model, averaged over its units. The test treats these rates as certain: an order set's code counts are tested against a multinomial distribution with them as probabilities. It does not read a reference.
Consequences
- A dealer's expected mix has one definition, shared with candidate selection.
- The evaluation measures how far an order set deviates from the rates candidate selection aims for. An order set that is typical of demand but unlike what the dealer stocks is not flagged.
- Candidate selection treats the take rates as certain, so the evaluation does too and allows nothing for their estimation error. Larger order sets are therefore flagged for smaller deviations.
- A code without a take rate has probability 0, so an order set with one gets the minimum p-value.
- Whether the take rates themselves are wrong is a different question. A test against the dealer's own sales history, allowing for their uncertainty, would answer it.
- An OEM can configure take-rate surprise without a reference.
ADR-014: An order set evaluation reads every input at the run's partition_date
Status: Proposed (2026-10-16)
Context
An ordering run selects its candidates from the warehouse partitions of one date, its
partition_date. The ordering service resolves that date from the run's requested as_of_date
before the run reads any data. It is the newest date within the run's lookback on which every table
the run reads is complete. It can be earlier than the day the run started. A simulated run can
request an as_of_date in the past.
The result of an evaluation that reads the latest partition of each input depends on when the evaluation runs. A partition that arrives later changes the result when the run is evaluated again. The evaluation can also read other dealers, models, features or take rates than the run did.
Decision
order_set_evaluations reads the run's partition_date from ordering/runs, as the
input partitions rule of the reporting layer requires.
It reads each of its inputs at that date:
consolidated/dealersandconsolidated/models;attributes/featuresandconsolidated/features;consolidated/inventory,attributes/vehiclesandreporting/placed_orders, from which it forms the reference order sets (ADR-011);output/demand_take_rates.
It raises when the run's partition_date is null, and when an input has no partition on that
date. reporting/placed_orders is the exception: its partition has no rows until the first order
is placed, and an empty partition cannot be told apart from a missing one.
Consequences
- Evaluating a run again gives the same rows, as long as its input partitions are unchanged.
- Take-rate surprise tests an order set against the take rates that candidate selection scored the run's candidates with.
- A run is evaluated when its partition is registered. If an input has no partition on the run's
partition_dateyet, the evaluation fails, and the run'sorder_set_evaluationspartition has to be materialized again once that partition exists. A run evaluated beforereporting/placed_ordershas its partition leaves no placed order out of its reference, and does not fail. - For a run whose
partition_date_sourceisrun_date,order_set_evaluationsraises when an input has no partition on the run'spartition_date. The run may have read an earlier partition.
ADR-015: An evaluation can select its attribute types by attribute weight
Status: Proposed (2026-10-27)
Context
An evaluation listed the attribute types it covers, and a reference listed the types it stores.
Stellantis evaluated PAINT_COLOR only. Candidate selection already assigns each attribute type a
weight per model in output/attribute_weights, and those weights change as the data changes.
Decision
An evaluation's attribute_types is a list or a selection from output/attribute_weights. A
selection takes the top_k heaviest types of the order set's model, or the types with at least
min_weight. The weights are read from the partition of the run's partition_date, the date of
the partitions the run selected its candidates from (ADR-014). A missing weights
partition raises; the evaluation does not read an older partition instead. A
reference stores the types its evaluations select, so it has no list of its own.
The weights are fitted on featurization/inventory, so the candidates are featurized by
VehicleFeaturizationSettings.featurize_candidates, the function that writes
featurization/synthetic. A candidate gets each <GROUP>_SET type, and it can have every type
that has a weight. The featurization reads no inventory column. A candidate has no inventory row,
so a type read from one would be NO_<type> on every candidate.
The output keeps one row per order set and attribute type, with attribute_weight. A score over an
order set's types is not stored.
Consequences
- The types evaluated follow the weights, so an evaluation's rows can change from one partition of
the weights to the next.
weights_partition_daterecords which partition a row's types came from. - A weighted score and an unweighted one can be computed from the rows. Where they diverge, the types with the most weight have p-values unlike those of the other selected types.
order_set_evaluationsreadsconsolidated/inventoryonly when an evaluation reads a reference.- An OEM sets the same
featurizationvalues inOrderSetEvaluationComponentandVehicleFeaturizationComponent. A difference featurizes candidates unlike the vehicles the weights, the take rates and the references come from. - When a model of a run's order sets has no weights, or the weights partition is missing, every evaluation of the run fails, including the evaluations that list their types.
- For each value of its restriction keys, such as a base model, a reference stores only the types
that the run's order sets with that value select, so its memory grows with
top_kand with the populations of the run's base models.
ADR-016: Configuration rarity scores each unit's weighted configuration against demand take rates
Status: Proposed (2026-10-27)
Context
Take-rate surprise tests one attribute type of an order set at a time. A candidate whose paint color, wheels and interior are each somewhat uncommon can be rare as a whole configuration, and no existing evaluation says which candidates are rare.
The product of a unit's take rates measures how common its configuration is. Its value depends on
how many codes each attribute type has, so one threshold does not fit every model. Many attribute
types restate others: a cross type restates its parts, and a binary option is counted again in its
<GROUP>_SET. Candidate selection weights each type by its attribute weight. The attribute
weights leave out such types or give them a lower weight.
Decision
Configuration rarity scores each unit by the sum of the logs of its codes' take rates, each times
the type's attribute weight when the evaluation selects its types by weight. A code without a
positive take rate gives -inf. The evaluation draws configurations from the same take rates, each
type's code independently, and estimates the probability that a drawn configuration scores at most
as high. A unit is rare when that probability is at most rare_mass.
An order set's result is the number of its rare units. Its p-value is the probability of at least
that many, when each unit is rare with the probability that a drawn configuration of its rate keys
is rare. The number of rare units then follows a Poisson binomial distribution. Each unit's result
is written to reporting/unit_rarities, and the run's rare units are tested together in its
materialization metadata.
Consequences
- Rarity is measured on the attribute types candidate selection weights, in the proportions it weights them. A rare code of a type without weight does not make a unit rare.
- A code the dealer's demand never takes makes its unit rare, whatever its other codes.
- The drawn configurations treat the attribute types as independent. Types that depend on each other and that the weights do not leave out, such as wheel diameter and tire type, can make a common configuration look rare.
- A unit's probability is estimated from
drawsdrawn configurations. With every weighted type, the configurations are too many to enumerate. - The baseline is units drawn from the take rates. Candidate selection does not draw from the take rates; it fills gaps between demand and supply. As with take-rate surprise (ADR-013), a large order set is flagged for a smaller deviation.
order_set_evaluationsandunit_raritiesare written by one materialization. A failed check or computation writes neither. Each table is written in its own commit,order_set_evaluationsfirst, so a failed write ofunit_raritiesleavesorder_set_evaluationswritten; a retry overwrites the run's partition of both.
ADR-017: Set size z-score compares an order set's size with the dealer's reference windows
Status: Proposed (2026-10-28)
Context
Volume optimization decides how many vehicles of a base model to order for a dealer. The existing evaluations judge which vehicles an order set holds and do not judge its size. A dealer's own history of arrivals per base model is the nearest baseline for that number.
A sliding-window reference whose step_days equals its width_days divides the population into
non-overlapping windows, so each window is one observation of the dealer's volume. A window without
a unit is an observation of size 0. The code counts that empirical evaluations read do not
include such a window, because a window without a unit has no code.
Decision
set_size_z_score reads the size of every reference order set of its reference, a new input
besides the code counts. A sliding-window reference gives a window without a unit a size of 0 for
each group_by value with a unit in another window. The evaluation standardizes the order set's
size by the mean and sample standard deviation of the reference order sets of the same
match_keys values. Its p-value is two-sided and empirical. With n such sets, of which r are
at least as far from their mean as the order set on either side, the p-value is (1 + r) / (n + 1).
As for every other evaluation, the population is the reference's, which leaves out the orders we placed.
Consequences
- A base model launched during the population period has windows of size 0 before its launch, which lower the mean.
- The population leaves out orders we placed, so windows after we started ordering hold only the dealer's own orders. A dealer whose orders we replaced looks like a dealer with less volume.
- An order set covers one run and a window covers
width_daysdays. The comparison is meaningful when a run orders about one window's volume, sowidth_daysshould match the number of days between the OEM's allocations. - With few windows, the p-value is coarse. Its smallest possible value is
1 / (baseline_sets + 1).