Skip to main content

Reporting Layer — Architecture Decision Records

These records cover the reporting layer. Terms are defined under Terminology on the design page. The ordering platform keeps its own series at Ordering decisions, numbered separately.

ADRDecision
001Shared components, configured per OEM
002Vehicle performance metrics read consolidated; build deltas come from the private sources
003Each OEM declares how an order matches an inventory record
004Build delta calculation is shared; classification thresholds are configurable
005Volume deviation is reported separately from build deltas
006Metrics group by whether an order reached the OMS
007Per-run and vehicle performance metrics have different cadences
008Each OEM marks its own control vehicles
009Cutover follows a comparison against the legacy output
010Order sets are evaluated against reference order sets formed from consolidated inventory
011Reference code counts are materialized daily by their own component; reference order sets are not stored
012Statistics, set generators and evaluations are subclasses selected by type
013Take-rate surprise compares order sets with demand take rates
014An order set evaluation reads every input at the run's partition_date
015An evaluation can select its attribute types by attribute weight
016Configuration rarity scores each unit's weighted configuration against demand take rates
017Set size z-score compares an order set's size with the dealer's reference windows

ADR-001: Shared components, configured per OEM​

Status: Proposed (2026-09-21)

Context​

Reporting today is two legacy Dataform projects, one for BMW and one for Mercedes, sharing no code. They compute the same concepts by different means, so a change to a definition has to be made twice. Stellantis and Audi have no formal reporting pipeline and are queried ad hoc.

Decision​

Reporting components are generic and live in packages/ai_dagster/src/ai_dagster/components/. Each OEM project configures them under projects/ai_<oem>/src/ai_<oem>/defs/<oem>/<market>/reporting/defs.yaml.

The boundary follows ConsolidatedComponent. Each metric and the calculation behind it is defined in the component, once, for every OEM. Outputs conform to a shared schema so they are comparable across OEMs. Configuration supplies what differs per OEM: which sources to read, which columns match an order to a vehicle, which features are mutually exclusive, and how each metric classifies a build delta.

An OEM whose source data needs its own handling may get code for that, as Audi's staged orders likely will.

Consequences​

A metric is defined once. Adding an OEM is mostly a defs.yaml.

ADR-002: Vehicle performance metrics read consolidated; build deltas come from the private sources​

Status: Proposed (2026-09-21)

Context​

Reporting answers two questions: what did an ordering run produce, and how did the resulting vehicle perform. The ordering event stream answers the first, and OrderingEventsComponent already writes it to Iceberg for four OEMs.

Consolidated inventory merges public and private sources and computes days_on_lot, inferred_sold_date and is_active, which is what vehicle performance metrics need.

A build delta needs something consolidated cannot give it. Consolidated resolves a vehicle's features by source priority, so where a public source outranks a private one, the merged features come from the public listing rather than from the OMS.

Decision​

Per-run metrics read the ordering event stream. Vehicle performance metrics read consolidated inventory. Build deltas are computed from the private OMS sources, which list a vehicle's features as the OMS records them.

Consequences​

Cross-dealer comparison becomes possible without a separate layer, because consolidated already holds the public sources.

Build deltas read each OEM's active private inventory directly, such as Stellantis dc_active and Mercedes netstar. A build changes only before production. So build deltas need neither the sold records nor a merge of a market's sources.

ADR-003: Each OEM declares how an order matches an inventory record​

Status: Proposed (2026-09-21)

Context​

An order has to be matched to the vehicle it became before that vehicle can be assessed. No single identifier serves every OEM.

Decision​

Each OEM declares the columns that match an order to an inventory record. Orders matching no record are counted, and that count has an asset check.

This excludes cancelled staged orders, which are expected not to match anything.

Consequences​

An order that never matches is counted rather than dropped silently. bmw-po-reporting added an assertion for that failure after the fact.

ADR-004: Build delta calculation is shared; classification thresholds are configurable​

Status: Proposed (2026-09-21)

Context​

Multiple reporting metrics depend on assessing the differences between build versions on a given order. Each of these metrics may require its own classification or calculation depending on the type and magnitude of these differences.

Reported metrics based on differences between builds may include:

  • OMS Curing: How much did the validation process change our recommended candidates?
  • Dealer Acceptance KPIs: To what extent are dealers keeping or modifying our placed orders?
  • Per-unit billing: Were our orders modified so much that they should no longer be invoiced in a per-unit billing scenario?

Types of differences are calculated across-the-board, but utilized separately per metric:

  • Count of mutually exclusive features replaced
  • Count of added or removed features
  • Types of features with the above changes
  • Prices of features with the above changes
  • Change in total MSRP for the whole build

Decision​

Build deltas capture the raw differences and their magnitude, but stop short of classifying the change. A given metric constructs its own classifications in defs.yaml.

Each OEM declares its mutually exclusive feature groups, so that a replacement inside a group counts once rather than as an addition and a removal. Changes outside any group count individually.

Consequences​

There is no one-size-fits-all threshold for assessing whether a given set of build changes is sufficient for classifying as major or minor.

Adding a classification does not change how deltas are calculated.

Every consumer reads the same numbers for the same change, whichever two builds are compared.

ai_dagster.components.synthetic_candidates models the same constraint as MutuallyExclusiveGroup.

ADR-005: Volume deviation is reported separately from build deltas​

Status: Proposed (2026-09-21)

Context​

Volume optimization decides how many vehicles of a model to order for a dealer.

Two things are knowable about a model at a dealer: how many of our orders survived, and how much each surviving build changed. Neither implies the other.

A dealer's order volumes can change after we place them. During an allocation window, dealers may be able to cancel our orders and create their own. Further on in the production pipeline, changes can occur via rejection pools, dealer trades and supplementary allocations.

The reasoning behind a volume decision is not yet persisted. volume_optimizations stores counts only, without the solver inputs, the per-model reasoning, or the models the optimizer considered and allocated zero.

Decision​

Volume deviation is its own dimension. It is counted on a daily basis per dealer, by model and model group.

Metrics covering additional volume optimizer data will follow when that data is available.

Consequences​

Volume optimization yields an additional layer of dealer acceptance beyond build modifications.

Volume count is tracked across time. Volume acceptance is best assessed when an order's status locks in the order.

ADR-006: Metrics group by whether an order reached the OMS​

Status: Proposed (2026-09-21)

Context​

Of the four RunMode values, SIMULATED and DRY_RUN place no order and STAGE and EXECUTE place one. UNPLACED_STATUSES draws the same line on the attempt side. Only a placed order reaches a dealer and becomes a vehicle.

Decision​

Per-run metrics cover every run mode. Vehicle performance metrics cover placed orders only. Run mode is a tag on the output rather than a split between outputs, so a mode can be reported on its own when that is wanted.

Consequences​

Whether STAGE survives as a distinct run mode is being settled elsewhere. Reporting does not depend on the answer.

Cancelled staged orders count toward volume deviation, but are excluded from vehicle performance metrics and asset check for unmatched orders.

ADR-007: Per-run and vehicle performance metrics have different cadences​

Status: Proposed (2026-09-21)

Context​

A run's own metrics are final once it finishes. A vehicle's performance changes daily, since days on lot advances and a sale can occur at any time.

Decision​

Per-run metrics are computed from the partition the ordering-events sensor registers. Vehicle performance metrics are computed on a daily partition.

Consequences​

The two use different partition definitions.

Per-run metrics are available as soon as a run finishes. Vehicle performance metrics settle over time.

ADR-008: Each OEM marks its own control vehicles​

Status: Proposed (2026-09-21)

Context​

Each OEM marks its control vehicles differently.

Decision​

Each OEM marks its control vehicles, and reporting consumes that marking. Each vehicle's group follows from what happened to it: no order placed, order kept, or order changed.

Consequences​

How a control set is constructed can change per OEM without touching reporting.

ADR-009: Cutover follows a comparison against the legacy output​

Status: Proposed (2026-09-21)

Context​

Reconciliation against the legacy system is a required capability. Running both systems in parallel delays decommissioning the legacy ones, so the comparison window should be kept short.

Decision​

New output is compared against the legacy tables, read with BigQueryAssetBackfillComponent. Cutover happens once the remaining differences are explained and the new output is judged at least as accurate. Exact agreement is not required. The legacy ingestion is removed after cutover, and no standing parity check is kept.

Consequences​

Differences are found while both systems still run. No later output depends on a decommissioned system continuing to produce numbers.

ADR-010: Order sets are evaluated against reference order sets formed from consolidated inventory​

Status: Proposed (2026-10-07)

Context​

Candidate selection can return an order set in which every build is valid but the whole set is implausible, for example every vehicle in one paint color. The evaluation asks whether the order set is far outside what dealers order. It does not ask whether we outperform the dealer.

Possible evaluations include:

  • concentration per attribute type;
  • take-rate surprise;
  • the probability that a dealer orders a candidate or an order set.

Each needs its own definition of what is typical.

Decision​

An order set is compared with reference order sets. Each reference is configured as a population of consolidated/inventory vehicles and a set generator that forms them into sets, for example every dealer's arrivals within a 30-day window. Each evaluation that reads a reference names it. Vehicles are compared by their (attribute_type, attribute_code) pairs, which describe both recommended candidates and the vehicles in attributes/vehicles.

Orders we placed are left out of every reference, so that our own recommendations do not become part of what is typical.

The evaluation reads the recommendations of a run, not the OMS outcome. Its output is reported only; it does not stop a run.

Consequences​

Runs of every mode are evaluated, including SIMULATED and DRY_RUN. Candidate selection can therefore be evaluated before any order is placed. What counts as typical is configured in defs.yaml, so changing it needs no code change. A reference can include only vehicles that consolidated inventory holds, by default those seen in the last 365 days.

ADR-011: Reference code counts are computed in memory for each run; neither reference order sets nor their code counts are stored​

Status: Proposed (2026-10-20)

Context​

A reference depends on its configuration and the inventory partition, not on the run. The evaluations depend only on how many vehicles of a set have each attribute code. Storing the reference order sets themselves would write one row per vehicle and set, and a sliding_window generator can put a vehicle in several sets.

The code counts could be materialized once a day in their own asset and read by every run of that day. That asset would need its own backfill before older runs could be evaluated, a start_date tied to reporting/placed_orders, and a partition for the run's partition_date before the run could be evaluated. A run is evaluated when its partition is registered, so a missing daily partition fails the evaluation. A day has few runs, and computing one day's code counts takes seconds. An empirical evaluation compares an order set only with reference order sets of the same match_keys values, so most sets of a daily asset would not be compared with any order set of a run.

Decision​

OrderSetEvaluationComponent lists the references and the evaluations. For each run, it reads consolidated/inventory, attributes/vehicles and reporting/placed_orders at the run's partition_date (ADR-014), forms the reference order sets in memory, and computes per set:

  • the number of the set's vehicles with each attribute code;
  • per attribute type, the number of the set's vehicles with a code of that type, as coded_units;
  • the set's number of vehicles, as set_size.

A reference forms sets only for the values of the match keys that every evaluation reading it shares, and only for the values that the run's order sets have.

The reference order sets get the columns of model_columns, such as base_model_id, from consolidated/models on the vehicle's model_id, as the order sets do. They do not read them from inventory.

Neither the reference order sets nor their code counts are stored.

Consequences​

  • Adding an evaluation, a reference or an attribute type needs no backfill of another asset. Older runs are evaluated with it by materializing their order_set_evaluations partitions again.
  • The reference of a run is computed again each time the run is evaluated. It can be rebuilt with the same configuration from the consolidated/inventory, attributes/vehicles and reporting/placed_orders partitions of the run's partition_date, for as long as those are kept.
  • order_set_evaluations reads inventory, so its ECS task needs the memory of a reference's population.
  • The component checks the references and evaluations together when the code location loads. An evaluation that names an unknown reference fails the load.
  • A statistic over a property of a vehicle other than its attribute codes, such as its price, needs the reference to compute that property.
  • No time series of the reference is stored. Another consumer of the same reference, such as a model of the probability that a dealer orders a candidate, would compute it again or add an asset that stores it.

ADR-012: Statistics, set generators and evaluations are subclasses selected by type​

Status: Proposed (2026-10-07)

Context​

More evaluation types and more ways of forming reference order sets will be added. Each addition should be one class. A new evaluation type that reads an existing input should leave the components unchanged.

Decision​

SetStatistic, SetGenerator and Evaluation are abstract base classes in packages/ai_evaluation/src/ai_evaluation/order_sets/. Each implementation is a Pydantic model with a type field. The implementations of each base class form a discriminated union, and the type value in defs.yaml selects one. The scorers, normalizers and data loaders of candidate_selection use the same pattern.

Each Evaluation declares the input it reads: the code counts of a reference, or take rates. Evaluation.evaluate receives the order sets and its rows of that input. An empirical evaluation combines a SetStatistic with size_ratio and tail. A model-based evaluation, such as take-rate surprise, implements evaluate itself. Evaluation.problems reports what is wrong with the schema of the input, and evaluate_order_sets raises for an evaluation with problems.

evaluate and evaluate_order_sets return a LazyFrame and collect nothing, so the query planner sees all evaluations of a run as one plan. The order sets and each input are cached in that plan, which the caller collects once. A check that needs the rows of an input, such as one code per unit, is attached to the evaluation's output and raises when the plan is collected. The components return their output lazily too, and the IO manager streams it to the table, so a failed check writes nothing.

The evaluations live in their own package, ai_evaluation, rather than in ai_core, which holds the shared entities and infrastructure. ai_evaluation depends on ai_core and imports no Dagster. Its only other dependencies are polars and pydantic. OrderSetEvaluationComponent in ai_dagster defines the asset and computes it with ai_evaluation.order_sets. Building order sets from the ordering events is in ai_dagster.components.order_set_evaluation, because it depends on the schema of the recommendation event.

Consequences​

An unknown type makes the code location fail to load. An evaluation whose input lacks a column it reads raises when a run is evaluated, and so does one whose input has no partition on the run's partition_date (ADR-014). An application outside Dagster, such as candidate selection, can compute an evaluation by depending on ai_evaluation. It could then evaluate the order sets of an ordering run before any order is placed, once it puts its candidates in the long form that code_counts reads.

ADR-013: Take-rate surprise compares order sets with demand take rates​

Status: Proposed (2026-10-12)

Context​

One possible source of the expected mix for take-rate surprise is the dealer's arrivals of the order set's base model, shrunk toward all dealers' arrivals of that base model. output/demand_take_rates already holds each dealer's expected mix per model. The mix is the sales take rates blended across the dealer, its neighborhood and the country. Candidate selection scores candidates against these rates.

Decision​

Take-rate surprise reads demand take rates. The expected code probabilities of an order set are the take rates of its units' dealer and model, averaged over its units. The test treats these rates as certain: an order set's code counts are tested against a multinomial distribution with them as probabilities. It does not read a reference.

Consequences​

  • A dealer's expected mix has one definition, shared with candidate selection.
  • The evaluation measures how far an order set deviates from the rates candidate selection aims for. An order set that is typical of demand but unlike what the dealer stocks is not flagged.
  • Candidate selection treats the take rates as certain, so the evaluation does too and allows nothing for their estimation error. Larger order sets are therefore flagged for smaller deviations.
  • A code without a take rate has probability 0, so an order set with one gets the minimum p-value.
  • Whether the take rates themselves are wrong is a different question. A test against the dealer's own sales history, allowing for their uncertainty, would answer it.
  • An OEM can configure take-rate surprise without a reference.

ADR-014: An order set evaluation reads every input at the run's partition_date​

Status: Proposed (2026-10-16)

Context​

An ordering run selects its candidates from the warehouse partitions of one date, its partition_date. The ordering service resolves that date from the run's requested as_of_date before the run reads any data. It is the newest date within the run's lookback on which every table the run reads is complete. It can be earlier than the day the run started. A simulated run can request an as_of_date in the past.

The result of an evaluation that reads the latest partition of each input depends on when the evaluation runs. A partition that arrives later changes the result when the run is evaluated again. The evaluation can also read other dealers, models, features or take rates than the run did.

Decision​

order_set_evaluations reads the run's partition_date from ordering/runs, as the input partitions rule of the reporting layer requires. It reads each of its inputs at that date:

  • consolidated/dealers and consolidated/models;
  • attributes/features and consolidated/features;
  • consolidated/inventory, attributes/vehicles and reporting/placed_orders, from which it forms the reference order sets (ADR-011);
  • output/demand_take_rates.

It raises when the run's partition_date is null, and when an input has no partition on that date. reporting/placed_orders is the exception: its partition has no rows until the first order is placed, and an empty partition cannot be told apart from a missing one.

Consequences​

  • Evaluating a run again gives the same rows, as long as its input partitions are unchanged.
  • Take-rate surprise tests an order set against the take rates that candidate selection scored the run's candidates with.
  • A run is evaluated when its partition is registered. If an input has no partition on the run's partition_date yet, the evaluation fails, and the run's order_set_evaluations partition has to be materialized again once that partition exists. A run evaluated before reporting/placed_orders has its partition leaves no placed order out of its reference, and does not fail.
  • For a run whose partition_date_source is run_date, order_set_evaluations raises when an input has no partition on the run's partition_date. The run may have read an earlier partition.

ADR-015: An evaluation can select its attribute types by attribute weight​

Status: Proposed (2026-10-27)

Context​

An evaluation listed the attribute types it covers, and a reference listed the types it stores. Stellantis evaluated PAINT_COLOR only. Candidate selection already assigns each attribute type a weight per model in output/attribute_weights, and those weights change as the data changes.

Decision​

An evaluation's attribute_types is a list or a selection from output/attribute_weights. A selection takes the top_k heaviest types of the order set's model, or the types with at least min_weight. The weights are read from the partition of the run's partition_date, the date of the partitions the run selected its candidates from (ADR-014). A missing weights partition raises; the evaluation does not read an older partition instead. A reference stores the types its evaluations select, so it has no list of its own.

The weights are fitted on featurization/inventory, so the candidates are featurized by VehicleFeaturizationSettings.featurize_candidates, the function that writes featurization/synthetic. A candidate gets each <GROUP>_SET type, and it can have every type that has a weight. The featurization reads no inventory column. A candidate has no inventory row, so a type read from one would be NO_<type> on every candidate.

The output keeps one row per order set and attribute type, with attribute_weight. A score over an order set's types is not stored.

Consequences​

  • The types evaluated follow the weights, so an evaluation's rows can change from one partition of the weights to the next. weights_partition_date records which partition a row's types came from.
  • A weighted score and an unweighted one can be computed from the rows. Where they diverge, the types with the most weight have p-values unlike those of the other selected types.
  • order_set_evaluations reads consolidated/inventory only when an evaluation reads a reference.
  • An OEM sets the same featurization values in OrderSetEvaluationComponent and VehicleFeaturizationComponent. A difference featurizes candidates unlike the vehicles the weights, the take rates and the references come from.
  • When a model of a run's order sets has no weights, or the weights partition is missing, every evaluation of the run fails, including the evaluations that list their types.
  • For each value of its restriction keys, such as a base model, a reference stores only the types that the run's order sets with that value select, so its memory grows with top_k and with the populations of the run's base models.

ADR-016: Configuration rarity scores each unit's weighted configuration against demand take rates​

Status: Proposed (2026-10-27)

Context​

Take-rate surprise tests one attribute type of an order set at a time. A candidate whose paint color, wheels and interior are each somewhat uncommon can be rare as a whole configuration, and no existing evaluation says which candidates are rare.

The product of a unit's take rates measures how common its configuration is. Its value depends on how many codes each attribute type has, so one threshold does not fit every model. Many attribute types restate others: a cross type restates its parts, and a binary option is counted again in its <GROUP>_SET. Candidate selection weights each type by its attribute weight. The attribute weights leave out such types or give them a lower weight.

Decision​

Configuration rarity scores each unit by the sum of the logs of its codes' take rates, each times the type's attribute weight when the evaluation selects its types by weight. A code without a positive take rate gives -inf. The evaluation draws configurations from the same take rates, each type's code independently, and estimates the probability that a drawn configuration scores at most as high. A unit is rare when that probability is at most rare_mass.

An order set's result is the number of its rare units. Its p-value is the probability of at least that many, when each unit is rare with the probability that a drawn configuration of its rate keys is rare. The number of rare units then follows a Poisson binomial distribution. Each unit's result is written to reporting/unit_rarities, and the run's rare units are tested together in its materialization metadata.

Consequences​

  • Rarity is measured on the attribute types candidate selection weights, in the proportions it weights them. A rare code of a type without weight does not make a unit rare.
  • A code the dealer's demand never takes makes its unit rare, whatever its other codes.
  • The drawn configurations treat the attribute types as independent. Types that depend on each other and that the weights do not leave out, such as wheel diameter and tire type, can make a common configuration look rare.
  • A unit's probability is estimated from draws drawn configurations. With every weighted type, the configurations are too many to enumerate.
  • The baseline is units drawn from the take rates. Candidate selection does not draw from the take rates; it fills gaps between demand and supply. As with take-rate surprise (ADR-013), a large order set is flagged for a smaller deviation.
  • order_set_evaluations and unit_rarities are written by one materialization. A failed check or computation writes neither. Each table is written in its own commit, order_set_evaluations first, so a failed write of unit_rarities leaves order_set_evaluations written; a retry overwrites the run's partition of both.

ADR-017: Set size z-score compares an order set's size with the dealer's reference windows​

Status: Proposed (2026-10-28)

Context​

Volume optimization decides how many vehicles of a base model to order for a dealer. The existing evaluations judge which vehicles an order set holds and do not judge its size. A dealer's own history of arrivals per base model is the nearest baseline for that number.

A sliding-window reference whose step_days equals its width_days divides the population into non-overlapping windows, so each window is one observation of the dealer's volume. A window without a unit is an observation of size 0. The code counts that empirical evaluations read do not include such a window, because a window without a unit has no code.

Decision​

set_size_z_score reads the size of every reference order set of its reference, a new input besides the code counts. A sliding-window reference gives a window without a unit a size of 0 for each group_by value with a unit in another window. The evaluation standardizes the order set's size by the mean and sample standard deviation of the reference order sets of the same match_keys values. Its p-value is two-sided and empirical. With n such sets, of which r are at least as far from their mean as the order set on either side, the p-value is (1 + r) / (n + 1).

As for every other evaluation, the population is the reference's, which leaves out the orders we placed.

Consequences​

  • A base model launched during the population period has windows of size 0 before its launch, which lower the mean.
  • The population leaves out orders we placed, so windows after we started ordering hold only the dealer's own orders. A dealer whose orders we replaced looks like a dealer with less volume.
  • An order set covers one run and a window covers width_days days. The comparison is meaningful when a run orders about one window's volume, so width_days should match the number of days between the OEM's allocations.
  • With few windows, the p-value is coarse. Its smallest possible value is 1 / (baseline_sets + 1).