Reporting Layer Design
The reporting layer measures what the ordering platform recommended and how the vehicles it ordered performed. It replaces the legacy Dataform reporting with shared Dagster components that each OEM project configures. This page describes the target architecture; Decisions records why.
Terminology
Orders
- Placement — an attempt that put an order into the OMS. An order can have several placements. Placed Orders states which attempts count. Avoid: submission, execution
- Placed order — an order with at least one placement, named by its order key. Its latest placement supplies its cured build.
- Order key — the identifier that names one order across placements and captures, such as an
order_number, or a dealer and anorder_referencewhile an order is staged. - Capture — one reading of an order from the OEM's private inventory, dated by the day it was collected. Avoid: snapshot (an Iceberg term), observation
- Locked — the state of an order whose capture shows that the OEM no longer accepts changes to it. Avoid: settled, final
Builds
- Build — a vehicle configuration at one moment. An ordering attempt records a build as a
BuildSnapshot. - Build source — where a build was read:
recommended(our recommendation),cured(the build the OMS accepted), orinventory(a capture). - Build delta — the raw differences between two builds and their magnitudes, before any metric classifies them. Avoid: diff, change set
- Mutually exclusive feature group — features a vehicle has at most one of.
- Replacement — one member of a mutually exclusive group exchanged for another between two builds. A replacement counts as one change, not an addition and a removal.
Dealer modifications
- Modification — a change a dealer makes to a placed order before it is produced. Avoid: edit, override
- Counted change — a changed feature that the OEM's
counted_features_filterselects. - Order filter — a named selection of placed orders that a rate covers. Avoid: population, cohort
- Modification level — a named limit on counted changes and MSRP change. An order reaches the level when it exceeds either limit. Avoid: threshold, severity
- Modified rate — the share of the orders an order filter selects that reach a modification level.
- Acceptance — a dealer keeping a build we placed, without modifying it. The acceptance rate is 1 minus the modified rate. Avoid: retention, keep rate
Runs and volume
- Volume deviation — the difference between the number of orders we placed for a model and the number the dealer ends up with.
- Alignment — how far a run moved supply take rates toward demand take rates.
run_metricsreports it asweighted_misalignment_reduction, and per attribute type asmisalignment_reduction. See Alignment.
The reporting layer
Per-run metrics are final once an ordering run finishes. They cover whether a configuration was validated and could be ordered, and the alignment the run achieved. Order Set Evaluation compares each run's recommended order sets with sets of vehicles that dealers stocked, or with the dealers' demand take rates.
Run Metrics describes the component that computes them.
Vehicle performance metrics cover aspects of our objective function, such as turn rate and profitability.
What exists today
Three legacy Dataform projects produce reporting tables, and three applications read them.
| System | Produces | State |
|---|---|---|
bmw-po-reporting | Cohort tables, modification and inventory metrics per grouping level, take-rate metrics, eleven Looker feeds, ten assertions | Maintained |
netstar-copilot-reporting | Per-run session statistics, recommendation-to-cured comparison, modification rates, contribution-margin comparison, profit and billing metrics | Maintained |
gcp-dataform-inventory | The BMW predictive-ordering pipeline. Its reporting output is modification rates, option-change tables, and order-flow and modification take rates by dealer and product | Maintained |
gcp-dataform-PO-rec-configs/dashboard | BMW Streamlit dashboard on Cloud Run | Maintained |
dealer-portal/apps/internal | The Mercedes dashboard at ppm.autointel.ai | Maintained |
ppm-reporting-dashboard | The Stellantis dashboard, reading ordering runs and attempts from Mongo | Local MVP |
Stellantis and Audi have no formal reporting pipeline and are queried ad hoc.
bmw-po-reporting and netstar-copilot-reporting compute the same concepts by different means.
Both derive a modification rate by comparing the build we recommended against the build the OMS
reports later, and both aggregate it by dealer, model and allocation period. They share no code.
Architecture
Components are generic and live in packages/ai_dagster/src/ai_dagster/components/. Each OEM
project configures them in
projects/ai_<oem>/src/ai_<oem>/defs/<oem>/<market>/reporting/defs.yaml.
The boundary follows ConsolidatedComponent. Each metric and its calculation is defined in the
component for every OEM, and outputs conform to a shared schema. Configuration supplies what differs
per OEM: sources, matching columns, mutually exclusive features, and how each metric classifies a
build delta. See
ADR-001.
Cross-OEM aggregation is not part of this design. It belongs in ai_org, and reads the per-OEM
outputs.
Inputs
OrderingEventsComponent already writes three topics to Iceberg for every OEM the ordering service
serves, typed by the models in packages/ai_ppm/src/ai_ppm/ordering_events.py. Its sensor registers
a partition when a run reaches a terminal status. A simulated run with several dealers has a
terminal status once its ordering_run_groups document has one. The sensor registers the
partition of such a run only when at least one of its ordering_runs documents also has a
terminal status.
The component also writes <oem>/<market>/ordering/runs, with one row per dealer of a run from
ordering_runs. Its columns derive from ai_ppm.ordering_runs.OrderingRunRecord, the model the
ordering service's RunSnapshot extends. Among them:
dealer_codeholds the code of the row's dealer, which theordering_runsdocument namesdealer_id.run_modeandrun_timestampmatch the columns of the same name on the event tables.requested_as_of_dateholds theas_of_datethe run started with.partition_dateholds the warehouse partition the run read.partition_date_sourceis set only on a run that did not recordpartition_dateitself.snapshotsholds the collection sessions the run resolved.
partition_date is null on a run that stopped before the partition probe answered, and on a run
that the probe stopped. Every dealer of a run records the same partition_date.
run_partition_date in ai_dagster.components.common raises when the run's rows hold different
values.
ordering_runs documents written before the ordering service recorded the partition have no
partition_date. For each of these runs that has at least one ordering_events document,
scripts/migrate_ordering_runs.py in services/ai_ordering sets one:
reconstructed: the partition probe evaluated against the Iceberg snapshot of each table that was current when the run started.run_date: the UTC date the run started, where a table has no snapshot that old. That date can be later than the partition the run read.
Vehicle performance metrics read consolidated inventory, which already computes days_on_lot,
inferred_sold_date and is_active. Build deltas are computed from the private OMS sources
instead, because consolidated resolves a vehicle's features by source priority and can return the
public listing's view. See ADR-002.
Input partitions
A per-run reporting asset reads each of its inputs, such as dealers, models and features, at the
run's partition_date from ordering/runs. It never reads the newest partition on the day it
materializes, so a rerun reads the partitions the run read. A per-run asset raises when it needs the
date and the run's partition_date is null.
A missing partition at a recorded or reconstructed partition_date means warehouse data was lost,
so a per-run asset raises on it. The exception is a run whose partition_date_source is
run_date. That date is the day the run started, which can be later than the partition it read. A
per-run asset may then leave a value that depends on a missing partition null instead of raising.
Cadence
Per-run metrics are computed from the partition the ordering-events sensor registers. Vehicle performance metrics are computed on a daily partition. See ADR-007.
Metrics aggregated over several runs are computed downstream of the per-run metrics.
Run modes
Per-run metrics are materialized for every ordering run mode. The run mode is a column on the asset, so any one mode can be reported on its own. Vehicle performance metrics cover only the modes that place an order into the OMS. See ADR-006.
Cancelled staged orders count toward volume deviation, but are excluded from vehicle performance metrics and the asset check for unmatched orders.
The daily unattended run is SIMULATED and places no order. Its events are written to the same
ordering_events collection as every other mode's. Each event records its run_mode, so a
simulated run's volume can be kept apart from the volume of the other modes.
Matching an order to an inventory record
Measuring an outcome requires finding the vehicle an order became. No single identifier serves every OEM, so each one declares an ordered list of foreign keys that match the two. An asset check counts orders that can neither be matched to an inventory record nor identified as a cancellation. It counts an order only after the dealer's OMS inventory is captured on a later day, and it leaves out orders whose vehicle may have left consolidated inventory. See ADR-003.
Placed Orders describes the component that matches each placed order to its inventory record.
Reporting inputs name a dealer by the OEM's dealer code. Each reporting asset resolves the code to
the consolidated dealer_id through consolidated/dealers. The asset keeps only dealer_id.
Builds and the deltas between them
Reporting reads builds from three build sources:
recommended— the build on arecommendationseventcured— the build on avalidated_candidatesevent, which the OMS acceptedinventory— a capture of the order from the private OMS sources
An ordering attempt in the ordering service also records original and final builds. The event
stream does not publish them, so reporting reads the build an order had before we placed it from
an earlier capture instead. Audi observes one more build between staging and submission.
One calculation produces the delta between any two of those builds:
- A dealer's modification
- The change OMS curing made
- The change we made to an order that reached the OMS already configured
See ADR-004.
OrderAttemptStatus stays a separate dimension. It records VALIDATED, SIMULATED, STAGED,
SUCCESS, FAILURE, REJECTED and CANCELLED.
Control vehicles
BMW allocates a percentage of control templates. Mercedes leaves a percentage of modifiable orders untouched. Reporting consumes whichever marking each OEM produces. Every vehicle falls into one of three groups:
- We recommended no build for it
- We recommended a build and the dealer kept it
- We recommended a build and the dealer changed it
Build deltas
A build delta reports the raw differences between two builds and their magnitudes:
- The features replaced within a mutually exclusive group, and the features added and removed
- The count of each of those three
- The properties of each changed feature
- The price of each build
- The model of each build
Each changed feature resolves to a feature_id from consolidated features, so every property of
the feature entity joins to the delta.
A delta classifies nothing. Metrics that read deltas classify them their own way — OMS curing, dealer acceptance, per-unit billing — and each classification is configured per OEM. See ADR-004.
Consumers outside the reporting layer may read the deltas too.
Build Deltas describes the component that computes them. Dealer Modifications describes the component that classifies the deltas of the orders we placed.
Volume
For OEMs with volume flexibility, a dealer may change the number of orders we placed during an allocation window by canceling our orders and creating their own. Further on in the production pipeline, changes can occur via rejection pools, dealer trades and supplementary allocations.
The counts are computed daily per dealer, by model and model group. Different order types and sales channels are excluded or calculated separately.
Dealer acceptance of builds reflects on the quality of our candidate selection process. Dealer acceptance of volume reflects on the quality of our volume optimization process. See ADR-005.
Grouping levels
Each metric is reported at the grouping levels its configuration names.
InventoryStatisticsComponent already emits one asset per configured grouping, and
GeographicSmootherComponent blends several source levels into one output level where a level is
too sparse to stand alone.
Metric filters
A metric can filter out vehicles that would distort it. Which vehicles a metric filters out is configured per OEM. For example:
- Customer orders, generally filtered out of KPIs
- Vehicles sold in a fleet sale, generally filtered out of turn stats
- Control vehicles from a period when we were not ordering at that dealer, which have nothing contemporaneous to compare against
Asset checks
Each reporting asset has asset checks attached. The computation is a plain function, a factory builds the check, and severity decides whether a failure blocks.
Checks worth writing include:
- Unmatched orders
- Feature references that do not resolve
- Duplicate vehicles
- Null modification flags
- Totals that should agree between grouping levels
Comparing against the legacy output
New output is compared against the legacy tables, read with BigQueryAssetBackfillComponent.
Cutover happens once the remaining differences are explained. See
ADR-009.
Open questions
- Where mutually exclusive feature groups are ultimately defined. Reporting configuration is the starting point, and the Rules entity currently being scoped needs the same information.
- Whether per-unit billing counts a modified order differently.
- How to handle a control group of dealers we don't order for. Consolidated inventory covers their performance only over the lifecycle of statuses the public feeds expose.
- How a feature whose code changes is recognized as the same feature. The new code resolves to a
different
feature_id, so a delta reports an addition and a removal for a vehicle that did not change. Metrics organized by attribute are unaffected. - How to account for a change of dealer that coincides with a change to the build.
Deferred
Why a given volume was chosen is not yet reportable, because the solver's reasoning is not persisted. See ADR-005.
DMS data is out of scope until it is added to the ingestion layer. It will serve as the basis of profitability KPIs.
From bmw-po-reporting: the iX experiment tables, bmw_predictive_pns and its per-dealer
significance tests, the modification rate curve, take-rate projections, demand-generation and lead
variables, and the L10 metrics. Service account billing is cost monitoring and belongs with the
other cost assets in ai_org.
allocations_norm_mapped is excluded entirely. It corrects drift in a positional string identifier
that encodes a configuration, so allocation-time identifiers still match the candidate configuration
table after the product catalog changes. This platform identifies a product by model_id, so that
identifier has no counterpart here. Drift in a feature's own code is a separate case, listed under
open questions.