Skip to main content

Reporting Layer Design

The reporting layer measures what the ordering platform recommended and how the vehicles it ordered performed. It replaces the legacy Dataform reporting with shared Dagster components that each OEM project configures. This page describes the target architecture; Decisions records why.

Terminology​

Orders​

  • Placement — an attempt that put an order into the OMS. An order can have several placements. Placed Orders states which attempts count. Avoid: submission, execution
  • Placed order — an order with at least one placement, named by its order key. Its latest placement supplies its cured build.
  • Order key — the identifier that names one order across placements and captures, such as an order_number, or a dealer and an order_reference while an order is staged.
  • Capture — one reading of an order from the OEM's private inventory, dated by the day it was collected. Avoid: snapshot (an Iceberg term), observation
  • Locked — the state of an order whose capture shows that the OEM no longer accepts changes to it. Avoid: settled, final

Builds​

  • Build — a vehicle configuration at one moment. An ordering attempt records a build as a BuildSnapshot.
  • Build source — where a build was read: recommended (our recommendation), cured (the build the OMS accepted), or inventory (a capture).
  • Build delta — the raw differences between two builds and their magnitudes, before any metric classifies them. Avoid: diff, change set
  • Mutually exclusive feature group — features a vehicle has at most one of.
  • Replacement — one member of a mutually exclusive group exchanged for another between two builds. A replacement counts as one change, not an addition and a removal.

Dealer modifications​

  • Modification — a change a dealer makes to a placed order before it is produced. Avoid: edit, override
  • Counted change — a changed feature that the OEM's counted_features_filter selects.
  • Order filter — a named selection of placed orders that a rate covers. Avoid: population, cohort
  • Modification level — a named limit on counted changes and MSRP change. An order reaches the level when it exceeds either limit. Avoid: threshold, severity
  • Modified rate — the share of the orders an order filter selects that reach a modification level.
  • Acceptance — a dealer keeping a build we placed, without modifying it. The acceptance rate is 1 minus the modified rate. Avoid: retention, keep rate

Runs and volume​

  • Volume deviation — the difference between the number of orders we placed for a model and the number the dealer ends up with.
  • Alignment — how far a run moved supply take rates toward demand take rates. run_metrics reports it as weighted_misalignment_reduction, and per attribute type as misalignment_reduction. See Alignment.

The reporting layer​

Per-run metrics are final once an ordering run finishes. They cover whether a configuration was validated and could be ordered, and the alignment the run achieved. Order Set Evaluation compares each run's recommended order sets with sets of vehicles that dealers stocked, or with the dealers' demand take rates.

Run Metrics describes the component that computes them.

Vehicle performance metrics cover aspects of our objective function, such as turn rate and profitability.

What exists today​

Three legacy Dataform projects produce reporting tables, and three applications read them.

SystemProducesState
bmw-po-reportingCohort tables, modification and inventory metrics per grouping level, take-rate metrics, eleven Looker feeds, ten assertionsMaintained
netstar-copilot-reportingPer-run session statistics, recommendation-to-cured comparison, modification rates, contribution-margin comparison, profit and billing metricsMaintained
gcp-dataform-inventoryThe BMW predictive-ordering pipeline. Its reporting output is modification rates, option-change tables, and order-flow and modification take rates by dealer and productMaintained
gcp-dataform-PO-rec-configs/dashboardBMW Streamlit dashboard on Cloud RunMaintained
dealer-portal/apps/internalThe Mercedes dashboard at ppm.autointel.aiMaintained
ppm-reporting-dashboardThe Stellantis dashboard, reading ordering runs and attempts from MongoLocal MVP

Stellantis and Audi have no formal reporting pipeline and are queried ad hoc.

bmw-po-reporting and netstar-copilot-reporting compute the same concepts by different means. Both derive a modification rate by comparing the build we recommended against the build the OMS reports later, and both aggregate it by dealer, model and allocation period. They share no code.

Architecture​

Components are generic and live in packages/ai_dagster/src/ai_dagster/components/. Each OEM project configures them in projects/ai_<oem>/src/ai_<oem>/defs/<oem>/<market>/reporting/defs.yaml.

The boundary follows ConsolidatedComponent. Each metric and its calculation is defined in the component for every OEM, and outputs conform to a shared schema. Configuration supplies what differs per OEM: sources, matching columns, mutually exclusive features, and how each metric classifies a build delta. See ADR-001.

Cross-OEM aggregation is not part of this design. It belongs in ai_org, and reads the per-OEM outputs.

Inputs​

OrderingEventsComponent already writes three topics to Iceberg for every OEM the ordering service serves, typed by the models in packages/ai_ppm/src/ai_ppm/ordering_events.py. Its sensor registers a partition when a run reaches a terminal status. A simulated run with several dealers has a terminal status once its ordering_run_groups document has one. The sensor registers the partition of such a run only when at least one of its ordering_runs documents also has a terminal status.

The component also writes <oem>/<market>/ordering/runs, with one row per dealer of a run from ordering_runs. Its columns derive from ai_ppm.ordering_runs.OrderingRunRecord, the model the ordering service's RunSnapshot extends. Among them:

  • dealer_code holds the code of the row's dealer, which the ordering_runs document names dealer_id.
  • run_mode and run_timestamp match the columns of the same name on the event tables.
  • requested_as_of_date holds the as_of_date the run started with.
  • partition_date holds the warehouse partition the run read.
  • partition_date_source is set only on a run that did not record partition_date itself.
  • snapshots holds the collection sessions the run resolved.

partition_date is null on a run that stopped before the partition probe answered, and on a run that the probe stopped. Every dealer of a run records the same partition_date. run_partition_date in ai_dagster.components.common raises when the run's rows hold different values.

ordering_runs documents written before the ordering service recorded the partition have no partition_date. For each of these runs that has at least one ordering_events document, scripts/migrate_ordering_runs.py in services/ai_ordering sets one:

  • reconstructed: the partition probe evaluated against the Iceberg snapshot of each table that was current when the run started.
  • run_date: the UTC date the run started, where a table has no snapshot that old. That date can be later than the partition the run read.

Vehicle performance metrics read consolidated inventory, which already computes days_on_lot, inferred_sold_date and is_active. Build deltas are computed from the private OMS sources instead, because consolidated resolves a vehicle's features by source priority and can return the public listing's view. See ADR-002.

Input partitions​

A per-run reporting asset reads each of its inputs, such as dealers, models and features, at the run's partition_date from ordering/runs. It never reads the newest partition on the day it materializes, so a rerun reads the partitions the run read. A per-run asset raises when it needs the date and the run's partition_date is null.

A missing partition at a recorded or reconstructed partition_date means warehouse data was lost, so a per-run asset raises on it. The exception is a run whose partition_date_source is run_date. That date is the day the run started, which can be later than the partition it read. A per-run asset may then leave a value that depends on a missing partition null instead of raising.

Cadence​

Per-run metrics are computed from the partition the ordering-events sensor registers. Vehicle performance metrics are computed on a daily partition. See ADR-007.

Metrics aggregated over several runs are computed downstream of the per-run metrics.

Run modes​

Per-run metrics are materialized for every ordering run mode. The run mode is a column on the asset, so any one mode can be reported on its own. Vehicle performance metrics cover only the modes that place an order into the OMS. See ADR-006.

Cancelled staged orders count toward volume deviation, but are excluded from vehicle performance metrics and the asset check for unmatched orders.

The daily unattended run is SIMULATED and places no order. Its events are written to the same ordering_events collection as every other mode's. Each event records its run_mode, so a simulated run's volume can be kept apart from the volume of the other modes.

Matching an order to an inventory record​

Measuring an outcome requires finding the vehicle an order became. No single identifier serves every OEM, so each one declares an ordered list of foreign keys that match the two. An asset check counts orders that can neither be matched to an inventory record nor identified as a cancellation. It counts an order only after the dealer's OMS inventory is captured on a later day, and it leaves out orders whose vehicle may have left consolidated inventory. See ADR-003.

Placed Orders describes the component that matches each placed order to its inventory record.

Reporting inputs name a dealer by the OEM's dealer code. Each reporting asset resolves the code to the consolidated dealer_id through consolidated/dealers. The asset keeps only dealer_id.

Builds and the deltas between them​

Reporting reads builds from three build sources:

  1. recommended — the build on a recommendations event
  2. cured — the build on a validated_candidates event, which the OMS accepted
  3. inventory — a capture of the order from the private OMS sources

An ordering attempt in the ordering service also records original and final builds. The event stream does not publish them, so reporting reads the build an order had before we placed it from an earlier capture instead. Audi observes one more build between staging and submission.

One calculation produces the delta between any two of those builds:

  • A dealer's modification
  • The change OMS curing made
  • The change we made to an order that reached the OMS already configured

See ADR-004.

OrderAttemptStatus stays a separate dimension. It records VALIDATED, SIMULATED, STAGED, SUCCESS, FAILURE, REJECTED and CANCELLED.

Control vehicles​

BMW allocates a percentage of control templates. Mercedes leaves a percentage of modifiable orders untouched. Reporting consumes whichever marking each OEM produces. Every vehicle falls into one of three groups:

  • We recommended no build for it
  • We recommended a build and the dealer kept it
  • We recommended a build and the dealer changed it

Build deltas​

A build delta reports the raw differences between two builds and their magnitudes:

  • The features replaced within a mutually exclusive group, and the features added and removed
  • The count of each of those three
  • The properties of each changed feature
  • The price of each build
  • The model of each build

Each changed feature resolves to a feature_id from consolidated features, so every property of the feature entity joins to the delta.

A delta classifies nothing. Metrics that read deltas classify them their own way — OMS curing, dealer acceptance, per-unit billing — and each classification is configured per OEM. See ADR-004.

Consumers outside the reporting layer may read the deltas too.

Build Deltas describes the component that computes them. Dealer Modifications describes the component that classifies the deltas of the orders we placed.

Volume​

For OEMs with volume flexibility, a dealer may change the number of orders we placed during an allocation window by canceling our orders and creating their own. Further on in the production pipeline, changes can occur via rejection pools, dealer trades and supplementary allocations.

The counts are computed daily per dealer, by model and model group. Different order types and sales channels are excluded or calculated separately.

Dealer acceptance of builds reflects on the quality of our candidate selection process. Dealer acceptance of volume reflects on the quality of our volume optimization process. See ADR-005.

Grouping levels​

Each metric is reported at the grouping levels its configuration names. InventoryStatisticsComponent already emits one asset per configured grouping, and GeographicSmootherComponent blends several source levels into one output level where a level is too sparse to stand alone.

Metric filters​

A metric can filter out vehicles that would distort it. Which vehicles a metric filters out is configured per OEM. For example:

  • Customer orders, generally filtered out of KPIs
  • Vehicles sold in a fleet sale, generally filtered out of turn stats
  • Control vehicles from a period when we were not ordering at that dealer, which have nothing contemporaneous to compare against

Asset checks​

Each reporting asset has asset checks attached. The computation is a plain function, a factory builds the check, and severity decides whether a failure blocks.

Checks worth writing include:

  • Unmatched orders
  • Feature references that do not resolve
  • Duplicate vehicles
  • Null modification flags
  • Totals that should agree between grouping levels

Comparing against the legacy output​

New output is compared against the legacy tables, read with BigQueryAssetBackfillComponent. Cutover happens once the remaining differences are explained. See ADR-009.

Open questions​

  • Where mutually exclusive feature groups are ultimately defined. Reporting configuration is the starting point, and the Rules entity currently being scoped needs the same information.
  • Whether per-unit billing counts a modified order differently.
  • How to handle a control group of dealers we don't order for. Consolidated inventory covers their performance only over the lifecycle of statuses the public feeds expose.
  • How a feature whose code changes is recognized as the same feature. The new code resolves to a different feature_id, so a delta reports an addition and a removal for a vehicle that did not change. Metrics organized by attribute are unaffected.
  • How to account for a change of dealer that coincides with a change to the build.

Deferred​

Why a given volume was chosen is not yet reportable, because the solver's reasoning is not persisted. See ADR-005.

DMS data is out of scope until it is added to the ingestion layer. It will serve as the basis of profitability KPIs.

From bmw-po-reporting: the iX experiment tables, bmw_predictive_pns and its per-dealer significance tests, the modification rate curve, take-rate projections, demand-generation and lead variables, and the L10 metrics. Service account billing is cost monitoring and belongs with the other cost assets in ai_org.

allocations_norm_mapped is excluded entirely. It corrects drift in a positional string identifier that encodes a configuration, so allocation-time identifiers still match the candidate configuration table after the product catalog changes. This platform identifies a product by model_id, so that identifier has no counterpart here. Drift in a feature's own code is a separate case, listed under open questions.