Skip to main content

ML CDF Supply Model — Operational Learnings

This document records empirical findings from validating MLCDFSupplyModel against the Dataform fwd_inventory_weight baseline at dealer 10742 / model XD and four other dealer/model pairs. It covers what constitutes reasonable model behaviour, known failure modes, the rationale for removing the days-to-arrival path, and how the resulting weights compare structurally to the current Dataform approach.

1. Why the days-to-arrival path was removed​

The original design used two formulas dispatched on whether a VIN appeared in pit_days_until_dealerarrival_inference:

ON_LOT   (absent from arrival table):  P(on lot at d) = 1 − sale_CDF(d)
PIPELINE (present in arrival table):   P(on lot at d) = clip(arr_CDF(d) − sale_CDF(d), 0, 1)

This produced systematically wrong rankings: status 5/6 (on-order/in-transit) VINs received near-zero weights while long-sitting status 1 VINs received the highest weights — the inverse of the intuitive ordering.

Three independent failure modes all produce the same wrong answer via the pipeline path:

Failure modeEffect
Arrival table is stale (features not refreshed upstream)arr_CDF predicts late arrival; arr_CDF − sale_CDF ≈ 0
Arrival model systematic bias (over-predicts late arrival for status 5/6)Same near-zero overlap
approx_ev_days_until_dealerarrival = 0 data quality issue (see §3)Sale model shifts probability to 0–30 days; arr_CDF − sale_CDF ≈ 0 at useful horizons

The root cause is that the sale model already encodes arrival delay: pit_days_until_sale_inference uses approx_ev_days_until_dealerarrival, orderstatus, eventid, and days_in_production as features. For a status 6 VIN with EV arrival at day 57, the sale EV (~81 days) naturally decomposes as ~57 days pipeline + ~24 days on-lot dwell. The pipeline path was double-counting the arrival delay instead of benefiting from it.

Resolution: drop the pipeline path entirely. Use 1 − sale_CDF(d) for all VINs regardless of orderstatus. The arrival table is no longer needed by the supply model. The vehicle_arrival_cdf dataset was removed from bmw_ml_cdf.yaml.

2. What constitutes reasonable predictions​

Orderstatus ranking​

The intuitive ordering of supply weights, from highest to lowest, is:

status 6 (on order) > status 5 (in transit/production) > status 2 (in transit, near arrival) > status 1 (on lot)

Rationale: a recently-ordered vehicle will be on lot for longer across the candidate's arrival window than a vehicle that has already been sitting 100+ days and is likely to sell before the new candidate arrives.

With 1 − sale_CDF universally applied, this ordering is now reproduced. At dealer 10742 / model XD, mean ml_weight by status:

orderstatusmean ml_weightmean fwd_weight_normratio
1 (on lot)0.00330.00241.43×
2 (in transit)0.00370.00580.66×
5 (in production)0.00550.00830.66×
6 (on order)0.00600.00780.78×

This pattern (status 6 > status 5 > status 1) is consistent across all five dealer/model pairs profiled.

Near-flat weights within status 1​

Status 1 (on-lot) VINs at 10742/XD show only 18 unique ml_weight values across 156 vehicles, with an IQR of 0.000015 (std/mean ≈ 2%). The sale model has low per-VIN discriminative power within the on-lot cohort at this dealer/model — all vehicles receive roughly the same survival curve. This is not a bug; it reflects genuine similarity in the sale model's prediction for on-lot vehicles here. The meaningful differentiation occurs between status cohorts, not within them.

3. Data quality: approx_ev_days_until_dealerarrival = 0 for status 5/6 VINs​

approx_ev_days_until_dealerarrival is a key feature in the sale model for pipeline vehicles. When this field defaults to 0 for a just-ordered VIN, the sale model receives contradictory signals:

  • orderstatus = 6 → vehicle not yet in production
  • approx_ev_days_until_dealerarrival = 0 → vehicle arriving immediately

This produced a bimodal sale distribution: ~30% probability of selling in 0–30 days (arrival-imminent signal) and ~56% in >100 days (high smart_days_of_supply_360d dampening). The EV looked plausible (~81 days) but the distribution was wrong.

After refreshing pit_days_until_sale_inference from a freshly-run pit_days_until_dealerarrival_features table, the same VIN showed approx_ev_days_until_dealerarrival = 57.6 and the 0–30 day bucket dropped from 29.7% → 2.5%.

Upstream dependency chain that must be run in order:

inventory_pit_status
  → pit_days_until_dealerarrival_features   ← commonly goes stale; must refresh first
    → pit_days_until_dealerarrival_inference
      → pit_days_until_sale_features        ← consumes approx_ev_days_until_dealerarrival
        → pit_days_until_sale_inference     ← consumer in MLCDFSupplyModel

If pit_days_until_sale_inference is re-run without first refreshing pit_days_until_dealerarrival_features, newly-ordered VINs will have stale or zero arrival EV values and incorrect sale distributions.

Recommended check before running the supply model:

# In MLCDFSupplyModel.compute(), the zero-arrival warning (removed with the pipeline
# path) should be re-instated at the feature level — flag any VINs where
# approx_ev_days_until_dealerarrival = 0 AND orderstatus IN (5, 6) in the inference table,
# since these are almost certainly data quality issues rather than genuine predictions.

The supply_days_until_sale_by_dealer_model DM template correctly hardcodes approx_ev_days_until_dealerarrival = prod_to_dealer_mean (defaulting to 30 days from inventory_pipeline_estimates), so the candidate curve is robust to this issue.

4. The candidate curve (DM template)​

compute_on_lot_curve() returns 1 − DM_sale_CDF from supply_days_until_sale_by_dealer_model. That table assumes:

  • orderstatus = 6, eventid = 111, prioritycode = 3 (a generic just-ordered vehicle)
  • approx_ev_days_until_dealerarrival = market × series average of prod_to_dealer_mean from inventory_pipeline_estimates, with a fallback to 30 days when no match exists
  • Rolling market/supply signals (smart_days_of_supply_360d, days_on_lot_90d, pre_inventory_sales_pct_90d) from current snapshots

For dealer 10742 / model XD, the DM template EV is 61 days. A real status 6 VIN with arrival EV ~58 days has sale EV ~81 days. The gap (~20 days) reflects the difference between the market-average pipeline lead time used by the template and the VIN's actual expected arrival timeline. The DM template is a reasonable stand-in for newly-ordered candidates and is robust to the approx_ev_days_until_dealerarrival = 0 data quality issue that can affect per-VIN predictions, since it derives its lead time from inventory_pipeline_estimates rather than the arrival inference table.

5. How ML weights differ structurally from Dataform fwd_inventory_weight​

The directional difference is consistent across all dealer/model pairs profiled:

StatusML vs fwd directionTypical ratio range
1 (on lot)ML higher1.4× – 3.3×
2 (in transit)Mixed0.7× – 1.6×
5 (in production)ML lower (but improved post data-quality fix)0.6× – 3.0×
6 (on order)ML lower0.6× – 0.8×

Why ML overweights status 1 relative to Dataform: the survival function 1 − sale_CDF stays elevated for long-sitting on-lot vehicles because the sale model learns that vehicles surviving 200+ days without selling are hard to move — their future sale probability is low and flat. Dataform's fwd_inventory_weight applies a forward-looking discount that reduces weight for aged stock more aggressively.

Why ML underweights status 5/6 relative to Dataform: the Dataform formula heavily weights pipeline vehicles as future supply. At dealer 10742/XD, Dataform assigns ~55% of total weight to the 70 status 6 VINs; the ML model assigns ~41%. The ML dot-product weight for a status 6 VIN is bounded by how much of the VIN's on-lot window overlaps with the candidate curve — if the VIN is expected to arrive at day 57 and sell at day 81, the candidate curve (peaking days 28–55) has mostly declined by then, limiting the overlap.

Net effect on supply take-rates: because status 1 VINs dominate by count (156 vs 70 status 6), the ML model effectively shifts weight from pipeline inventory back to existing on-lot stock. Whether this is more or less correct than Dataform depends on the allocation decision being made — for predicting near-term supply competition for a candidate ordered today, the ML view (heavily weighted to on-lot stock) may actually be more conservative and appropriate.

6. Sale model bucket structure​

The sale model (pit_days_until_sale_inference) uses only 4 coarse buckets: 0-30, 31-60, 61-100, >100 (plus <0 for already-sold). The arrival model uses 12 finer buckets (0-6, 7-13, … ≥105). The coarse sale buckets produce step-function PDFs that are visually distinct from smooth curves. This is expected — the sale model was designed for speed and interpretability over granularity. The resulting on-lot probability curve 1 − sale_CDF has visible step inflections at days 30, 60, and 100 that are artefacts of bucket interpolation, not genuine inflections in the underlying distribution.