Implementation Plan: BMW ML CDF Batch Evaluation

Run the bmw_ml_cdf candidate selection pipeline over a representative sample of dealer/model pairs, write per-attribute supply take-rates and ranked recommendations to temporary BigQuery tables, and compare them against the current Dataform fwd_inventory_weight baseline.

---

1. Objectives

1. Supply take-rate comparison — for each dealer/model/attribute_type combination, compare ML CDF supply take-rates to pit_inv_take_rate_fwd_wgt from Dataform. This surfaces systematic directional biases (e.g. paint colours skewed toward pipeline vs on-lot stock).

2. Recommendation comparison — for each dealer/model, compare the top-N ranked candidates produced by bmw_ml_cdf to the current production recommendations. Measure overlap, rank shifts, and score deltas.

---

2. Dealer/model subset

Select a stratified sample to cover:

• Dealers with a high status-6 fraction (pipeline-heavy) — e.g. 41336/XD (n_status56=640)
• Dealers with a balanced mix — e.g. 10742/XD (156 on-lot, 70 on-order), 46891/XD
• A different model from XD — e.g. 50923/XG, 16815/XG

Exclude dealer/model pairs with no rows in pit_days_until_sale_inference (the sale inference table only covers BMW-specific markets; pairs outside its scope will raise MLCDFSupplyModel.compute() requires vehicle_sale_cdf to be non-empty).

Suggested starter set (all confirmed to have sale CDF data):

SCOPE_PAIRS = [
    ("10742", "XD"),
    ("10742", "XT"),
    ("46891", "XD"),
    ("46891", "XO"),
    ("50923", "XG"),
    ("16815", "XG"),
    ("20626", "XD"),
    ("36721", "XD"),
]

---

3. New script: scripts/run_bmw_batch_eval.py

3a. Outer loop

for dealer_id, model_id in SCOPE_PAIRS:
    scope = SelectionScope(dealer_id=dealer_id, product_id=model_id)
    config = _load_config("bmw_ml_cdf")
    pipeline = create_pipeline(config, scope)

    # 1. Compute supply take-rates for all attribute types
    take_rate_rows = _collect_take_rates(pipeline, dealer_id, model_id)

    # 2. Get top-N recommendations with metadata
    recommendations = pipeline.get_recommendations(rec_count=5, include_metadata=True)
    rec_rows = _collect_recommendations(recommendations, dealer_id, model_id)

    all_take_rates.extend(take_rate_rows)
    all_recs.extend(rec_rows)

# Write both tables to BQ
_write_bq(all_take_rates, TAKE_RATE_TABLE)
_write_bq(all_recs, REC_TABLE)

3b. _collect_take_rates()

Reuse the logic from compare_attribute_take_rates.py but iterate over all available attribute types (PAINT, MODEL_YEAR, OPTION_LINE, etc.) rather than a single one.

def _collect_take_rates(pipeline, dealer_id, model_id):
    supply_matrix = pipeline.context.get_supply_matrix()
    candidate_curve = pipeline.context.compute_supply_horizon(model_id, None)
    # ... same dot-product weight logic as compare_attribute_take_rates.py ...
    # Return a list of dicts: {dealer_id, model_id, attribute_type, attribute_code,
    #                          ml_supply_rate, n_vehicles}

3c. _collect_recommendations()

def _collect_recommendations(recommendations, dealer_id, model_id):
    rows = []
    for rec in recommendations:
        row = {
            "dealer_id": dealer_id,
            "model_id": model_id,
            "iteration": rec.iteration,
            "score": rec.score,
            "candidate_id": rec.candidate.candidate_id,
        }
        # Flatten module_scores (demand_supply_gap, etc.)
        row.update({f"score_{k}": v for k, v in rec.module_scores.items()})
        # Flatten key candidate attributes from metadata if available
        if rec.metadata:
            for attr in rec.metadata.candidate_attributes:
                row[f"attr_{attr.attribute_type.lower()}"] = attr.attribute_code
        rows.append(row)
    return rows

---

4. BigQuery output tables

Both tables live in a temporary dataset, e.g. bmw-prod-309117.candidate_selection_eval.

ml_cdf_supply_take_rates

column	type	notes
run_date	DATE	CURRENT_DATE() at script execution
dealer_id	STRING
model_id	STRING
attribute_type	STRING	PAINT, MODEL_YEAR, OPTION_LINE, …
attribute_code	STRING	e.g. P0C7A
attribute_name	STRING	human-readable label
n_vehicles	INT64	VINs in supply matrix with this attribute
ml_supply_rate	FLOAT64	ML CDF take-rate (dot-product weighted)
dataform_fwd_rate	FLOAT64	pit_inv_take_rate_fwd_wgt from Dataform
dataform_simple_rate	FLOAT64	pit_inv_take_rate (count-based)
delta	FLOAT64	ml_supply_rate − dataform_fwd_rate

Fetch Dataform rates in bulk for all dealer/model pairs up front to avoid per-pair round-trips:

SELECT dealer_id, model_id, attribute_type, attribute_code,
       pit_inv_take_rate_fwd_wgt AS dataform_fwd_rate,
       pit_inv_take_rate         AS dataform_simple_rate
FROM `bmw-prod-309117.dataform.pit_inv_take_rates_by_attribute`
WHERE (dealer_id, model_id) IN UNNEST(@scope_pairs)
  AND date = (SELECT MAX(date) FROM `bmw-prod-309117.dataform.pit_inv_take_rates_by_attribute`)

(Adjust table name once confirmed — look for the table that powers the current allocation pipeline's supply take-rate inputs.)

ml_cdf_recommendations

column	type	notes
run_date	DATE
dealer_id	STRING
model_id	STRING
iteration	INT64	1-indexed pick order
candidate_id	STRING
score	FLOAT64	composite weighted score
score_demand_supply_gap	FLOAT64	scorer sub-score
attr_paint	STRING	paint code of this candidate
attr_model_year	STRING
attr_option_line	STRING

Add a companion query that fetches the current production recommendations from the existing allocation output table (if one exists) and joins on (dealer_id, model_id, iteration) to compute rank overlap and score correlation.

---

5. Comparison analyses

Supply take-rate comparison

Run compare_attribute_take_rates.py in batch mode (extend it to accept a list of dealer/model pairs) or query the BQ output table directly:

-- Paints most consistently over/underweighted by ML vs Dataform
SELECT attribute_code, attribute_name,
       AVG(delta) AS mean_delta,
       STDDEV(delta) AS std_delta,
       COUNT(*) AS n_pairs
FROM `candidate_selection_eval.ml_cdf_supply_take_rates`
WHERE attribute_type = 'PAINT'
GROUP BY attribute_code, attribute_name
ORDER BY ABS(AVG(delta)) DESC

Key questions to answer:

• Are certain paint codes systematically over/underweighted across dealers?
• Does the direction of the delta correlate with orderstatus composition (pipeline-heavy dealers vs on-lot-heavy dealers)?
• For which attribute types is the ML rate closest to/furthest from Dataform?

Recommendation comparison

-- Rank overlap: is candidate_id at iteration=1 the same across methods?
SELECT
  m.dealer_id, m.model_id,
  m.candidate_id AS ml_rec,
  p.candidate_id AS prod_rec,
  m.candidate_id = p.candidate_id AS top1_match
FROM `candidate_selection_eval.ml_cdf_recommendations` m
JOIN `<production_recommendations_table>` p
  USING (dealer_id, model_id, iteration)
WHERE m.run_date = CURRENT_DATE()
  AND m.iteration = 1

---

6. Prerequisites before running

1. Confirm pit_days_until_sale_inference is current. Re-run the full chain from inventory_pit_status → pit_days_until_dealerarrival_features → pit_days_until_dealerarrival_inference → pit_days_until_sale_features → pit_days_until_sale_inference. Running only the final table leaves approx_ev_days_until_dealerarrival stale for newly-ordered VINs (see ML_CDF_LEARNINGS.md §3).

2. Verify attribute types available per dealer/model. The pipeline requires bmw_attribute_weights to be non-empty for scoring. Check that the attribute_type_weights table has rows for the attribute types you want to compare.

3. Create the BQ output dataset if it does not exist:

   bq --project_id=bmw-prod-309117 mk --dataset candidate_selection_eval

4. Confirm the production recommendations table name that the allocation pipeline currently writes to, so the recommendation comparison join can be wired up.

---

7. Open questions

• Which attribute types to compare? PAINT is confirmed to work. MODEL_YEAR and OPTION_LINE may also be informative. Confirm which attribute types have rows in attribute_type_weights before iterating.

• What is the production recommendations table? The comparison in §5 requires a known table name for current recommendations. Identify this before implementing the recommendation comparison query.

• Error handling for missing sale CDF data. Some dealer/model pairs in the scope list may have no rows in pit_days_until_sale_inference (e.g. for non-BMW markets or recently added dealers). The script should catch ValueError from MLCDFSupplyModel.compute() and log a skip rather than aborting the batch.

• Handling multiple attribute types for the same candidate. The current compare_attribute_take_rates.py operates on one attribute type at a time. The batch script should either run all attribute types per pipeline build (sharing the supply matrix computation) or explicitly loop with a single create_pipeline call per dealer/model and multiple take-rate passes.