Context Layer
The context layer provides scope data and access to supply state.
| Component | Role | Location |
|---|---|---|
ContextStore | Abstract interface for scope data and supply access | interfaces/context_store.py |
TakeRateContextStore | Stores named datasets and provides take-rate lookups | context_stores/take_rate_context_store.py |
DerivedLayerContextStore | Uses data already filtered to one dealer and model | context_stores/derived_layer_context_store.py |
DatasetLoader | Loads one named dataset | data_loaders/base.py |
SupplyModel | Computes and updates supply state | interfaces/supply_model.py |
SupplyTakeRateProvider | Provides and updates attribute supply take rates | interfaces/supply_take_rate_provider.py |
ContextStore
Scorers read candidates and scope data through ContextStore. Subclasses provide
the datasets and attribute lookups. A supply model is passed to the store at
construction.
candidate_pool contains candidates in dictionary-encoded DataFrames. The
selection strategy calls materialize() for the candidates it selects. See the
candidate pool design.
The store exposes these supply operations:
| Method | Effect |
|---|---|
get_supply_matrix() | Returns the supply model's cached matrix |
compute_supply_horizon(model_id, production_date) | Returns the candidate's daily supply estimates |
update_from_candidate(candidate, order_id, production_date) | Adds the candidate to inventory and updates the supply model |
restrictions provides attribute rules to scoring and selection. get(key)
provides named datasets in subclasses that support them.
Shared supply state
Supply changes after each confirmed order. Both context-store implementations
return a new store from with_scope(scope, candidate_ids). The new store has its
own scope and candidate subset. It shares inventory and supply state with the
original store, so later selections account for confirmed orders.
The candidate subset belongs to the new store. Narrowing it does not remove candidates from other stores that share the pool.
SupplyModel owns the supply matrix. DirectRateProvider owns direct-rate
updates. Context stores control candidate filtering and reuse cached datasets
when their scope keys match. This separation allows new production dates and
restrictions to be evaluated without rebuilding supply from the original
inventory or discarding confirmed orders.
TakeRateContextStore
The constructor accepts loaded datasets and normalized candidates.
from_config() loads datasets when the caller does not provide them, then uses
the configured normalizer to construct candidates.
| Input | Meaning |
|---|---|
supply_model | Computes the inventory supply matrix |
scope | Dealer, model, and dates for selection |
datasets | Loaded DataFrames keyed by dataset name |
candidates | A SharedCandidates value with a pool and optional candidate IDs |
inventory_candidates | Inventory vehicles keyed by order ID |
supply_take_rate_provider | Supplies attribute rates; defaults to MatrixDerivedProvider |
restrictions | Attribute restrictions for the scope |
get(key) returns a loaded dataset. get(key, schema) validates it against the
Patito model before returning a typed DataFrame.
get_demand_take_rates() reads the demand_take_rate column and filters by dealer
and model. get_weights_df() filters weights by model. Both results are cached.
with_scope() retains each cache only when its filtering keys still match.
get_contribution_margin() reads a lookup built from the
attribute_contribution_margin dataset. Missing values return 0.0.
get_supply_take_rates() calls the configured provider. When configured,
min_supply_rate_delta sets minimum rates after a confirmation. Rates within an
attribute type are then scaled down if their sum exceeds one.
update_from_candidate() updates the supply model and calls the provider's
update() method.
Dataset names used during assembly
TakeRateContextStore.from_config() uses these dataset names to construct its
components. They are configuration keys with specific meanings.
| Dataset name | Assembly behavior |
|---|---|
candidate_set and candidate_attributes | When both are present, join them on candidate_code_id before candidate normalization. |
candidates | Use this candidate input when the two-table combination is absent. |
inventory_candidates | Normalize inventory separately and index the resulting candidates by their inventory identifiers. |
restrictions | Convert rows to AttributeRestriction values for the context. |
supply_take_rate_provider | Construct DirectRateProvider from these rates. Without this dataset, the context defaults to MatrixDerivedProvider. |
The assembly method removes the inputs it consumes from the dataset mapping.
Other datasets remain available through get(key). When the method loads inputs
itself, it requires a loader named demand_take_rates. A caller supplying
DataFrames is responsible for providing the inputs its configured components
need.
The dataset field names the input consumed by the context; the loader's type
selects the implementation that fetches it. For example,
type: bmw_direct_supply_rates with dataset: supply_take_rate_provider loads
BMW rates and selects the direct-rate provider. Renaming that dataset changes
assembly behavior even when the returned columns are unchanged.
Dataset loading
Each DatasetLoader defines configuration fields for one source query.
DatasetAssembler calls load_bulk() on each loader and returns DataFrames keyed
by dataset name. SQL, GraphQL queries, and source column mappings belong to the
loaders. Context stores receive the resulting DataFrames.
See Dataset Loading for BigQuery and GCS usage.
IcebergContextAssembler
IcebergTableMap declares the tables needed by DerivedLayerContextStore.
Each IcebergTableRef specifies:
- The table namespace, and
extra_namespaceswhose rows a LazyFrame scan appends, such asfeaturization/syntheticafterfeaturization/inventory. The loaders that scan with pyiceberg, for models, dealers and status history, read the namespace alone. - Columns to select and rename.
- Columns to filter using the resolved scope values.
The assembler filters Iceberg scans by partition date and scope values. It loads
the inputs and constructs a DerivedLayerContextStore.
Inventory and arrival estimates
The simulation input needs these inventory columns after selection and renaming:
inventory_idandvin.model_idandstatus_code.is_on_lotandinferred_sold_date.
prepare_sim_inventory() copies inventory_id to vin and adds the configured
grouping columns. The Stellantis configuration renames is_active to
is_on_lot.
Days-in-status is not read from consolidated inventory: the assembler reads the
trailing days_in_status_lookback_days partitions of inventory, narrowed to
(inventory_id, status_code, _partition_date), and derives each vehicle's days in
its current status per inventory_id. The arrival estimator subtracts that from the
learned days-to-lot, capped so the offset never exceeds the lookback window.
SimulationSupplyModel joins inventory to the status_code_days table on
(model_id, status_code). The table provides the estimated days until arrival.
The model uses zero for vehicles already on the lot. Other vehicles require an
estimate and a null inferred_sold_date to enter the projection.
Check estimate coverage for the whole partition before filtering by dealer and
model. The scoped simulation disables this check. For partition estimates,
compute_status_code_estimates() defaults to a minimum coverage of 0.75 for
in-transit vehicles. Setting required_coverage=None disables the check.
Tables shared across a run
RUN_LEVEL_TABLES contains demand_take_rates, attribute_weights and
sold_vehicles. load_run_datasets_split() loads these tables for a dealer and
encodes them as Parquet. It splits tables by model when they contain the model
column. It keeps only the rows of the models that the workflow passes in, which are
the models of the run's scopes. Other tables remain encoded as a whole.
sold_vehicles has no dealer scope filter, so it holds the vehicles of the run's
models at every dealer.
RunLevelDatasets.narrowed() reads the encoded data for a scope and applies its
filters. It raises KeyError if a required filter column is missing. This
prevents a scope from receiving another model's rows.
Sharing these tables avoids loading them separately for each scope. A table in
RUN_LEVEL_TABLES must fit in memory for one dealer's run. The assembler loads
other tables separately for each scope.
load_run_datasets_split() also reads the candidates table once for the run and
splits its rows by model, under CANDIDATES. A scope reads only its own model's
rows. The table has no dealer column, so the run reads every model's candidates.
It keeps only the rows of the run's models. The read uses pyarrow on the
partition's data files and does not go through scan_iceberg. It matches columns by Parquet column name, so it does not find a
column renamed in Iceberg. It raises on a data file that has delete files.
load_run_datasets_split() also computes days-in-status for the dealer's whole
inventory and encodes it as a whole, under DAYS_IN_STATUS. The history it reads
is not sorted by dealer or model, so a filter on either skips no row groups. Reading
it for one model costs as much as reading it for the whole dealer. The arrival
estimator joins the frame on inventory_id, so each scope uses the dealer-wide
frame unchanged. When the run datasets hold no DAYS_IN_STATUS frame, the assembler
computes days-in-status for the scope from the history.
Iceberg configurations accept these shared inputs through run_datasets.
Configurations using DatasetAssembler accept loaded DataFrames through
datasets.
SupplyModel
SupplyModel produces daily supply estimates for inventory vehicles. The meaning
of each estimate depends on the model. MatrixSupplyModel and
SimulationSupplyModel estimate the probability that a vehicle is on the lot.
MLCDFSupplyModel uses the probability that it has not sold, including vehicles
that have not arrived. See the
MLCDFSupplyModel design
for its sale-CDF calculation.
The model caches the initial matrix and updates supply state after each
confirmation. The matrix's horizon_days gives the number of daily columns.
| Method | Effect |
|---|---|
compute(context) | Builds the initial supply matrix |
get_supply_matrix(context) | Computes the matrix on first access and caches it |
compute_on_lot_curve(model_id, production_date, scope_date) | Returns a candidate's daily supply estimates in models that support this calculation |
update_from_candidate(candidate, order_id, production_date) | Updates the model after a selection |
reset() | Clears the cached matrix |
Separate supply models allow each configuration to choose its inventory projection without changing the context-store interface.
MatrixSupplyModel
MatrixSupplyModel uses Weibull days-on-lot distributions and Burr Type XII
arrival distributions. It chooses the calculation by inventory status:
| Status | Calculation |
|---|---|
ON_LOT | Conditional Weibull survival based on days already on the lot |
IN_PRODUCTION | Arrival probability after the expected production date, combined with days-on-lot survival |
IN_TRANSIT | Conditional arrival probability based on days already in transit, combined with days-on-lot survival |
AT_PROCESSING | Uniform arrival probability over the remaining processing period, combined with days-on-lot survival |
MatrixDerivedProvider uses the candidate's on-lot curve to weight inventory
that is expected to be available at the same time.
SupplyTakeRateProvider
TakeRateContextStore uses a provider for attribute supply rates.
get_supply_take_rates() returns rates keyed by (attribute_type, attribute_value).
update() runs after a candidate is confirmed.
Supply models and rate providers are separate so a configuration can change its
inventory projection without changing its scorers. MatrixDerivedProvider can
use either MatrixSupplyModel or MLCDFSupplyModel. DirectRateProvider uses
supplied rates with NoneSupplyModel and does not request a supply matrix.
MatrixDerivedProvider
For each inventory vehicle, MatrixDerivedProvider computes the dot product of
its daily supply estimates and the candidate's daily supply estimates.
This value weights that vehicle's contribution to supply take rates.
The provider scales rates by how much the most relevant inventory vehicle overlaps the candidate.
When exactly one vehicle in the supply matrix matches an inventory candidate by
order_id, the provider caps each rate. Matrix vehicles that match no inventory
candidate do not count toward that condition.
The provider reads the updated matrix after each confirmation. Its update()
method does not maintain separate rate state.
DerivedLayerContextStore supply rates
DerivedLayerContextStore uses no SupplyTakeRateProvider.
SimulationSupplyModel.supply_matrix_with_new_order() simulates current supply plus the
order being scored. That order has no code and arrives at the arrival offset.
supply_vehicle_weights() weights each vehicle in that matrix, the order included, by
its on-lot probability averaged over the horizon's days, each day weighted by the
probability that the order being scored is on the lot that day:
weight(v) = Σ_d p_v(d) · p_order(d) / Σ_d p_order(d)
p_v(d) is the probability that vehicle v is on the lot on day d, and p_order(d) is
that probability for the order being scored.
DerivedLayerContextStore.get_supply_vehicle_weights() returns these weights.
A vehicle's weight grows with the probability that it is on the lot on the same days as that order. A vehicle that sells before the order arrives has a weight of zero. A vehicle that arrives after the order has likely sold has a small weight. A code's rate is the weight of the vehicles that have the code divided by the total weight.
A code's rate is a share of supply as it would stand after the next order, so a code already in supply is weighed against the supply that order would leave. The dealer wants chassis A on 84% of cars and B on 16%, and supply holds one car with A:
| Supply rate of A | Gap A | Gap B | Next order |
|---|---|---|---|
| 1 / 1 = 1.0 | 0.84 − 1.0 = −0.16 | 0.16 | B, making supply 50% A |
| 1 / (1 + 1) = 0.5 | 0.84 − 0.5 = 0.34 | 0.16 | A, making supply 100% A |
With two cars, 100% A is closer to the dealer's 84% than 50% is. The added order's weight matters less as supply grows, so a dealer with a large pipeline sees almost the plain share.
DirectRateProvider
DirectRateProvider stores supplied rates and inventory counts. After each
confirmation, it updates every tracked attribute:
new_rate = (inv * old_rate + c) / (inv + 1)
new_inv = inv + 1
inv is the inventory count used as the historical sample size. c is one when
the selected candidate has the attribute and zero otherwise. Each selection adds
one observation without storing the full selection history.