Skip to main content

Context Layer

The context layer provides scope data and access to supply state.

ComponentRoleLocation
ContextStoreAbstract interface for scope data and supply accessinterfaces/context_store.py
TakeRateContextStoreStores named datasets and provides take-rate lookupscontext_stores/take_rate_context_store.py
DerivedLayerContextStoreUses data already filtered to one dealer and modelcontext_stores/derived_layer_context_store.py
DatasetLoaderLoads one named datasetdata_loaders/base.py
SupplyModelComputes and updates supply stateinterfaces/supply_model.py
SupplyTakeRateProviderProvides and updates attribute supply take ratesinterfaces/supply_take_rate_provider.py

ContextStore​

Scorers read candidates and scope data through ContextStore. Subclasses provide the datasets and attribute lookups. A supply model is passed to the store at construction.

candidate_pool contains candidates in dictionary-encoded DataFrames. The selection strategy calls materialize() for the candidates it selects. See the candidate pool design.

The store exposes these supply operations:

MethodEffect
get_supply_matrix()Returns the supply model's cached matrix
compute_supply_horizon(model_id, production_date)Returns the candidate's daily supply estimates
update_from_candidate(candidate, order_id, production_date)Adds the candidate to inventory and updates the supply model

restrictions provides attribute rules to scoring and selection. get(key) provides named datasets in subclasses that support them.

Shared supply state​

Supply changes after each confirmed order. Both context-store implementations return a new store from with_scope(scope, candidate_ids). The new store has its own scope and candidate subset. It shares inventory and supply state with the original store, so later selections account for confirmed orders.

The candidate subset belongs to the new store. Narrowing it does not remove candidates from other stores that share the pool.

SupplyModel owns the supply matrix. DirectRateProvider owns direct-rate updates. Context stores control candidate filtering and reuse cached datasets when their scope keys match. This separation allows new production dates and restrictions to be evaluated without rebuilding supply from the original inventory or discarding confirmed orders.

TakeRateContextStore​

The constructor accepts loaded datasets and normalized candidates. from_config() loads datasets when the caller does not provide them, then uses the configured normalizer to construct candidates.

InputMeaning
supply_modelComputes the inventory supply matrix
scopeDealer, model, and dates for selection
datasetsLoaded DataFrames keyed by dataset name
candidatesA SharedCandidates value with a pool and optional candidate IDs
inventory_candidatesInventory vehicles keyed by order ID
supply_take_rate_providerSupplies attribute rates; defaults to MatrixDerivedProvider
restrictionsAttribute restrictions for the scope

get(key) returns a loaded dataset. get(key, schema) validates it against the Patito model before returning a typed DataFrame.

get_demand_take_rates() reads the demand_take_rate column and filters by dealer and model. get_weights_df() filters weights by model. Both results are cached. with_scope() retains each cache only when its filtering keys still match.

get_contribution_margin() reads a lookup built from the attribute_contribution_margin dataset. Missing values return 0.0.

get_supply_take_rates() calls the configured provider. When configured, min_supply_rate_delta sets minimum rates after a confirmation. Rates within an attribute type are then scaled down if their sum exceeds one.

update_from_candidate() updates the supply model and calls the provider's update() method.

Dataset names used during assembly​

TakeRateContextStore.from_config() uses these dataset names to construct its components. They are configuration keys with specific meanings.

Dataset nameAssembly behavior
candidate_set and candidate_attributesWhen both are present, join them on candidate_code_id before candidate normalization.
candidatesUse this candidate input when the two-table combination is absent.
inventory_candidatesNormalize inventory separately and index the resulting candidates by their inventory identifiers.
restrictionsConvert rows to AttributeRestriction values for the context.
supply_take_rate_providerConstruct DirectRateProvider from these rates. Without this dataset, the context defaults to MatrixDerivedProvider.

The assembly method removes the inputs it consumes from the dataset mapping. Other datasets remain available through get(key). When the method loads inputs itself, it requires a loader named demand_take_rates. A caller supplying DataFrames is responsible for providing the inputs its configured components need.

The dataset field names the input consumed by the context; the loader's type selects the implementation that fetches it. For example, type: bmw_direct_supply_rates with dataset: supply_take_rate_provider loads BMW rates and selects the direct-rate provider. Renaming that dataset changes assembly behavior even when the returned columns are unchanged.

Dataset loading​

Each DatasetLoader defines configuration fields for one source query. DatasetAssembler calls load_bulk() on each loader and returns DataFrames keyed by dataset name. SQL, GraphQL queries, and source column mappings belong to the loaders. Context stores receive the resulting DataFrames.

See Dataset Loading for BigQuery and GCS usage.

IcebergContextAssembler​

IcebergTableMap declares the tables needed by DerivedLayerContextStore. Each IcebergTableRef specifies:

  • The table namespace, and extra_namespaces whose rows a LazyFrame scan appends, such as featurization/synthetic after featurization/inventory. The loaders that scan with pyiceberg, for models, dealers and status history, read the namespace alone.
  • Columns to select and rename.
  • Columns to filter using the resolved scope values.

The assembler filters Iceberg scans by partition date and scope values. It loads the inputs and constructs a DerivedLayerContextStore.

Inventory and arrival estimates​

The simulation input needs these inventory columns after selection and renaming:

  • inventory_id and vin.
  • model_id and status_code.
  • is_on_lot and inferred_sold_date.

prepare_sim_inventory() copies inventory_id to vin and adds the configured grouping columns. The Stellantis configuration renames is_active to is_on_lot.

Days-in-status is not read from consolidated inventory: the assembler reads the trailing days_in_status_lookback_days partitions of inventory, narrowed to (inventory_id, status_code, _partition_date), and derives each vehicle's days in its current status per inventory_id. The arrival estimator subtracts that from the learned days-to-lot, capped so the offset never exceeds the lookback window.

SimulationSupplyModel joins inventory to the status_code_days table on (model_id, status_code). The table provides the estimated days until arrival. The model uses zero for vehicles already on the lot. Other vehicles require an estimate and a null inferred_sold_date to enter the projection.

Check estimate coverage for the whole partition before filtering by dealer and model. The scoped simulation disables this check. For partition estimates, compute_status_code_estimates() defaults to a minimum coverage of 0.75 for in-transit vehicles. Setting required_coverage=None disables the check.

Tables shared across a run​

RUN_LEVEL_TABLES contains demand_take_rates, attribute_weights and sold_vehicles. load_run_datasets_split() loads these tables for a dealer and encodes them as Parquet. It splits tables by model when they contain the model column. It keeps only the rows of the models that the workflow passes in, which are the models of the run's scopes. Other tables remain encoded as a whole. sold_vehicles has no dealer scope filter, so it holds the vehicles of the run's models at every dealer.

RunLevelDatasets.narrowed() reads the encoded data for a scope and applies its filters. It raises KeyError if a required filter column is missing. This prevents a scope from receiving another model's rows.

Sharing these tables avoids loading them separately for each scope. A table in RUN_LEVEL_TABLES must fit in memory for one dealer's run. The assembler loads other tables separately for each scope.

load_run_datasets_split() also reads the candidates table once for the run and splits its rows by model, under CANDIDATES. A scope reads only its own model's rows. The table has no dealer column, so the run reads every model's candidates. It keeps only the rows of the run's models. The read uses pyarrow on the partition's data files and does not go through scan_iceberg. It matches columns by Parquet column name, so it does not find a column renamed in Iceberg. It raises on a data file that has delete files.

load_run_datasets_split() also computes days-in-status for the dealer's whole inventory and encodes it as a whole, under DAYS_IN_STATUS. The history it reads is not sorted by dealer or model, so a filter on either skips no row groups. Reading it for one model costs as much as reading it for the whole dealer. The arrival estimator joins the frame on inventory_id, so each scope uses the dealer-wide frame unchanged. When the run datasets hold no DAYS_IN_STATUS frame, the assembler computes days-in-status for the scope from the history.

Iceberg configurations accept these shared inputs through run_datasets. Configurations using DatasetAssembler accept loaded DataFrames through datasets.

SupplyModel​

SupplyModel produces daily supply estimates for inventory vehicles. The meaning of each estimate depends on the model. MatrixSupplyModel and SimulationSupplyModel estimate the probability that a vehicle is on the lot. MLCDFSupplyModel uses the probability that it has not sold, including vehicles that have not arrived. See the MLCDFSupplyModel design for its sale-CDF calculation.

The model caches the initial matrix and updates supply state after each confirmation. The matrix's horizon_days gives the number of daily columns.

MethodEffect
compute(context)Builds the initial supply matrix
get_supply_matrix(context)Computes the matrix on first access and caches it
compute_on_lot_curve(model_id, production_date, scope_date)Returns a candidate's daily supply estimates in models that support this calculation
update_from_candidate(candidate, order_id, production_date)Updates the model after a selection
reset()Clears the cached matrix

Separate supply models allow each configuration to choose its inventory projection without changing the context-store interface.

MatrixSupplyModel​

MatrixSupplyModel uses Weibull days-on-lot distributions and Burr Type XII arrival distributions. It chooses the calculation by inventory status:

StatusCalculation
ON_LOTConditional Weibull survival based on days already on the lot
IN_PRODUCTIONArrival probability after the expected production date, combined with days-on-lot survival
IN_TRANSITConditional arrival probability based on days already in transit, combined with days-on-lot survival
AT_PROCESSINGUniform arrival probability over the remaining processing period, combined with days-on-lot survival

MatrixDerivedProvider uses the candidate's on-lot curve to weight inventory that is expected to be available at the same time.

SupplyTakeRateProvider​

TakeRateContextStore uses a provider for attribute supply rates. get_supply_take_rates() returns rates keyed by (attribute_type, attribute_value). update() runs after a candidate is confirmed.

Supply models and rate providers are separate so a configuration can change its inventory projection without changing its scorers. MatrixDerivedProvider can use either MatrixSupplyModel or MLCDFSupplyModel. DirectRateProvider uses supplied rates with NoneSupplyModel and does not request a supply matrix.

MatrixDerivedProvider​

For each inventory vehicle, MatrixDerivedProvider computes the dot product of its daily supply estimates and the candidate's daily supply estimates. This value weights that vehicle's contribution to supply take rates.

The provider scales rates by how much the most relevant inventory vehicle overlaps the candidate.

When exactly one vehicle in the supply matrix matches an inventory candidate by order_id, the provider caps each rate. Matrix vehicles that match no inventory candidate do not count toward that condition.

The provider reads the updated matrix after each confirmation. Its update() method does not maintain separate rate state.

DerivedLayerContextStore supply rates​

DerivedLayerContextStore uses no SupplyTakeRateProvider. SimulationSupplyModel.supply_matrix_with_new_order() simulates current supply plus the order being scored. That order has no code and arrives at the arrival offset. supply_vehicle_weights() weights each vehicle in that matrix, the order included, by its on-lot probability averaged over the horizon's days, each day weighted by the probability that the order being scored is on the lot that day:

weight(v) = Σ_d p_v(d) · p_order(d) / Σ_d p_order(d)

p_v(d) is the probability that vehicle v is on the lot on day d, and p_order(d) is that probability for the order being scored. DerivedLayerContextStore.get_supply_vehicle_weights() returns these weights.

A vehicle's weight grows with the probability that it is on the lot on the same days as that order. A vehicle that sells before the order arrives has a weight of zero. A vehicle that arrives after the order has likely sold has a small weight. A code's rate is the weight of the vehicles that have the code divided by the total weight.

A code's rate is a share of supply as it would stand after the next order, so a code already in supply is weighed against the supply that order would leave. The dealer wants chassis A on 84% of cars and B on 16%, and supply holds one car with A:

Supply rate of AGap AGap BNext order
1 / 1 = 1.00.84 − 1.0 = −0.160.16B, making supply 50% A
1 / (1 + 1) = 0.50.84 − 0.5 = 0.340.16A, making supply 100% A

With two cars, 100% A is closer to the dealer's 84% than 50% is. The added order's weight matters less as supply grows, so a dealer with a large pipeline sees almost the plain share.

DirectRateProvider​

DirectRateProvider stores supplied rates and inventory counts. After each confirmation, it updates every tracked attribute:

new_rate = (inv * old_rate + c) / (inv + 1)
new_inv = inv + 1

inv is the inventory count used as the historical sample size. c is one when the selected candidate has the attribute and zero otherwise. Each selection adds one observation without storing the full selection history.