Skip to main content

Take Rates

Computes per-attribute take rates and KPIs from featurized inventory, then blends them across geographic layers to produce smooth dealer-level estimates.

Attribute Statistics​

Produces per-partition take rates and metric KPIs for vehicle attribute values (paint colors, options, packages, …), grouped by configurable dimensions like dealer or carline.

Filter, weight, and metric columns arrive per-vehicle from consolidated/inventory — see ConsolidatedComponent. Columns that depend on the partition date, or that combine columns from more than one input, are computed here via derived_fields.

Inputs​

  • Vehicle featurized — <oem>/<market>/featurization/inventory: one row per (inventory_id, attribute_type, attribute_value) with identity columns model_id, dealer_id, is_inventory. The statistics use only the rows with is_inventory true. Featurization Groups describes how attributes are encoded. Every column other than the attribute fields (attribute_type, attribute_value, attribute_class) has one value per inventory_id. The component reads those columns once per vehicle and runs the inventory join, joins and derived_fields on one row per vehicle. Synthetic candidates explains why it holds no synthetic candidates.
  • Inventory — always <oem>/<market>/consolidated/inventory; not configurable. Supplies filter, weight, and metric columns plus any per-vehicle dimension columns referenced by aggregations.
  • joins — Reference tables joined onto each vehicle to expose dimension columns used by aggregations (e.g. dealer → country, model → carline).
  • derived_fields — Polars-SQL expressions evaluated after inventory and joins complete, applied sequentially so a later expression can reference an earlier one. Use these to combine columns coming from different inputs (for example, abnormal_days_on_lot = days_on_lot - predicted_days_on_lot). Any expr may contain {partition_date}, replaced at materialization time with the current partition key.

Aggregations​

An aggregation defines the GROUP BY dimensions for one output asset. Each aggregation produces a separate Dagster asset.

aggregations:
- key: country_by_carline
group_by: [carline_id, model_code, model_year, trim_code]
- key: country_by_dealer
group_by: [dealer_id]
- key: country_by_dealer_carline
group_by: [dealer_id, carline_id]
FieldDescription
keyLeaf segment of the asset key. Written to <oem>/attribute_stats/<key>
group_byColumns to group over. Must be present on the joined frame

Statistics​

A statistic defines one vehicle sub-population and what to measure. Multiple statistics are computed together and stored as columns in the same aggregation asset.

statistics:
- label: sales
filter_column: is_sold_in_window
metric_columns: [days_on_lot]

- label: sales_hl90
filter_column: is_sold_in_window
half_life: 90
metric_columns: [days_on_lot]

- label: inventory
filter_column: is_in_inventory
metric_columns: [days_on_lot]
FieldRequiredDescription
labelYesColumn prefix for output columns (e.g. sales, sales_hl90)
filter_columnNoBoolean column. Only rows where it is true are included; excluded rows are weighted 0.0. Omitted → all rows
weight_columnNoFloat weight applied uniformly to take rate and all metric aggregations. Omitted → 1.0
half_lifeNoThe half-life in days: 30, 90 or 180. Weights each vehicle by its recency weight. A vehicle not sold in the window has a null weight, so the statistic counts only vehicles sold in the window. With weight_column set too, a vehicle's weight is the product of the two
metric_columnsNoNumeric columns to compute weighted avg, diff, and z for
diff_center_byNoColumns, such as model_year, that divide each group into cells for the diff. A cell holds the vehicles that share a group and a diff_center_by value. Each vehicle is compared with the mean of its cell. See Diffs within a model year. Needs metric_columns

Every column a statistic names must exist on the joined frame — either from an input or from a derived_fields entry. A statistic with half_life also needs is_sold_in_window and a Date column inferred_sold_date. The component adds the hl<N>_weight columns after derived_fields, so a derived field cannot read one. If the frame already has the hl<N>_weight column of a statistic's half_life, the component raises ValueError. A named column that is absent is an error, not a fallback: missing metric_columns raise ValueError, and a missing filter_column or weight_column raises from Polars at collect time.

note

A single weight_column applies to both the take rate and the metric aggregations, so all output columns for a given statistic reflect the same "perspective" on the market. For two HL variants, define two statistics.

Dealer-modifiable weight​

Audi and Mercedes derive dealer_modifiable_weight from the inventory column is_dealer_modifiable. Each sales statistic that demand_take_rates projects sets it as its weight_column. A sale the dealer could not change or refuse counts 0.25; any other sale counts 1.

A sale the dealer could not change or refuse shows less of what the dealer's customers want, so it has less effect on the dealer's take rate. It also adds less to the statistic's group_nobs. The geographic smoother therefore gives more weight to the neighborhood and country rates of a dealer with many such sales. The inventory statistic that the smoother blends in as the inventory prior counts every vehicle in full, so such a vehicle counts 1 while it is in stock.

A weight of 0 would give the take rate of dealer-modifiable sales alone, and 1 the take rate of every sale.

Recency weights​

A half-life weight decays a sale's contribution continuously with its age, so recent sales dominate the take rate without discarding older ones. Weight halves every HL days:

hl30_weight  = 2^(−days_since_sale / 30)   # aggressive recency; 30 days ago → 0.5
hl90_weight = 2^(−days_since_sale / 90) # moderate recency; 90 days ago → 0.5
hl180_weight = 2^(−days_since_sale / 180) # broad window; 180 days ago → 0.5

HalfLife in ai_dagster.derived.recency lists the half-lives a statistic can name with half_life. The age is measured from inferred_sold_date to the partition date. A vehicle whose is_sold_in_window is false weighs NULL.

A statistic's filter_column may narrow the population further than the sold window. For example, a statistic with half_life: 90 may filter to vehicles of recent model years only.

Model-year weighting​

A statistic with model_year_weighted: true counts a vehicle of its base model's newest model year (as max_my_models selects it from consolidated/models) at weight 1. It counts a vehicle of any other model year at a weight per base model and attribute type, which falls as the type's national mix changed between the two:

  • d is the share of the type's vehicles that would have to change value to turn the other model years' mix into the newest one's: half the sum of the absolute differences of each value's two shares. The mixes pool the vehicles of mix_statistics; NO_<type> counts as a value, so an option that becomes standard changes the mix. d is 0 when either mix is empty.
  • d is scaled by n / (n + shrinkage_vehicles), where n is the number of the newest model year's vehicles with a row of the type.
  • The weight is 1 up to full_weight_below, 0 from zero_weight_above, and linear between.

This weight multiplies the vehicle's recency and column weights. A vehicle whose base model and type have no weight, such as one with no base model, counts 1. The group size therefore differs by attribute type. A value only older vehicles have gets a rate of 0 once its type's weight is 0, and a group of only older vehicles gets a NaN rate and a group size of 0 in that type. A model-year weighted statistic takes no metric columns.

statistics:
- label: sales_hl90_my_weighted
filter_column: is_sold_recent_my
half_life: 90
model_year_weighted: true
model_year_weighting:
mix_statistics: [sales, inventory]
full_weight_below: 0.05
zero_weight_above: 0.3
shrinkage_vehicles: 50

Audi and Stellantis add sales_my_weighted, sales_hl90_my_weighted and inventory_my_weighted with these settings. Their smoother blends the two sales statistics, with inventory_statistic: inventory_my_weighted as the inventory prior.

Output columns​

Each asset contains the grouping columns, then attribute_type / attribute_value, then a block of {label}_* columns for every statistic:

ColumnDescription
{label}_take_rateShare of group weight carried by vehicles with this attribute
{label}_group_nobsTotal weight (or count) in the group — the take-rate denominator
{label}_avg_{col}Weighted mean of metric column for vehicles carrying this attribute
{label}_{col}_diffAttribute mean minus group mean (negative → faster sales). With diff_center_by, each vehicle is compared with the mean of its cell instead of the group mean
{label}_{col}_zDiff divided by group weighted std. With diff_center_by, the std is computed within cells. Null when std is zero or null

Each asset carries one take_rate_sums_{label} asset check per statistic.

Diffs within a model year​

AttributeTurnScorer in candidate selection reads sales_abnormal_days_on_lot_diff. Without diff_center_by, the diff compares an attribute with the mean of its whole group, such as a base model. In the first months of a model year, only that year's fast sellers have sold, so the year's sold vehicles have a low mean abnormal days on lot. An attribute offered only on the new model year has that offset in its diff and looks like a fast seller.

With diff_center_by: [model_year], the diff is computed in two steps:

e_i  = metric_i − mean(metric | group, model_year of i)     weighted by the statistic's weight
diff = Σ w_i e_i / Σ w_i over the attribute's vehicles

The model-year offset then cancels for every attribute. An attribute sold on several model years gets the weighted average of its diffs within each model year. {label}_avg_{col} and the take rate are unchanged.

{label}_{col}_z divides the diff by the weighted standard deviation within cells, so the spread between model years does not enter the standard deviation.

Diffs computed from few vehicles are not shrunk toward 0. The geographic smoother already blends dealer and neighborhood diffs with the country diff. Shrinking each layer toward 0 before that blend would pull a dealer's diff toward 0, not toward the country value.

diff_center_by removes the offset between model years only. Within a model year, vehicles that arrived later have had less time to sell, so their attributes still look faster.

Memory​

The attribute group-by holds one hash-table entry per output row until it has read all of its input, so its memory grows with the number of output rows. A dealer-level aggregation has few vehicles per group and nearly as many output rows as attribute rows.

The component splits the groups into partitions by a hash of the group key. The partition count is the estimated output rows of all groups divided by _ROWS_PER_PARTITION, rounded up. To pick the count, the component scans the featurized input once more for its distinct (attribute_type, attribute_value) pairs and its attribute row count. It estimates a group's output rows as the smaller of two numbers:

  • the group's vehicle count times the mean attribute rows per vehicle;
  • the number of distinct pairs.

The estimate does not bound a partition's output rows. A group with more attribute rows per vehicle than the mean exceeds its estimate. A single group always falls in one partition, however large it is.

Each partition scans the featurized input for its own vehicles by inventory_id. One scan shared by every partition would become a cached subplan that Polars holds in memory.

The Polars streaming engine runs the group-bys of all partitions at the same time, so partitioning does not bound memory. Partitioning keeps each hash table small, which lowers peak memory compared with one group-by over all groups.

The component collects the vehicle frame, one row per vehicle and group, before it builds the partitions. Each partition plan joins its own slice of that frame. A vehicle frame left lazy would be a subplan that every partition shares, which Polars caches in memory as well.

The component computes each weight × metric product and each conditional weight as a column of the vehicle frame before the attribute join. The group-by then computes only plain column sums. A filter(...).sum() expression inside the aggregation would make the streaming group-by buffer its input rows.

The asset returns the result as a lazy frame. The Iceberg IO manager and the take_rate_sums checks each read it, so the computation runs twice. All take_rate_sums checks share one read.

Weighting scheme​

When weight_column or half_life is set:

  • Take rate numerator: sum(weight) over vehicles with the attribute.
  • Take rate denominator: sum(weight) over all vehicles in the group.
  • Weighted metric mean: sum(weight × metric) / sum(weight where metric not null).
  • Weighted variance: sum(w × x²) / sum(w) − (sum(w × x) / sum(w))².

Geographic Smoothing​

GeographicSmootherComponent produces a single dealer-level asset that blends multiple Attribute Statistics layers into one smoothed estimate per (dealer_id, attribute_type, attribute_value). Dealers with few sales borrow strength from coarser geographies — neighborhood, then country — proportionally to sample size. The pattern replicates attribute_stats_smart from CDP.

Shrinkage blend​

blend selects the method by its required method key: divisor (below) or nested_pseudo_count (below). Each method takes its own layer fields, and a field of the other method fails the defs.yaml load.

For each (dealer_id, attribute_type, attribute_value) cell and each declared field:

raw_weight_i = eval(field.weight_expr on layer_i stats)
weight_i = raw_weight_i / divisor_i
if raw_weight_i > min_obs AND field value IS NOT NULL
= 0 otherwise

blended = Σ(weight_i × value_i) / Σ(weight_i) [null when Σ = 0]

The weight_expr is a Polars-SQL expression — typically the group observation count or the attribute-level count (group_nobs * take_rate). The divisor normalizes weights so coarser layers (with more data) contribute equally only when the dealer layer has too few observations.

Attribute statistics store only observed values. Set is_group_weighted: true on take rates: every value of a (group, attribute_type) then gets the group's weight, which weight_expr (e.g. sales_group_nobs) must give, and a value without a row counts as 0. The warning check take_rate_sums_<label> verifies that their rates sum to 1 per attribute type.

Layers​

Layers are ordered finest → coarsest. Every dealer in the assembled joins dimension frame appears in the output for every (attribute_type, attribute_value) pair observed in any layer. A layer contributes weight=0 for dealers it has no data for.

Layerdivisorjoin_onMeaning
dealer1[dealer_id]Dealer's own sales — full weight per observation
neighborhood10[neighborhood]Dealers in the same neighborhood. Gets most of the weight when the dealer has few sales of the base model
country7000[country]All dealers in the market. Gets ~2.5% of the sales_take_rate weight for the median dealer on the OEM's highest-volume base models, matching the national level's weight in BMW's take-rate blend. Each OEM sets its own divisor from its country sales_group_nobs.

The joins list assembles the dimension frame mapping dealer_id to coarser geographies. The component joins coarser-layer stats onto that frame to fan them out to dealer granularity, so stat assets never need to carry a dealer_id column.

The inventory prior​

Setting inventory_divisor on a layer makes it contribute a second, weaker signal: the same asset's take rate of the inventory_statistic (default inventory), weighted by its group size and blended into the fields with is_group_weighted: true at that divisor. Days-on-lot fields are untouched.

A large divisor (e.g. 5000) makes the prior a weak always-on fallback. Its rate passes through only where every sales layer was excluded for want of observations, which is what keeps a dealer with no sales history from reporting a null rate.

Nested pseudo-count blend​

method: nested_pseudo_count replaces the divisor weights. layers lists the layers finer than coarsest_layer, finest first. Each of them shrinks toward the blend of the layers coarser than it, with a fixed pseudo_count k:

neighborhood* = (n_nbhd · r_nbhd + k_nbhd · r_country) / (n_nbhd + k_nbhd)
dealer* = (n_dealer · r_dealer + k_dealer · neighborhood*) / (n_dealer + k_dealer)

Each layer's share of the weight is therefore:

LayerShare
dealern_d / (n_d + k_d)
neighborhoodk_d / (n_d + k_d) · n_n / (n_n + k_n)
countryk_d / (n_d + k_d) · k_n / (n_n + k_n)

k has the units of the layer's count n. For a take rate, both count vehicles of the group. A dealer with k_dealer vehicles of a base model weighs the same as the neighborhood and country together. The country's share is what the dealer and the neighborhood both leave, so it is large only where both have small counts.

  • Every layer in layers sets pseudo_count. coarsest_layer takes the weight the finer layers leave and sets neither pseudo_count nor max_share.
  • inventory_weight (λ) adds each layer's inventory_statistic group size, times λ, to the layer's count n for the fields with is_group_weighted: true. The layer's rate r is then the mean of its sales and inventory rates, weighted by those counts. Without inventory_weight, the blend reads no inventory.
  • inventory_gate restricts the inventory prior to the (base_model_id, attribute_type) pairs with few national sales. For each pair, the gate sums the coarsest layer's sales_statistic and inventory_statistic group sizes over its join_on groups. The pair keeps the inventory prior when its sales fall below max(count_threshold, ratio_threshold · (sales + inventory)). The gate applies at every layer, so a pair either blends inventory at all layers or at none. Without inventory_gate, every pair blends inventory.
  • A field without is_group_weighted uses its weight_expr as n and the same pseudo-counts.
  • max_share, on a layer in layers, caps the layer's share n / (n + k) of the weight the finer layers leave. A layer below the cap keeps its share unchanged. With max_share: 0.8 on the dealer layer, the neighborhood and country keep at least 20% wherever the country has data for the attribute group.
  • The blend takes no min_obs; k already discounts a layer with a small count.
  • A stat frame without its layer's join_on columns raises.
  • A layer without data for a key, or whose count is NaN, passes its share on to the coarser layers. Where the coarsest layer has no data either, the shares of the others are rescaled to sum to 1, so a capped dealer layer that is the only layer with data takes all the weight. The result is null when every layer is empty.

Audi's smoothed_demand_take_rates sets:

blend:
method: nested_pseudo_count
inventory_weight: 1
layers:
- name: dealer_base_model
pseudo_count: 40
max_share: 0.8
join_on: [dealer_id]
asset: audi/us/attribute_stats/country_by_dealer_base_model
- name: neighborhood_base_model
pseudo_count: 150
join_on: [neighborhood]
asset: audi/us/attribute_stats/country_by_neighborhood_base_model
coarsest_layer:
name: country_base_model
join_on: [country]
asset: audi/us/attribute_stats/country_by_country_base_model

With these values and λ = 1, a dealer with 16 vehicles of a base model in a neighborhood with 129 gets 29%, the neighborhood 33% and the country 38%. A dealer with 160 vehicles in a neighborhood with 3,400 reaches the cap: it gets 80%, the neighborhood 19% and the country 0.8%.

Restricting the output​

dealer_scope_asset restricts the output to the dealers that asset contains, keyed by dealer_scope_col and optionally narrowed to rows where dealer_scope_filter_col is truthy. Omit it and the output covers every dealer in the assembled dimension frame.

Streaming and memory​

The output is the cross product of dealers and the attribute values of every base model. The blend builds it as one streaming plan with these steps:

  1. The cross join of dealers and attribute values.
  2. For each layer, a left join of the layer's aggregates, plus a left join of its group weights when a field is group-weighted.
  3. The weighted sums.

The Iceberg IO manager writes the result batch by batch, so the full output is never in memory. Each layer join builds its hash table on the layer side. The memory the joins use therefore grows with the size of the layers and does not grow with the size of the output.

These settings keep that plan's memory bounded on Polars 2.0:

  • Every layer join sets build_side="force_right". Without it, Polars may build the hash table on the cross-join side and hold the whole output.
  • The component writes the output and runs the take_rate_sums_<label> checks inside pl.Config(streaming_chunk_size=...). The value is the component's streaming_chunk_size field (default 10,000 rows). Without that setting, the cross join of two small in-memory frames emits batches so large that the joins after it hold rows in proportion to the output.

The blend scans each layer a fixed number of times, however many base models there are. The checks evaluate the output plan a second time, as AttributeStatisticsComponent does. The write and the checks must both finish before the vended credentials copied into each scan expire.

Fields​

Each BlendFieldSpec declares one column to blend and the SQL expression that drives its weight. Only declared fields appear in the output. Z-score columns ({label}_{col}_z) are intentionally excluded — z-scores from different geographic layers have incompatible denominators and cannot be meaningfully combined.

fields:
- name: sales_take_rate
weight_expr: sales_group_nobs
is_group_weighted: true

- name: sales_avg_days_on_lot
weight_expr: "sales_group_nobs * sales_take_rate"

Full YAML example​

type: ai_dagster.components.GeographicSmootherComponent
attributes:
oem: mb
name: demand_take_rates
start_date: "2025-01-01"
joins:
- asset: mb/consolidated/dealers
join_on: [dealer_id]
fields: [country]
- asset: mb/dealers/neighborhoods
join_on: [dealer_id]
fields: [neighborhood]
blend:
method: divisor
min_obs: 5
layers:
- name: dealer
divisor: 1
join_on: [dealer_id]
asset: mb/attribute_stats/country_by_dealer
- name: neighborhood
divisor: 10
join_on: [neighborhood]
asset: mb/attribute_stats/country_by_neighborhood
- name: country
divisor: 7000
join_on: [country]
asset: mb/attribute_stats/country_by_country
fields:
- name: sales_take_rate
weight_expr: sales_group_nobs
is_group_weighted: true
- name: sales_avg_days_on_lot
weight_expr: "sales_group_nobs * sales_take_rate"

Output schema: [dealer_id, attribute_type, attribute_value, <declared fields>, _partition_date].

Demand Projection​

ProjectedDemandComponent adjusts the smoothed take-rates for a model-year transition, so next-year synthetic candidates are scored against demand that reflects what actually changed rather than the prior year's option set. It reads smoothed_demand_take_rates and consolidated/models, resolves each rule's model_filters (regex on model-table fields, incl. trim_identifier) to concrete model_id values, and rewrites the affected (attribute_type, attribute_value) rows. All rules operate at the attribute level — never on raw OEM option codes.

Terms​

TermMeaningAvoid
Projection ruleA statement, scoped to a set of models, that sets or moves demand for one target cell whose measured demand is missing or out of date.imputation rule, demand rule
Target cellThe attribute value of one model that a projection rule writes demand to, named by attribute type and attribute value together.target value (a value alone is ambiguous across attribute types)
Rule kindThe category a projection rule belongs to: rename, remove, assumed rate or proxy estimate.rule type (collides with attribute type), stage
Hand ruleA projection rule a person wrote in a rules file for a model-year transition.seed rule, manual rule
Derived ruleA projection rule generated on each run from the values a model has (see Derived rules).auto rule, generated rule
PruneThe derived remove for each attribute value that has a token the model does not have.
Saturated valueThe derived assumed rate of 1.0 for an attribute value nearly every inventory vehicle of the model has.
Estimate kindA rule kind that fills in demand no measurement covers: assumed rate and proxy estimate. Hand rules of an estimate kind are gated, and derived rules are not.
Transform kindA rule kind that changes measured demand: rename and remove. No rule of a transform kind is gated.
GateThe conditions under which a rule skips a dealer: the dealer already has enough history of the model (max_dealer_nobs), or its existing rate is already at least existing_rate_fraction of the rate the rule would set.

Rule kinds​

The component reads hand rules, the rules a person writes for a model-year transition, from the YAML files listed in rules_sources. Each rule kind has its own top-level key in a rules file.

RuleRules-file keyPurpose
RenameRulerename_rulesTransfer demand from a discontinued attribute to its replacement (zero-sum; rules sharing a from_value are additive, cross-group multiplicative).
RemoveRuleremove_rulesZero an attribute that is no longer a customer choice on the trim — either a genuine drop, or (the common transition case) a feature that became standard, whose stale optional take-rate must be cleared since the pipeline has no next-year sales yet to observe the new 100%.
ProxyEstimateRuleproxy_estimate_rulesEstimate a newly-available attribute's rate from a same-population proxy attribute's own rate (portion scales it; 1.0 = deterministic co-occurrence). Leave attribute_type unset when the proxy and the target are binary attribute types (see Binary attributes). Set attribute_type to a multiclass type such as PAINT_COLOR, which the proxy and the target then share, to write only a row of that type.
AssumedRateRuleassumed_rate_rulesSet a newly-available attribute to a manually assumed flat rate — last resort when no same-population proxy exists. An explicit guess, to be replaced by real sales data.

A vehicle has exactly one value of each attribute type, including each binary type and each <GROUP>_SET type, so the rates of every type sum to 1. After the remove rules and the other rules run, the engine renormalizes each type's rates to sum to 1. A single removed or added value propagates automatically to any _X_ combination and _SET composite attribute that contains it, via exact token matching.

Binary attributes​

A binary attribute, such as SUNROOF, is its own attribute type with the values TRUE and FALSE. Each top-level category of the feature hierarchy has a <GROUP>_SET type, such as ADAS_SET. A value of that type lists the attributes a vehicle has in that category: the name of each binary type, and NAME=VALUE for each multiclass attribute.

An assumed-rate or proxy-estimate rule without attribute_type names binary attribute types in target_value, and in proxy_value for a proxy rule. ProjectedDemandComponent raises when it loads such a rule that names a type that is not binary.

  • Assumed rate. The rule adds the target with probability rate, independently of what a vehicle already has. A dealer at TRUE rate t ends at t + rate × (1 − t) for TRUE and at the complement for FALSE. The same fraction rate of each value of the target's own <GROUP>_SET moves to that value with the target token added.
  • Proxy estimate. The rule reads the proxy's TRUE rate. The target is added in the same way, at each dealer with the probability portion × proxy TRUE rate. When the proxy and the target are in the same <GROUP>_SET, only the set values that contain the proxy token move, each by portion, to a value with the target token added. Otherwise every value of the target's set moves that fraction of its rate, the dealer's probability, to a value with the target token added.
  • Rename. A rename between two binary types of one <GROUP>_SET moves portion of the from_value type's TRUE rate to the new_value type, and renames the token in that set's values. A rename between binary types of different sets raises.

A rule with attribute_type set, such as PAINT_COLOR, names values of that one type and writes only rows of that type and its _X_ crosses.

binary_representation on ProjectedDemandComponent decides whether the projection keeps the <GROUP>_SET types (see Binary representation). Under SET, the rules above keep each <GROUP>_SET type consistent with its binary types. Under INDIVIDUAL:

  • the projection drops every <GROUP>_SET row from smoothed_stats, so its output has none;
  • an assumed-rate, proxy-estimate or rename rule of a binary type writes only that type's TRUE and FALSE rows, and a rename between binary types of different categories is allowed;
  • ProjectedDemandComponent raises when it loads any of these:
    • a remove rule whose attribute_type is a <GROUP>_SET type;
    • a remove rule without attribute_type whose from_value is a binary type, TRUE or FALSE;
    • a <GROUP>_SET type in prune_attribute_types or mutually_inclusive_attribute_types.
  • A remove rule for a binary attribute sets attribute_type to the binary type and from_value to TRUE.

Apply order​

Rules are applied in a fixed order: renames → removes → assumed rates → proxy estimates → cross-model estimates. Assumed rates run before proxy estimates so an assumed-rate attribute can itself be a ProxyEstimateRule proxy — e.g. a new package set to an assumed rate, then a feature that ships with that package estimated off it (portion: 1.0), keeping the assumed rate defined in exactly one place.

Every resolved rule names one model_id and writes only that model's rows. Renormalization also runs within one model. The engine therefore applies the rules one model at a time: each model's rows go through the rule kinds in this order with that model's rules alone.

The hand rules of one kind form one batch, and its derived rules another. An assumed-rate or proxy-estimate rule sets the null rates of every row it does not skip to 0.0, in every model its batch names. In a batch that names several models, a model's null rates are therefore set to 0.0 at the batch's first rule of another model. This holds even at a dealer that every rule of its own model skips.

The cross-model estimate stage runs last, against the post-rule frame. A cell an explicit rule already filled reads as present and is skipped, and the source pool includes the cells the hand rules established. That ordering is what lets the automatic stage fill only the gaps the authored rules left.

A cross-model estimate of a mutually-inclusive code, such as a FUNCTIONAL_ATTRIBUTE, also moves _estimate of each _SET row's rate into the set with the code added. The codes are treated as independent, so every code estimated for a dealer's model can double that model's _SET rows. With 20 or more codes per model, the full distribution has millions of rows per dealer, and nearly all of those rows hold a negligible rate. cross_model_estimates.min_set_split_rate bounds the growth. A _SET row is split only where the rate it would move reaches the floor. A smaller row keeps its whole rate without the code. Each code then adds at most 1 / min_set_split_rate rows per dealer and model.

With the floor, the order in which a dealer's codes split the rows matters. Each split lowers the rate the original row keeps. After several splits, a row can be too small for a later code to split, although that code would have split it first. Each dealer and model therefore applies its codes in order of _estimate, largest first, with ties broken by attribute type and code. The codes with the most demand split the rows before any other code has divided them, and the floor drops demand mostly from the smallest codes. The standalone FUNCTIONAL_ATTRIBUTE row still carries the full _estimate, so the code's share across the _SET rows can fall short of it by the rate of the rows left unsplit.

Derived rules​

The derived rules cover the types of prune_attribute_types. They read the values each model has from featurization/inventory (see Synthetic candidates) and from the model's features in consolidated/features, with their attributes in featurization/features.

The rules read a model's values from:

  • its inventory vehicles and its features, when it has at least min_inventory_vehicles inventory vehicles. Such a model is stocked.
  • its features and its few inventory vehicles, on the types its features have, when it has fewer inventory vehicles and its base model has no later model year in consolidated/models. Such a model is a newest model. It has just launched or has not launched yet, or it is a trim of the current model year whose next model year is not in consolidated/models yet.

Any other model keeps all its demand. It has fewer than min_inventory_vehicles inventory vehicles and its base model has a later model year.

The prune:

  • removes each demand value that has a token the model does not have. The tokens of 18,20 are 18 and 20, so 18,20 is removed when the model has no 18. A value that a feature of the model has, but no inventory vehicle has yet, keeps its demand.
  • removes NO_<type> and FALSE from a stocked model when every inventory vehicle of the model has a value of the type. It never removes them from a newest model, because a feature does not show that a vehicle lacks the type.
  • prunes no value of a type when, at every dealer, it would remove every value of the type. On a newest model, it also prunes no value of a type when, at every dealer, it would remove every value except NO_<type> and FALSE. It prunes such a type of a newest model anyway when the cross-model estimates can fill it. They can fill it when every feature value of the type is configurable, the model matches cross_model_estimates.model_filter, and no dealer has at least max_dealer_nobs observations of the model, because the estimates skip such a dealer. A dealer with no source model for any of those values then has no demand for the type. The asset check gaps_have_donor lists those gaps.
  • decides per model, not per dealer. When it prunes a type, it can remove every value that one dealer has of the type, except the NO_<type> and FALSE it keeps. That dealer then has no other demand for the type unless the cross-model estimates give it a rate. They give none to a dealer with at least max_dealer_nobs observations of the model.

When estimate_saturated_values is set, a value of a stocked model gets a saturated value when it has no demand and more than saturation_min_coverage of the model's inventory vehicles have it. A mutually inclusive type gets no saturated value. A newest model gets none either, because a feature does not show whether every vehicle has the value.

For each dealer, the cross-model estimates treat these values as gaps:

  • on a stocked model, each value that more than cross_model_estimates.min_inventory_coverage of the model's inventory vehicles have, and for which the dealer has no demand;
  • on a newest model, each value of a configurable feature of the model that no demand value of the dealer contains, so 18,20 demand leaves no 18 or 20 gap.

Scoping to trims​

The same option code often changes differently across trims of one model (standard on one, a new choice on another, repackaged on a third), so rules are scoped with model_filters.trim_identifier rather than model-wide. A feature that became standard is removed only on the trims where that happened; a feature that stayed a choice is left untouched.

YAML config​

type: ai_dagster.components.ProjectedDemandComponent
attributes:
name: demand_take_rates
start_date: "2025-01-01"
smoothed_stats: <oem>/<market>/smoothed_demand_take_rates
include_models: true
model_col: base_model_id
rules_sources:
- jeep-demand-rules.yaml
demand_cols_to_project: [sales_take_rate, sales_hl90_take_rate]

jeep-demand-rules.yaml, next to defs.yaml:

model_years: [2027]                    # every rule below applies to MY2027 only
remove_rules:
- model_filters: {model_code: "^JLJS74", trim_identifier: "^R$"}
from_value: "PURPLE" # no purple paint on Rubicon R
attribute_type: PAINT_COLOR
assumed_rate_rules:
- model_filters: {model_code: "^JLJS74", trim_identifier: "^R$"}
target_value: "TOW_PACKAGE" # new package, no proxy → assumed rate
rate: 0.35
proxy_estimate_rules:
- model_filters: {model_code: "^JLJS74", trim_identifier: "^R$"}
target_value: "PREMIUM_AUDIO" # ships with the package above
proxy_value: "TOW_PACKAGE" # inherits its rate (applied after assumed rates)
portion: 1.0

A rules file fails to load if it has a top-level key other than model_years or a rules-file key from the Rule kinds table.

Output schema matches the smoothed input: [dealer_id, attribute_type, attribute_value, <demand cols>, _partition_date].