Skip to main content

Order Set Evaluation

Order set evaluation compares the vehicles an ordering run recommended for a dealer with sets of vehicles that dealers stocked, or with the dealer's demand take rates. A small p-value indicates an order set that is far outside what is typical, such as an order set of a single paint color, or an order set with many candidates whose configuration is rare for the dealer, or an order set far larger than the dealer's usual volume of its base model. The evaluation reads the recommendations, not what became of the orders. It therefore evaluates runs of every mode, including runs that place no order. Its output is reported only and does not stop a run. See ADR-010 to ADR-017.

Terminology​

  • Unit — one member of a set: a recommended candidate, one per target, in an order set, or an inventory vehicle in a reference order set.
  • Order set — the units one ordering run recommended for one combination of order_sets.group_by values, such as a dealer and a base_model_id.
  • Reference order set — a set of vehicles from consolidated inventory, formed by a reference. One vehicle can belong to several reference order sets.
  • Reference — a named definition of which inventory vehicles are comparable and how they are grouped into reference order sets.
  • Code counts — the number of units in a set that have each attribute code.
  • Statistic — a number computed from an order set or a reference order set, per attribute type, such as the concentration of PAINT_COLOR.
  • Multi-coded unit — a unit with more than one code of an attribute type. Every evaluation type that selects attribute types raises on one.
  • Baseline — what an evaluation compares an order set with: the comparable reference order sets of one reference, or sets simulated from take rates.
  • Evaluation — a configured comparison of an order set with its baseline. Its configuration sets:
    • the reference or take rates it reads;
    • what it computes;
    • how it turns the comparison into a p-value.

Every vehicle is described by its (attribute_type, attribute_code) pairs. Inventory vehicles have them in featurization/inventory. A recommended candidate gets them from its recommendation.candidate.feature_ids, featurized the way featurization/synthetic featurizes a synthetic candidate. The attributes stored on the recommendation event are not used, so a change to the featurization applies to order sets and reference order sets alike, including when an earlier run is evaluated again. One function then computes a statistic for both.

Data flow​

OrderSetEvaluationComponent defines order_set_evaluations. For each run, it forms the reference order sets in memory and computes their code counts:

  • the number of the set's vehicles with each attribute code;
  • per attribute type, the number of the set's vehicles with a code of that type, as coded_units;
  • the set's number of vehicles, as set_size.

It also computes the set_size of every reference order set, which set_size_z_score reads. A sliding_window window without a vehicle is a set of size 0 for each group_by value with a vehicle in another window. It computes every evaluation from the code counts of the order sets and from one of:

  • the code counts of the reference order sets;
  • the sizes of the reference order sets;
  • take rates.

The reference order sets, their code counts and their sizes are not stored. See ADR-011.

An evaluation names its reference. A reference stores the code counts of the attribute types that the evaluations reading it select. order_set_evaluations raises when featurization/inventory has no inventory vehicle with a code of an attribute type a reference stores. An order set for whose match_keys values the reference forms no set gets a baseline_sets of 0 and a null p_value.

Order sets​

An order set is built from the run's recommendations partition:

  1. Skipped recommendations are dropped.
  2. Each event's dealer_code is resolved to dealer_id through consolidated/dealers. order_set_evaluations raises for a dealer_code with no dealer.
  3. For each target, the recommendation with the lowest iteration is kept, which is candidate selection's first choice. The evaluation judges candidate selection, so it keeps that choice even when the OMS refused it and a later iteration replaced it.
  4. The columns listed in model_columns, such as base_model_id, are joined from consolidated/models on the candidate's model_id.
  5. The candidates are featurized as synthetic candidates of featurization/synthetic, with VehicleFeaturizationSettings.featurize_candidates. It reads the featurization/features, featurization/models, consolidated/features and consolidated/models partitions of the run's partition_date, and featurizes no inventory vehicle. featurization has the fields of VehicleFeaturizationSettings, which VehicleFeaturizationComponent has too. An OEM sets the same values in both.
  6. The remaining recommendations are grouped by order_sets.group_by, which needs at least one key. Each key must be dealer_id, model_id or one of model_columns. order_set_evaluations raises for a unit with a null key, such as a candidate whose model_id has no row in consolidated/models, and names the model_id values.

Each target is one unit of the order set. set_size is the number of units. Per attribute type, coded_units is the number of units that have a code of that type. A candidate without a code of a type of a featurized feature or model gets the code NO_<type>, such as NO_PAINT_COLOR, and every candidate gets a code of each group type and each cross type. NO_<type> counts like any other code, so coded_units equals set_size for those types. See Featurization Groups.

A run with fill_allocations orders each allocation's full cap. The recommendation event of a unit beyond the volume optimization's recommendation has the volume_type fill (see Filling every allocation). Every evaluation runs on the recommended units, and on every unit of a run with a fill unit. Each output row's volume_type names the units it evaluates:

volume_typeUnitsWritten for
recommendedThe units the volume optimization recommended, whose volume_type is not fillEvery run
fillEvery unit the run ordersA run with a fill unit

A unit with a null volume_type, from a run recorded before the field existed, is a recommended unit. An order set whose units all have the volume_type fill has only fill rows.

References​

OrderSetEvaluationComponent lists its references under references. A reference has:

  • a name;
  • a population;
  • a set generator under sets.

Population​

The population consists of the inventory vehicles that count as comparable. A run's population reads the consolidated/inventory partition of the run's partition_date, joined to the same partition of featurization/inventory. Each window of a reference, including the earliest, is formed from that one partition. Two groups of vehicles are never part of a population:

  • Synthetic vehicles, whose is_inventory is false.
  • Orders we placed, which are the inventory records that the reporting/placed_orders partition of the same date resolves. That partition holds only orders placed on or before its date, so a later placement does not change the reference of an earlier run. See Placed Orders.

The columns listed in model_columns are joined from the consolidated/models partition of the same date on the vehicle's model_id. The component does not read them from inventory, so order sets and reference order sets get them from the same rows. A vehicle whose model_id has no consolidated/models row gets null model columns. When an evaluation reads a reference, the materialization records the number of such vehicles as the inventory_without_model metadata value. A reference that groups by a model column leaves those vehicles out.

FieldMeaning
date_columnThe inventory date column that days and sliding_window compare against, such as arrival_date.
daysThe population holds vehicles whose date_column falls within the days days ending on the partition date.
whereOptional Polars SQL predicate over inventory columns and model_columns, such as NOT is_cancelled. The component reads the columns it names.

One consolidated/inventory partition holds vehicles seen within its source groups' history_days, 365 by default. A larger days adds no older vehicles.

Set generators​

A set generator assigns population vehicles to reference order sets. Each set has the generator's group_by keys as columns. An empty group_by puts all vehicles in one set.

typeReference order setsCan a vehicle be in more than one set?
fixed_groupsOne set per value of group_byNo
sliding_windowPer value of group_by, one set per window of width_days days, holding the vehicles whose date_column falls in the window. Windows end on the partition date and every step_days days before it. A window that would start before the population starts is not formed.Yes, when step_days is smaller than width_days

A reference forms sets only for the values of the keys that every evaluation reading it lists in match_keys, and only for the values that the run's order sets have. An evaluation that reads a reference compares an order set only with reference order sets of the same match_keys values, so the other sets would never be compared. A reference read by an evaluation without match_keys forms every set. For each value of those keys, a reference codes its vehicles only in the attribute types that the run's order sets with that value select. For example, when every evaluation that reads a reference has match_keys: [base_model_id], the reference holds the populations of the run's base models, and codes each population in the types of its own base model.

When the code location loads, OrderSetEvaluationComponent rejects:

  • an evaluation whose reference is not listed;
  • an evaluation with a match_keys key that is not in its reference's group_by;
  • a set size z-score whose reference is a sliding_window with a step_days other than its width_days;
  • a reference that no evaluation reads.

Evaluations​

Every evaluation type except set size z-score selects the attribute types it covers; see Attribute types. An empirical evaluation and set size z-score name a reference; take-rate surprise and configuration rarity read <oem>/<market>/output/demand_take_rates. An empirical evaluation and take-rate surprise produce one output row per order set and selected attribute type. Configuration rarity and set size z-score produce one row per order set, with a null attribute_type. Every output row has these columns:

  • evaluation
  • baseline, the reference an empirical evaluation or set size z-score compared with, or the take rate column that take-rate surprise or configuration rarity read
  • set_size and coded_units; coded_units is null for set size z-score
  • observed
  • baseline_sets, the number of sets in the baseline: the compared reference order sets of an empirical evaluation or set size z-score, or the simulated sets of take-rate surprise; null for configuration rarity
  • baseline_p05, baseline_median and baseline_p95, the quantiles of the baseline sets' values
  • p_value
  • attribute_weight, set when the evaluation selects by weight; null for configuration rarity and set size z-score

Every evaluation type that selects attribute types raises when a unit of an order set has more than one code of an attribute type. An empirical evaluation also raises when the code counts of a reference order set sum to more than its coded_units. The checks are part of the LazyFrame that evaluate returns, and raise when it is collected, so a run that fails a check writes no rows.

A statistic and a log probability are rounded to 12 significant digits before they are compared, so two sets tie when their counts differ only in which code has which count.

Attribute types​

attribute_types is a list or a selection by attribute weight:

attribute_types: [PAINT_COLOR]
# or
attribute_types:
from_weights: {model_column: base_model_id, top_k: 8}

A list applies to every order set and must not name a type twice.

from_weights selects each order set's types from the output/attribute_weights partition of the run's partition_date, using the weights of the order set's model_column value. Its fields:

FieldMeaning
model_columnThe column the weights are keyed by, such as base_model_id. It must be in order_sets.group_by, and an empirical evaluation must list it in match_keys. Every evaluation that selects by weight uses the same model_column.
top_kSelects the top_k heaviest types. Equal weights are ordered by attribute type.
min_weightSelects the types whose weight is at least min_weight.

At most one of top_k and min_weight is set, and it is positive. With neither, every type with a weight is selected. The weights are fitted on featurization/inventory, and a candidate is featurized the same way, so a candidate can have every type that has a weight.

attribute_weight is the raw weight of the row's type. An order set gets no row when no type is selected for it. order_set_evaluations raises when:

  • a model of an order set has no weights;
  • a type of the model has more than one weight or a null weight;
  • a selection picks one of mutually_inclusive_attribute_types.

The rows of one order set can be combined into one score per evaluation. For example, the mean of −log10 p_value over the types of an evaluation that selects by weight, weighted by attribute_weight or unweighted:

score = -pl.col("p_value").log10()
weight = pl.col("attribute_weight")
scores = (
evaluations.filter(pl.col("p_value").is_not_null() & weight.is_not_null())
.group_by("run_id", "evaluation", *group_by)
.agg(
((weight * score).sum() / weight.sum()).alias("score_weighted"),
score.mean().alias("score_equal"),
)
)

p_value is at least 1 / (baseline_sets + 1), so a type compared with few reference order sets contributes less to such a score.

Empirical evaluation​

type: empirical computes statistic on the order set and on every reference order set, then reports how often the reference order sets were at least as extreme.

A reference order set is compared with an order set when both conditions hold:

  • Its match_keys columns equal the order set's. Each key must be in order_sets.group_by and in the reference's group_by.
  • Its coded_units is between N / size_ratio and N * size_ratio, where N is the order set's coded_units. A size_ratio of 1 requires the same size. N is the count the statistics use, so units without a code of the type do not affect matching.

p_value is (1 + r) / (n + 1), where n is baseline_sets and r is the number of compared reference order sets at least as extreme as the order set. It is therefore never 0. tail sets the direction:

  • upper: a reference order set is at least as extreme when its statistic is at least the observed one.
  • lower: a reference order set is at least as extreme when its statistic is at most the observed one.

When baseline_sets is 0, p_value is null.

Take-rate surprise​

type: take_rate_surprise scores an order set by the probability of its code counts if the dealer ordered according to its take rates. It reads <oem>/<market>/output/demand_take_rates at the run's partition_date. Candidate selection scored the run's candidates with the take rates of that date. It raises when the take rates have no rows for one of its attribute types, or lack a column it reads. See ADR-013.

  • take_rate_column names the rate column, such as sales_hl90_my_weighted_take_rate. Use the column candidate selection scores with, so the evaluation judges the candidates against the demand they were chosen for. sales_take_rate counts every sale of the window across model years, including codes the target model year no longer has.
  • rate_keys name the columns that select a unit's take rates, such as dealer_id and model_id. Each must be dealer_id, model_id or one of model_columns.

Only the order set's units that have a code of the attribute type and take rates for their rate keys are scored. A unit of a model without take rates, such as a model of the previous model year once the take rates no longer include it, is left out. For one order set and one attribute type, with m the number of scored units:

r_gk  take rate of code k for rate keys g, divided by the sum of g's take rates for the type
w_g share of the scored units with rate keys g

p_k = Σ_g w_g · r_gk

The codes k are those with a take rate for one of the order set's rate keys, and those in the order set. When no unit of the order set is scored, observed and p_value are null.

observed is the log probability of the order set's code counts under the multinomial distribution with probabilities p and m trials. The evaluation draws draws count vectors that each sum to m from the same distribution. p_value is (1 + r) / (draws + 1), where r is the number of drawn vectors whose log probability is at most observed. draws must be at least 100, and baseline_sets is draws. Each order set and attribute type draws from its own random generator, seeded from the evaluation's seed, the order set's run_id and group_by keys, and the attribute type, so a p-value does not depend on the other order sets of the run.

A code without a take rate, such as a paint color the dealer has not sold, has probability 0. An order set with such a code has an observed of -inf, which no drawn vector reaches, so its p_value is the minimum, 1 / (draws + 1).

Configuration rarity​

type: configuration_rarity counts the units of an order set whose configuration is rare for the dealer's demand. A unit's configuration is its codes of the selected attribute types. rate_keys and take_rate_column select a unit's take rates as they do for take-rate surprise, and the take rates are divided by their sum per rate keys and attribute type in the same way. See ADR-016.

- type: configuration_rarity
name: weighted_configuration_rarity
attribute_types:
from_weights: {model_column: base_model_id, min_weight: 0.001}
rate_keys: [dealer_id, model_id]
take_rate_column: sales_hl90_my_weighted_take_rate
rare_mass: 0.05
max_log_probability: -1.5
draws: 2000

A unit is scored on each selected type that it has a code of and that its rate keys have take rates for. A listed type has weight 1. With from_weights, a type's weight is its attribute_weight, which is positive. For a unit with rate keys g:

r_gt(c)                   take rate of code c for rate keys g and type t, divided by the sum of
g's rates for t; 0 for a code without a take rate
w_t the weight of type t
weighted_log_probability Σ_t w_t · log r_gt(c_t), -inf when a code's rate is 0

A unit with a code whose take rate is 0 or missing has a configuration the dealer's demand never produces, so its weighted_log_probability is -inf. zero_rate_types lists the types of such codes.

A code whose take rate is 0 makes weighted_log_probability -inf whatever the weight of its type, even a negligible one. The attribute weights average the weights fitted at several hierarchy levels, which gives many types weights such as 0.0000001. A weight that small changes no candidate's rank in candidate selection. min_weight: 0.001 leaves those types out, so only a type whose weight can change a candidate's rank can make a unit rare.

The evaluation draws draws configurations from the take rates of g, each type's code drawn independently, and scores them the same way. A unit's configuration_tail_mass is (1 + r) / (draws + 1), where r is the number of drawn configurations scoring at most the unit's. It estimates the probability that a drawn configuration scores at most as high, and is never 0. The draws for one rate keys value and one set of scored types and weights are seeded from seed, the rate keys, the types and the weights, so a unit's result does not depend on the other units of the run. Scores are rounded to 12 significant digits, so configurations of equal score tie.

A unit is rare when it meets each of these conditions:

  • Its configuration_tail_mass is at most rare_mass, which must be greater than 0 and less than 1.
  • When max_log_probability is set, its weighted_log_probability is at most max_log_probability, which must be less than 0.

A unit whose weighted_log_probability is -inf passes any max_log_probability.

The attribute weights of a model sum to 1. With from_weights, the scored types hold nearly all of that weight, so the exponential of a weighted_log_probability is about the weighted geometric mean of the unit's take rates. A max_log_probability of -1.5 makes rare only a unit whose weighted geometric mean take rate is at most about 0.22.

rare_probability is the share of the drawn configurations that are rare by the same conditions. It estimates the probability that a configuration drawn from the take rates is rare. That probability is at most rare_mass, and it is lower when no group of lowest-scoring configurations has a total probability of exactly rare_mass. For example, with one type whose least common code has a take rate of 0.06, no configuration is rare at a rare_mass of 0.05.

For an order set, coded_units counts the scored units and observed is the number of them that are rare, k. Each scored unit is rare with its rare_probability, so k follows a Poisson binomial distribution. p_value is the probability of at least k rare units, and baseline_p05, baseline_median and baseline_p95 are the quantiles of that distribution. An order set without a scored unit has a null observed and p_value. The evaluation raises when the take rates have more than one row for a code of one rate keys.

reporting/unit_rarities has one row per unit, configuration rarity evaluation and volume_type that evaluates the unit, with the run columns, the order set's group_by keys, model_id, and:

  • evaluation and unit_id;
  • scored_types, the number of types the unit is scored on;
  • weighted_log_probability, configuration_tail_mass, rare and rare_probability, null for a unit with no scored type;
  • zero_rate_types, null for a unit with no scored type;
  • volume_type, recommended or fill, which names the units evaluated together with the unit (see Recommended and fill units).

The materialization of unit_rarities tests the run's scored units together and records the test in its metadata. Per evaluation, the metadata has:

  • <evaluation>/scored_units;
  • <evaluation>/rare_units;
  • <evaluation>/expected_rare_units, the sum of the scored units' rare_probability;
  • <evaluation>/p_value, the probability of at least rare_units rare units.

These keys record the test of the recommended units. For a run with a fill unit, the same keys prefixed with fill/ record the test of every unit.

Set size z-score​

type: set_size_z_score compares the number of units in an order set with the sizes of the reference order sets of the same match_keys values. It selects no attribute types, and produces one row per order set with a null attribute_type and a null coded_units. See ADR-017.

- type: set_size_z_score
name: dealer_base_model_volume_z_score
reference: dealer_base_model_windows
match_keys: [dealer_id, base_model_id]

Its reference must be fixed_groups, or a sliding_window whose step_days equals width_days, so that each vehicle of the population is in exactly one reference order set. With a sliding_window reference, the reference order sets are the dealer's non-overlapping windows, including windows of size 0. With n reference order sets of mean m and sample standard deviation s, an order set of size N has:

  • observed, the z-score (N − m) / s;
  • baseline_sets, n;
  • baseline_p05, baseline_median and baseline_p95, the quantiles of the reference order sets' sizes, as z-scores;
  • p_value, (1 + r) / (n + 1), where r is the number of reference order sets at least as far from m as N, on either side.

observed and the quantiles are null when s is 0 or n is less than 2. When n is 0, p_value is null too.

Code distribution shift​

CodeDistributionShift compares the codes of an order set with the codes of another algorithm's order sets, the comparison order sets. OrderSetEvaluationComponent does not run it; ai_evaluation.comparison does, as the mix test. It reads the comparison order sets in the same long form as the order sets, as the comparison_sets input. compared_with names the algorithm that produced them and is the output's baseline.

evaluation = CodeDistributionShift(
name="paint_shift_vs_rules",
compared_with="rules",
attribute_types=["PAINT_COLOR"],
match_keys=["dealer_id", "model_id"],
draws=2000,
)
evaluate_order_sets(
order_sets,
[evaluation],
{EvaluationInput.COMPARISON_SETS: comparison_order_sets},
{
evaluation.name: select_types(
evaluation.selection(), order_sets.group_by(SET_ID).agg(), None
)
},
)

For one order set and one attribute type, the comparison sample is every comparison unit with a code of the type whose match_keys values equal the order set's, whichever comparison order set holds it. With n_k and c_k the order set's and the comparison sample's units of code k, and N and C their sums:

  • observed is the total variation distance ½ Σ_k |n_k / N − c_k / C|, from 0 for equal shares to 1 for disjoint codes;
  • each of draws permutations splits the pooled N + C units at random into samples of sizes N and C, and baseline_p05, baseline_median and baseline_p95 are the quantiles of their distances;
  • p_value is (1 + r) / (draws + 1), where r is the number of permutations at least as far apart as observed;
  • baseline_sets is draws, which must be at least 100.

observed and p_value are null when the order set or the comparison sample has no unit with a code of the type. The random generator is seeded like take-rate surprise's. The evaluation raises when a unit of the comparison order sets has more than one code of an attribute type. A weight selection's model_column must be one of match_keys.

Statistics​

typeValue for counts n_k, N = Σ n_kNull when
unbiased_simpsonΣ n_k(n_k − 1) / (N(N − 1)), the probability that two different units share a codeN < 2
miller_madow_entropyln N − (Σ n_k ln n_k) / N + (K − 1) / (2N), where K is the number of codes presentNever

A higher Simpson index means a more concentrated set, and so does a lower entropy. A concentration evaluation therefore uses tail: upper with unbiased_simpson and tail: lower with miller_madow_entropy.

Both estimators correct the bias of the plug-in estimator. That bias depends on N. Size matching through size_ratio is still needed for the p-value, because the variance of either statistic shrinks as N grows.

Both statistics treat an attribute type as one code per vehicle. When the code location loads, OrderSetEvaluationComponent rejects an evaluation that lists one of mutually_inclusive_attribute_types.

Assets​

AssetPartitionsOne row per
<oem>/<market>/reporting/order_set_evaluationsOrdering run (run_id)volume_type, evaluation, order set and attribute type, or volume_type, evaluation and order set
<oem>/<market>/reporting/unit_raritiesOrdering run (run_id)volume_type, configuration rarity evaluation and unit

One materialization writes both assets. unit_rarities has no rows when no evaluation is a configuration rarity.

order_set_evaluations reads the run's partition_date from ordering/runs. It is the date of the warehouse partitions the run selected its candidates from. order_set_evaluations reads every input at that date. See ADR-014 and Input partitions. The inputs are consolidated/dealers, consolidated/models, consolidated/features, featurization/features and featurization/models, and each evaluation's input:

  • consolidated/inventory, featurization/inventory and reporting/placed_orders, when an evaluation reads a reference;
  • output/demand_take_rates, when an evaluation is take-rate surprise or configuration rarity;
  • output/attribute_weights, when an evaluation selects its attribute types by weight.

order_set_evaluations raises when the run's partition_date is null, and when an input other than reporting/placed_orders has no partition on that date, naming the input. It also raises on a missing input partition for a run whose partition_date_source is run_date. A reporting/placed_orders partition without rows, as before the first order is placed, cannot be told apart from a missing one, so the population then leaves out no placed order.

Each row has:

  • the run's run_id, run_mode, run_timestamp and partition_date;
  • the order set's group_by keys;
  • the evaluation's columns;
  • weights_partition_date, the date of the output/attribute_weights partition the evaluation's types were selected from, or null when the evaluation lists its types;
  • volume_type, recommended or fill, which names the units the row evaluates (see Recommended and fill units).

order_set_evaluations materializes when a new run partition is registered, like Build Deltas. It is not materialized again when an input partition arrives later. When a run's evaluation fails because an input partition is missing, the run's order_set_evaluations partition has to be materialized again by hand once the input partition exists. Its rows are then the same as if the input partition had existed at the first materialization.

Adding an evaluation​

A new statistic type, set generator type or evaluation type is implemented as a subclass of the matching base class in packages/ai_evaluation/src/ai_evaluation/order_sets/:

Base classModuleUnion the subclass is added to
SetStatisticstatistics.pyAnySetStatistic
SetGeneratorsets.pyAnySetGenerator
Evaluationevaluations.pyAnyEvaluation

Each subclass has a type field with a Literal value. Setting type to that value in defs.yaml selects the subclass. An Evaluation subclass implements:

  • reads, the input it reads;
  • config_problems, which checks its configuration;
  • input_columns, the columns of its input it reads, which are the only ones the component loads;
  • _baseline_name, the value of the output's baseline column;
  • _compare, which compares the order sets with its input and returns its output rows as a LazyFrame.

Evaluation.evaluate_sets runs these steps in order:

  1. It raises when its input lacks one of the input_columns.
  2. It calls _evaluate, which by default calls _compare and gives no unit rows. ConfigurationRarity overrides _evaluate to return its unit rows too.
  3. It adds the evaluation and baseline columns to the rows _evaluate returns.

Evaluation.evaluate returns the rows of evaluate_sets. evaluate_order_sets in ai_evaluation runs a list of evaluations. It returns the evaluations' rows as one LazyFrame and their unit rows as another, and collects nothing. It caches the order sets and each input in the plan, so the caller's single pl.collect_all call on both frames reads each of them once. Each evaluation attaches its checks to its output with ai_core.polars.with_row_checks, because a check on an input can be skipped when a later join discards that input. OrderSetEvaluationComponent passes its list of evaluations to evaluate_order_sets, so it needs no change for a new evaluation type that reads an existing input. An evaluation that uses only existing statistic, set generator and evaluation types needs no code. It is a new entry under evaluations.

Comparing two sets of order sets​

ai_evaluation.comparison compares two OrderSets, a baseline and another side, such as the recommendations of two versions of candidate selection or of two ordering runs. An OrderSets (ai_evaluation.order_sets.units) holds:

  • units, one row per unit, with unit_id, the order-set keys columns, and optionally price;
  • codes, one row per unit and code, with unit_id, attribute_type, attribute_value and optionally numeric columns of a unit's code, such as the take rate a selection read for it.

It is stored as a directory of units.parquet, codes.parquet and keys.json. candidate_selection.diagnostics writes that directory from a file of candidate selection's recommendations or from a past ordering run, without importing ai_evaluation. OrderSets.read checks the columns, the unit ids and the code columns.

compute_diff(baseline, other, levels, settings, evaluations) compares the two at the global level and per value of each key in levels:

SectionNeedsMeasures
Unit countsalwaysunits per side
Attribute mixalwayseach code's share per side, and a code distribution shift test per attribute type
Code value shifta code column on both sidesthe mean value of each side's codes per attribute type, such as the mean take rate of the selected options, and a test of the difference
Code value drifta code column on both sideswhether both sides read the same value for the same order set and code
Priceprice on both sidesquantiles, Kolmogorov–Smirnov statistic and Wasserstein distance
Evaluationsan EvaluationRequesteach side's results of the requested evaluations, compared order set by order set

The mix test is code distribution shift, reading the baseline as the comparison order sets. It leaves out an attribute type for which a unit of either side has more than one code, such as options or packages, and the report names the types it left out. At a level, it also leaves out the units without a value of the level's key.

The code value shift takes each unit's mean value over its codes of a type. Its test splits both sides' unit values at random into samples of the two sides' sizes, and p_value is (1 + r) / (draws + 1), where r is the number of splits whose difference in means is at least as far from 0 as the observed one.

ComparisonSettings sets the number of random draws of both tests, their seed and the significance level. Each test's p_adjusted is the Benjamini–Hochberg adjustment of its p_value within its level, and for the code value shift within its level and code column. A test whose p_adjusted is below the significance level counts as a shift, which keeps the expected share of false alarms among the counted shifts below that level.

An EvaluationRequest names evaluations of AnyEvaluation, their inputs, and the order-set keys whose order sets they score. Each evaluation runs on both sides. For each evaluation and attribute type, the comparison counts the order sets each side has below the significance level, and the order sets whose p_value is higher on the other side (less unusual) or lower (more unusual). Per evaluation, it counts the order sets each side has flagged on any attribute type, and the order sets that are less or more unusual on the other side, by most of their attribute types, and tests those two counts with a sign test. The flagged counts use unadjusted p-values, so about one order set in twenty per side is flagged by chance.

python -m ai_evaluation.comparison compare and render write the comparison as Parquet and render it as Markdown, such as for a pull request comment. compare --subset also compares the units of some key values, such as one model, as a comparison of its own. render takes several comparisons, such as one per OEM, and writes one document. Each comparison shows its summary and test tables, then its detail sections in a collapsed block, then the summary and test tables of each of its subsets. Every comparison has the same sections in the same order, and a section whose data the inputs lack says so. The legend and notes appear once, at the top. Each summary and test row has a status: green for no difference beyond chance, yellow for a difference to look at, and red for a large difference beyond chance or a fit to the evaluations' inputs that is worse beyond chance. The take rates follow their adjusted tests, prices a permutation test of the mean price, and each evaluation a sign test of how many order sets are more unusual on the other side against how many are less. The price distance has no status. Units are compared by count.

Known limitations​

Every statistic must be computable from code counts and set_size. See ADR-011.