Order Set Evaluation
Order set evaluation compares the vehicles an ordering run recommended for a dealer with sets of vehicles that dealers stocked, or with the dealer's demand take rates. A small p-value indicates an order set that is far outside what is typical, such as an order set of a single paint color, or an order set with many candidates whose configuration is rare for the dealer, or an order set far larger than the dealer's usual volume of its base model. The evaluation reads the recommendations, not what became of the orders. It therefore evaluates runs of every mode, including runs that place no order. Its output is reported only and does not stop a run. See ADR-010 to ADR-017.
Terminology
- Unit — one member of a set: a recommended candidate, one per target, in an order set, or an inventory vehicle in a reference order set.
- Order set — the units one ordering run recommended for one combination of
order_sets.group_byvalues, such as a dealer and abase_model_id. - Reference order set — a set of vehicles from consolidated inventory, formed by a reference. One vehicle can belong to several reference order sets.
- Reference — a named definition of which inventory vehicles are comparable and how they are grouped into reference order sets.
- Code counts — the number of units in a set that have each attribute code.
- Statistic — a number computed from an order set or a reference order set, per attribute
type, such as the concentration of
PAINT_COLOR. - Multi-coded unit — a unit with more than one code of an attribute type. Every evaluation type that selects attribute types raises on one.
- Baseline — what an evaluation compares an order set with: the comparable reference order sets of one reference, or sets simulated from take rates.
- Evaluation — a configured comparison of an order set with its baseline. Its configuration
sets:
- the reference or take rates it reads;
- what it computes;
- how it turns the comparison into a p-value.
Every vehicle is described by its (attribute_type, attribute_code) pairs. Inventory vehicles have
them in featurization/inventory. A recommended candidate gets them from its
recommendation.candidate.feature_ids, featurized the way
featurization/synthetic featurizes a synthetic candidate. The attributes stored on the
recommendation event are not used, so a change to the featurization applies to order sets and
reference order sets alike, including when an earlier run is evaluated again. One function then
computes a statistic for both.
Data flow
OrderSetEvaluationComponent defines order_set_evaluations. For each run, it forms the reference
order sets in memory and computes their code counts:
- the number of the set's vehicles with each attribute code;
- per attribute type, the number of the set's vehicles with a code of that type, as
coded_units; - the set's number of vehicles, as
set_size.
It also computes the set_size of every reference order set, which set_size_z_score reads. A
sliding_window window without a vehicle is a set of size 0 for each group_by value with a
vehicle in another window. It computes every evaluation from the code counts of the order sets and
from one of:
- the code counts of the reference order sets;
- the sizes of the reference order sets;
- take rates.
The reference order sets, their code counts and their sizes are not stored. See ADR-011.
An evaluation names its reference. A reference stores the code counts of the attribute types that
the evaluations reading it select. order_set_evaluations raises when featurization/inventory has no
inventory vehicle with a code of an attribute type a reference stores. An order set for whose
match_keys values the reference forms no set gets a baseline_sets of 0 and a null p_value.
Order sets
An order set is built from the run's recommendations partition:
- Skipped recommendations are dropped.
- Each event's
dealer_codeis resolved todealer_idthroughconsolidated/dealers.order_set_evaluationsraises for adealer_codewith no dealer. - For each target, the recommendation with the lowest
iterationis kept, which is candidate selection's first choice. The evaluation judges candidate selection, so it keeps that choice even when the OMS refused it and a later iteration replaced it. - The columns listed in
model_columns, such asbase_model_id, are joined fromconsolidated/modelson the candidate'smodel_id. - The candidates are featurized as synthetic candidates of
featurization/synthetic, withVehicleFeaturizationSettings.featurize_candidates. It reads thefeaturization/features,featurization/models,consolidated/featuresandconsolidated/modelspartitions of the run'spartition_date, and featurizes no inventory vehicle.featurizationhas the fields ofVehicleFeaturizationSettings, whichVehicleFeaturizationComponenthas too. An OEM sets the same values in both. - The remaining recommendations are grouped by
order_sets.group_by, which needs at least one key. Each key must bedealer_id,model_idor one ofmodel_columns.order_set_evaluationsraises for a unit with a null key, such as a candidate whosemodel_idhas no row inconsolidated/models, and names themodel_idvalues.
Each target is one unit of the order set. set_size is the number of units. Per attribute type,
coded_units is the number of units that have a code of that type. A candidate without a code of a
type of a featurized feature or model gets the code NO_<type>, such as NO_PAINT_COLOR, and every
candidate gets a code of each group type and each cross type. NO_<type> counts like any other
code, so coded_units equals set_size for those types. See
Featurization Groups.
Recommended and fill units
A run with fill_allocations orders each allocation's full cap. The recommendation event of a
unit beyond the volume optimization's recommendation has the volume_type fill (see
Filling every allocation). Every evaluation runs
on the recommended units, and on every unit of a run with a fill unit. Each output row's
volume_type names the units it evaluates:
volume_type | Units | Written for |
|---|---|---|
recommended | The units the volume optimization recommended, whose volume_type is not fill | Every run |
fill | Every unit the run orders | A run with a fill unit |
A unit with a null volume_type, from a run recorded before the field existed, is a
recommended unit. An order set whose units all have the volume_type fill has only fill
rows.
References
OrderSetEvaluationComponent lists its references under references. A reference has:
- a
name; - a
population; - a set generator under
sets.
Population
The population consists of the inventory vehicles that count as comparable. A run's population
reads the consolidated/inventory partition of the run's partition_date, joined to the same
partition of featurization/inventory. Each window of a reference, including the earliest, is formed
from that one partition. Two groups of vehicles are never part of a population:
- Synthetic vehicles, whose
is_inventoryis false. - Orders we placed, which are the inventory records that the
reporting/placed_orderspartition of the same date resolves. That partition holds only orders placed on or before its date, so a later placement does not change the reference of an earlier run. See Placed Orders.
The columns listed in model_columns are joined from the consolidated/models partition of the
same date on the vehicle's model_id. The component does not read them from inventory, so order
sets and reference order sets get them from the same rows. A vehicle whose model_id has no
consolidated/models row gets null model columns. When an evaluation reads a reference, the
materialization records the number of such vehicles as the inventory_without_model metadata
value. A reference that groups by a model column leaves those vehicles out.
| Field | Meaning |
|---|---|
date_column | The inventory date column that days and sliding_window compare against, such as arrival_date. |
days | The population holds vehicles whose date_column falls within the days days ending on the partition date. |
where | Optional Polars SQL predicate over inventory columns and model_columns, such as NOT is_cancelled. The component reads the columns it names. |
One consolidated/inventory partition holds vehicles seen within its source groups'
history_days, 365 by default. A larger days adds no older vehicles.
Set generators
A set generator assigns population vehicles to reference order sets. Each set has the generator's
group_by keys as columns. An empty group_by puts all vehicles in one set.
type | Reference order sets | Can a vehicle be in more than one set? |
|---|---|---|
fixed_groups | One set per value of group_by | No |
sliding_window | Per value of group_by, one set per window of width_days days, holding the vehicles whose date_column falls in the window. Windows end on the partition date and every step_days days before it. A window that would start before the population starts is not formed. | Yes, when step_days is smaller than width_days |
A reference forms sets only for the values of the keys that every evaluation reading it lists in
match_keys, and only for the values that the run's order sets have. An evaluation that reads a
reference compares an order set only with reference order sets of the same match_keys values, so
the other sets would never be compared. A reference read by an evaluation without match_keys
forms every set. For each value of those keys, a reference codes its vehicles only in the attribute
types that the run's order sets with that value select. For example, when every evaluation that
reads a reference has match_keys: [base_model_id], the reference holds the populations of the
run's base models, and codes each population in the types of its own base model.
When the code location loads, OrderSetEvaluationComponent rejects:
- an evaluation whose
referenceis not listed; - an evaluation with a
match_keyskey that is not in its reference'sgroup_by; - a set size z-score whose reference is a
sliding_windowwith astep_daysother than itswidth_days; - a reference that no evaluation reads.
Evaluations
Every evaluation type except set size z-score selects the attribute types it covers; see
Attribute types. An empirical evaluation and set size z-score name a reference;
take-rate surprise and configuration rarity read <oem>/<market>/output/demand_take_rates. An
empirical evaluation and take-rate surprise produce one output row per order set and selected
attribute type. Configuration rarity and set size z-score produce one row per order set, with a
null attribute_type. Every output row has these columns:
evaluationbaseline, the reference an empirical evaluation or set size z-score compared with, or the take rate column that take-rate surprise or configuration rarity readset_sizeandcoded_units;coded_unitsis null for set size z-scoreobservedbaseline_sets, the number of sets in the baseline: the compared reference order sets of an empirical evaluation or set size z-score, or the simulated sets of take-rate surprise; null for configuration raritybaseline_p05,baseline_medianandbaseline_p95, the quantiles of the baseline sets' valuesp_valueattribute_weight, set when the evaluation selects by weight; null for configuration rarity and set size z-score
Every evaluation type that selects attribute types raises when a unit of an order set has more
than one code of an attribute type. An empirical evaluation also raises when the code counts of a reference order set sum to more
than its coded_units. The checks are part of the LazyFrame that evaluate returns, and raise
when it is collected, so a run that fails a check writes no rows.
A statistic and a log probability are rounded to 12 significant digits before they are compared, so two sets tie when their counts differ only in which code has which count.
Attribute types
attribute_types is a list or a selection by attribute weight:
attribute_types: [PAINT_COLOR]
# or
attribute_types:
from_weights: {model_column: base_model_id, top_k: 8}
A list applies to every order set and must not name a type twice.
from_weights selects each order set's types from the output/attribute_weights partition of the
run's partition_date, using the weights of the order set's model_column value. Its fields:
| Field | Meaning |
|---|---|
model_column | The column the weights are keyed by, such as base_model_id. It must be in order_sets.group_by, and an empirical evaluation must list it in match_keys. Every evaluation that selects by weight uses the same model_column. |
top_k | Selects the top_k heaviest types. Equal weights are ordered by attribute type. |
min_weight | Selects the types whose weight is at least min_weight. |
At most one of top_k and min_weight is set, and it is positive. With neither, every type with a
weight is selected. The weights are fitted on featurization/inventory, and a candidate is
featurized the same way, so a candidate can have every type that has a weight.
attribute_weight is the raw weight of the row's type. An order set gets no row when no type is
selected for it. order_set_evaluations raises when:
- a model of an order set has no weights;
- a type of the model has more than one weight or a null weight;
- a selection picks one of
mutually_inclusive_attribute_types.
The rows of one order set can be combined into one score per evaluation. For example, the mean of
−log10 p_value over the types of an evaluation that selects by weight, weighted by
attribute_weight or unweighted:
score = -pl.col("p_value").log10()
weight = pl.col("attribute_weight")
scores = (
evaluations.filter(pl.col("p_value").is_not_null() & weight.is_not_null())
.group_by("run_id", "evaluation", *group_by)
.agg(
((weight * score).sum() / weight.sum()).alias("score_weighted"),
score.mean().alias("score_equal"),
)
)
p_value is at least 1 / (baseline_sets + 1), so a type compared with few reference order sets
contributes less to such a score.
Empirical evaluation
type: empirical computes statistic on the order set and on every reference order set, then
reports how often the reference order sets were at least as extreme.
A reference order set is compared with an order set when both conditions hold:
- Its
match_keyscolumns equal the order set's. Each key must be inorder_sets.group_byand in the reference'sgroup_by. - Its
coded_unitsis betweenN / size_ratioandN * size_ratio, whereNis the order set'scoded_units. Asize_ratioof 1 requires the same size.Nis the count the statistics use, so units without a code of the type do not affect matching.
p_value is (1 + r) / (n + 1), where n is baseline_sets and r is the number of compared
reference order sets at least as extreme as the order set. It is therefore never 0. tail sets the
direction:
upper: a reference order set is at least as extreme when its statistic is at least the observed one.lower: a reference order set is at least as extreme when its statistic is at most the observed one.
When baseline_sets is 0, p_value is null.
Take-rate surprise
type: take_rate_surprise scores an order set by the probability of its code counts if the dealer
ordered according to its take rates. It reads <oem>/<market>/output/demand_take_rates at the
run's partition_date. Candidate selection scored the run's candidates with the take rates of
that date. It raises when the take rates have no rows for one of its attribute types, or lack a
column it reads. See ADR-013.
take_rate_columnnames the rate column, such assales_hl90_my_weighted_take_rate. Use the column candidate selection scores with, so the evaluation judges the candidates against the demand they were chosen for.sales_take_ratecounts every sale of the window across model years, including codes the target model year no longer has.rate_keysname the columns that select a unit's take rates, such asdealer_idandmodel_id. Each must bedealer_id,model_idor one ofmodel_columns.
Only the order set's units that have a code of the attribute type and take rates for their rate
keys are scored. A unit of a model without take rates, such as a model of the previous model year
once the take rates no longer include it, is left out. For one order set and one attribute type,
with m the number of scored units:
r_gk take rate of code k for rate keys g, divided by the sum of g's take rates for the type
w_g share of the scored units with rate keys g
p_k = Σ_g w_g · r_gk
The codes k are those with a take rate for one of the order set's rate keys, and those in the
order set. When no unit of the order set is scored, observed and p_value are null.
observed is the log probability of the order set's code counts under the multinomial
distribution with probabilities p and m trials. The evaluation draws draws count vectors that
each sum to m from the same distribution. p_value is (1 + r) / (draws + 1), where r is the
number of drawn vectors whose log probability is at most observed. draws must be at least 100,
and baseline_sets is draws.
Each order set and attribute type draws from its own random generator, seeded from the
evaluation's seed, the order set's run_id and group_by keys, and the attribute type, so a
p-value does not depend on the other order sets of the run.
A code without a take rate, such as a paint color the dealer has not sold, has probability 0. An
order set with such a code has an observed of -inf, which no drawn vector reaches, so its
p_value is the minimum, 1 / (draws + 1).
Configuration rarity
type: configuration_rarity counts the units of an order set whose configuration is rare for the
dealer's demand. A unit's configuration is its codes of the selected attribute types. rate_keys
and take_rate_column select a unit's take rates as they do for take-rate surprise, and the take
rates are divided by their sum per rate keys and attribute type in the same way. See
ADR-016.
- type: configuration_rarity
name: weighted_configuration_rarity
attribute_types:
from_weights: {model_column: base_model_id, min_weight: 0.001}
rate_keys: [dealer_id, model_id]
take_rate_column: sales_hl90_my_weighted_take_rate
rare_mass: 0.05
max_log_probability: -1.5
draws: 2000
A unit is scored on each selected type that it has a code of and that its rate keys have take rates
for. A listed type has weight 1. With from_weights, a type's weight is its attribute_weight,
which is positive. For a unit with rate keys g:
r_gt(c) take rate of code c for rate keys g and type t, divided by the sum of
g's rates for t; 0 for a code without a take rate
w_t the weight of type t
weighted_log_probability Σ_t w_t · log r_gt(c_t), -inf when a code's rate is 0
A unit with a code whose take rate is 0 or missing has a configuration the dealer's demand never
produces, so its weighted_log_probability is -inf. zero_rate_types lists the types of
such codes.
A code whose take rate is 0 makes weighted_log_probability -inf whatever the weight of its
type, even a negligible one. The attribute weights average the weights fitted at several hierarchy
levels, which gives many types weights such as 0.0000001. A weight that small changes no
candidate's rank in candidate selection. min_weight: 0.001 leaves those types out, so only a
type whose weight can change a candidate's rank can make a unit rare.
The evaluation draws draws configurations from the take rates of g, each type's code drawn
independently, and scores them the same way. A unit's configuration_tail_mass is
(1 + r) / (draws + 1), where r is the number of drawn configurations scoring at most the
unit's. It estimates the probability that a drawn configuration scores at most as high, and is never
0. The draws for one rate keys value and one set of scored types and weights are seeded from seed,
the rate keys, the types and the weights, so a unit's result does not depend on the other units of
the run.
Scores are rounded to 12 significant digits, so configurations of equal score tie.
A unit is rare when it meets each of these conditions:
- Its
configuration_tail_massis at mostrare_mass, which must be greater than 0 and less than 1. - When
max_log_probabilityis set, itsweighted_log_probabilityis at mostmax_log_probability, which must be less than 0.
A unit whose weighted_log_probability is -inf passes any max_log_probability.
The attribute weights of a model sum to 1. With from_weights, the scored types hold nearly all of
that weight, so the exponential of a weighted_log_probability is about the weighted geometric mean
of the unit's take rates. A max_log_probability of -1.5 makes rare only a unit whose weighted
geometric mean take rate is at most about 0.22.
rare_probability is the share of the drawn configurations that are rare by the same conditions. It
estimates the probability that a configuration drawn from the take rates is rare. That probability
is at most rare_mass, and it is lower when no group of lowest-scoring configurations has a total
probability of exactly rare_mass. For example, with one type whose least common code has a take
rate of 0.06, no configuration is rare at a rare_mass of 0.05.
For an order set, coded_units counts the scored units and observed is the number of them that
are rare, k. Each scored unit is rare with its rare_probability, so k follows a Poisson
binomial distribution. p_value is the probability of at least k rare units, and
baseline_p05, baseline_median and baseline_p95 are the quantiles of that distribution. An
order set without a scored unit has a null observed and p_value. The evaluation raises when the
take rates have more than one row for a code of one rate keys.
reporting/unit_rarities has one row per unit, configuration rarity evaluation and volume_type
that evaluates the unit, with the run columns, the order set's group_by keys, model_id, and:
evaluationandunit_id;scored_types, the number of types the unit is scored on;weighted_log_probability,configuration_tail_mass,rareandrare_probability, null for a unit with no scored type;zero_rate_types, null for a unit with no scored type;volume_type,recommendedorfill, which names the units evaluated together with the unit (see Recommended and fill units).
The materialization of unit_rarities tests the run's scored units together and records the test
in its metadata. Per evaluation, the metadata has:
<evaluation>/scored_units;<evaluation>/rare_units;<evaluation>/expected_rare_units, the sum of the scored units'rare_probability;<evaluation>/p_value, the probability of at leastrare_unitsrare units.
These keys record the test of the recommended units. For a run with a fill unit, the same keys
prefixed with fill/ record the test of every unit.
Set size z-score
type: set_size_z_score compares the number of units in an order set with the sizes of the
reference order sets of the same match_keys values. It selects no attribute types, and produces
one row per order set with a null attribute_type and a null coded_units. See
ADR-017.
- type: set_size_z_score
name: dealer_base_model_volume_z_score
reference: dealer_base_model_windows
match_keys: [dealer_id, base_model_id]
Its reference must be fixed_groups, or a sliding_window whose step_days equals width_days,
so that each vehicle of the population is in exactly one reference order set. With a
sliding_window reference, the reference order sets are the dealer's non-overlapping windows,
including windows of size 0. With n reference order sets of mean m and sample standard
deviation s, an order set of size N has:
observed, the z-score(N − m) / s;baseline_sets,n;baseline_p05,baseline_medianandbaseline_p95, the quantiles of the reference order sets' sizes, as z-scores;p_value,(1 + r) / (n + 1), whereris the number of reference order sets at least as far frommasN, on either side.
observed and the quantiles are null when s is 0 or n is less than 2. When n is 0,
p_value is null too.
Code distribution shift
CodeDistributionShift compares the codes of an order set with the codes of another
algorithm's order sets, the comparison order sets. OrderSetEvaluationComponent
does not run it; ai_evaluation.comparison does, as the
mix test. It reads the comparison order sets in
the same long form as the order sets, as the comparison_sets input.
compared_with names the algorithm that produced them and is the output's
baseline.
evaluation = CodeDistributionShift(
name="paint_shift_vs_rules",
compared_with="rules",
attribute_types=["PAINT_COLOR"],
match_keys=["dealer_id", "model_id"],
draws=2000,
)
evaluate_order_sets(
order_sets,
[evaluation],
{EvaluationInput.COMPARISON_SETS: comparison_order_sets},
{
evaluation.name: select_types(
evaluation.selection(), order_sets.group_by(SET_ID).agg(), None
)
},
)
For one order set and one attribute type, the comparison sample is every comparison unit with a
code of the type whose match_keys values equal the order set's, whichever comparison order set
holds it. With n_k and c_k the order set's and the comparison sample's units of code k, and
N and C their sums:
observedis the total variation distance½ Σ_k |n_k / N − c_k / C|, from 0 for equal shares to 1 for disjoint codes;- each of
drawspermutations splits the pooledN + Cunits at random into samples of sizesNandC, andbaseline_p05,baseline_medianandbaseline_p95are the quantiles of their distances; p_valueis(1 + r) / (draws + 1), whereris the number of permutations at least as far apart asobserved;baseline_setsisdraws, which must be at least 100.
observed and p_value are null when the order set or the comparison sample has no unit with a
code of the type. The random generator is seeded like take-rate surprise's. The evaluation raises
when a unit of the comparison order sets has more than one code of an attribute type. A weight
selection's model_column must be one of match_keys.
Statistics
type | Value for counts n_k, N = Σ n_k | Null when |
|---|---|---|
unbiased_simpson | Σ n_k(n_k − 1) / (N(N − 1)), the probability that two different units share a code | N < 2 |
miller_madow_entropy | ln N − (Σ n_k ln n_k) / N + (K − 1) / (2N), where K is the number of codes present | Never |
A higher Simpson index means a more concentrated set, and so does a lower entropy. A concentration
evaluation therefore uses tail: upper with unbiased_simpson and tail: lower with
miller_madow_entropy.
Both estimators correct the bias of the plug-in estimator. That bias depends on N. Size matching
through size_ratio is still needed for the p-value, because the variance of either statistic
shrinks as N grows.
Both statistics treat an attribute type as one code per vehicle. When the code location loads,
OrderSetEvaluationComponent rejects an evaluation that lists one of
mutually_inclusive_attribute_types.
Assets
| Asset | Partitions | One row per |
|---|---|---|
<oem>/<market>/reporting/order_set_evaluations | Ordering run (run_id) | volume_type, evaluation, order set and attribute type, or volume_type, evaluation and order set |
<oem>/<market>/reporting/unit_rarities | Ordering run (run_id) | volume_type, configuration rarity evaluation and unit |
One materialization writes both assets. unit_rarities has no rows when no evaluation is a
configuration rarity.
order_set_evaluations reads the run's partition_date from ordering/runs. It is the date of the
warehouse partitions the run selected its candidates from. order_set_evaluations reads every input
at that date. See ADR-014 and
Input partitions. The inputs are consolidated/dealers,
consolidated/models, consolidated/features, featurization/features and
featurization/models, and each evaluation's input:
consolidated/inventory,featurization/inventoryandreporting/placed_orders, when an evaluation reads a reference;output/demand_take_rates, when an evaluation is take-rate surprise or configuration rarity;output/attribute_weights, when an evaluation selects its attribute types by weight.
order_set_evaluations raises when the run's partition_date is null, and when an input other than
reporting/placed_orders has no partition on that date, naming the input. It also raises on a
missing input partition for a run whose partition_date_source is run_date. A
reporting/placed_orders partition without rows, as before the first order is placed, cannot be
told apart from a missing one, so the population then leaves out no placed order.
Each row has:
- the run's
run_id,run_mode,run_timestampandpartition_date; - the order set's
group_bykeys; - the evaluation's columns;
weights_partition_date, the date of theoutput/attribute_weightspartition the evaluation's types were selected from, or null when the evaluation lists its types;volume_type,recommendedorfill, which names the units the row evaluates (see Recommended and fill units).
order_set_evaluations materializes when a new run partition is registered, like
Build Deltas. It is not materialized again when an input partition
arrives later. When a run's evaluation fails because an input partition is missing, the run's
order_set_evaluations partition has to be materialized again by hand once the input partition
exists. Its rows are then the same as if the input partition had existed at the first
materialization.
Adding an evaluation
A new statistic type, set generator type or evaluation type is implemented as a subclass of the
matching base class in packages/ai_evaluation/src/ai_evaluation/order_sets/:
| Base class | Module | Union the subclass is added to |
|---|---|---|
SetStatistic | statistics.py | AnySetStatistic |
SetGenerator | sets.py | AnySetGenerator |
Evaluation | evaluations.py | AnyEvaluation |
Each subclass has a type field with a Literal value. Setting type to that value in
defs.yaml selects the subclass. An Evaluation subclass implements:
reads, the input it reads;config_problems, which checks its configuration;input_columns, the columns of its input it reads, which are the only ones the component loads;_baseline_name, the value of the output'sbaselinecolumn;_compare, which compares the order sets with its input and returns its output rows as aLazyFrame.
Evaluation.evaluate_sets runs these steps in order:
- It raises when its input lacks one of the
input_columns. - It calls
_evaluate, which by default calls_compareand gives no unit rows.ConfigurationRarityoverrides_evaluateto return its unit rows too. - It adds the
evaluationandbaselinecolumns to the rows_evaluatereturns.
Evaluation.evaluate returns the rows of evaluate_sets. evaluate_order_sets in ai_evaluation
runs a list of evaluations. It returns the evaluations' rows as one LazyFrame and their unit rows
as another, and collects nothing. It caches the order sets and each input in the plan, so the
caller's single pl.collect_all call on both frames reads each of them once. Each evaluation
attaches its checks to its output with ai_core.polars.with_row_checks, because a check on an
input can be skipped when a later join discards that input. OrderSetEvaluationComponent passes
its list of evaluations to evaluate_order_sets, so it needs no change for a new evaluation type
that reads an existing input. An evaluation that uses only existing statistic, set generator and
evaluation types needs no code. It is a new entry under evaluations.
Comparing two sets of order sets
ai_evaluation.comparison compares two OrderSets, a baseline and another side,
such as the recommendations of two versions of candidate selection or of two
ordering runs. An OrderSets (ai_evaluation.order_sets.units) holds:
units, one row per unit, withunit_id, the order-setkeyscolumns, and optionallyprice;codes, one row per unit and code, withunit_id,attribute_type,attribute_valueand optionally numeric columns of a unit's code, such as the take rate a selection read for it.
It is stored as a directory of units.parquet, codes.parquet and keys.json.
candidate_selection.diagnostics writes that directory from a file of candidate
selection's recommendations or from a past ordering run, without importing
ai_evaluation. OrderSets.read checks the columns, the unit ids and the code
columns.
compute_diff(baseline, other, levels, settings, evaluations) compares the two at
the global level and per value of each key in levels:
| Section | Needs | Measures |
|---|---|---|
| Unit counts | always | units per side |
| Attribute mix | always | each code's share per side, and a code distribution shift test per attribute type |
| Code value shift | a code column on both sides | the mean value of each side's codes per attribute type, such as the mean take rate of the selected options, and a test of the difference |
| Code value drift | a code column on both sides | whether both sides read the same value for the same order set and code |
| Price | price on both sides | quantiles, Kolmogorov–Smirnov statistic and Wasserstein distance |
| Evaluations | an EvaluationRequest | each side's results of the requested evaluations, compared order set by order set |
The mix test is code distribution shift, reading the baseline as the comparison order sets. It leaves out an attribute type for which a unit of either side has more than one code, such as options or packages, and the report names the types it left out. At a level, it also leaves out the units without a value of the level's key.
The code value shift takes each unit's mean value over its codes of a type. Its test
splits both sides' unit values at random into samples of the two sides' sizes, and
p_value is (1 + r) / (draws + 1), where r is the number of splits whose
difference in means is at least as far from 0 as the observed one.
ComparisonSettings sets the number of random draws of both tests, their seed and
the significance level. Each test's p_adjusted is the Benjamini–Hochberg
adjustment of its p_value within its level, and for the code value shift within
its level and code column. A test whose p_adjusted is below the significance level
counts as a shift, which keeps the expected share of false alarms among the counted
shifts below that level.
An EvaluationRequest names evaluations of AnyEvaluation, their inputs, and the
order-set keys whose order sets they score. Each evaluation runs on both sides. For
each evaluation and attribute type, the comparison counts the order sets each side
has below the significance level, and the order sets whose p_value is higher on
the other side (less unusual) or lower (more unusual). Per evaluation, it counts the
order sets each side has flagged on any attribute type, and the order sets that are
less or more unusual on the other side, by most of their attribute types, and tests
those two counts with a sign test. The flagged counts use unadjusted p-values, so
about one order set in twenty per side is flagged by chance.
python -m ai_evaluation.comparison compare and render write the comparison as
Parquet and render it as Markdown, such as for a pull request comment. compare --subset also compares the units of some key values, such as one model, as a comparison
of its own. render takes several comparisons, such as one per OEM, and writes one
document. Each comparison shows its summary and test tables, then its detail sections in
a collapsed block, then the summary and test tables of each of its subsets. Every
comparison has the same sections in the same order, and a section whose data the inputs
lack says so. The legend and notes appear once, at the top. Each summary and test row
has a status: green for no difference beyond chance, yellow for a difference to look at,
and red for a large difference beyond chance or a fit to the evaluations' inputs that is
worse beyond chance. The take rates follow their adjusted tests, prices a permutation
test of the mean price, and each evaluation a sign test of how many order sets are more
unusual on the other side against how many are less. The price distance has no status.
Units are compared by count.
Known limitations
Every statistic must be computable from code counts and set_size. See
ADR-011.