Candidate Coverage Checks
The asset checks on this page run over <oem>/<market>/output/candidates_output, the
table that unions real inventory configurations with any synthetic ones. They live in
ai_dagster.components.candidate_coverage_check.
| check | asks | builder |
|---|---|---|
models_meet_candidate_baseline | does every orderable model have its baseline of candidates? | build_model_coverage_check |
features_covered_on_models_carrying_candidates | on a model that has candidates, does each orderable option code have its baseline of candidates, and do as many leave it out? | build_feature_coverage_check |
demand_covered_by_candidates | on a model that meets its baseline, are the candidates spread across each attribute type's values in line with demand? | build_demand_coverage_check |
demand_covered_given_each_feature | for an option code that meets its baseline, are the candidates having the code spread across the values of each attribute type that choosing the code does not settle in line with demand? | build_feature_demand_coverage_check |
The checks form two families:
- The baseline checks ask whether we can recommend for a model. A count per model cannot show whether each orderable code is offered, because a model can have thousands of candidates that never offer one paint. The feature check therefore counts per code.
- The variety checks ask whether the candidates vary across the values dealers order, in proportion to how often they order each one.
None of the checks is blocking. A coverage gap is a reporting concern: the downstream assets work correctly on the models that are covered.
The models in scope
A model with no candidates has no rows in candidates_output, so the checks read the
models in scope from consolidated/models and join the candidate counts onto them. Two
settings select the models:
orderable_models_filteris a SQL predicate overconsolidated/modelsthat says what makes a model orderable, such ascode_guides_exists AND feature_catalog_exists. Each OEM'sdefs.yamlrecords which columns it picked and why.active_model_yearsis the number of newest model years still being allocated, counted back from the newest year the predicate matches. It defaults to 2 and has no upper bound. It is separate from the predicate so that a catalog backfill that widens the years a feed covers does not widen the models in scope.
The model years never reach below the partition's own year, because candidates_output
inner-joins the models on the same bound.
A model the predicate does not select is not measured and appears in no metadata. Prefer an over-wide predicate: a model wrongly included fails and gets investigated, while a model wrongly excluded goes unnoticed.
The configuration space
Both baselines scale with a model's configuration space U, the number of configurations
a dealer can order. U is computed from the model's orderable option codes and their
attribute types in featurization/features. An option code that consolidated/features
lists once per parent package counts once.
The orderable codes are partitioned into groups, one per attribute type. A code with
several types joins the largest of its groups, and a tie goes to the group whose name sorts
first. A type such as PAINT_MATERIAL, which is on exactly the codes that have a
PAINT_COLOR, therefore adds nothing. A group counts option codes, not attribute values:
eight paint codes that map to seven colors are eight alternatives.
states(group) = k a group of k codes, of which a car has exactly one
= 2 a code of a mutually inclusive type, which a car has or not
= 1 such a code that the model has as standard equipment
U = product over the model's groups of states(group)
A group has k states because a car has exactly one value of every attribute type. The
take_rate_sums_<label> checks on attribute_stats assert this per OEM.
mutually_inclusive_types in candidate_baselines declares the exception, a type whose
codes a car can have together. Each code of such a type is a group of its own. No OEM
declares one, and a type listed by mistake multiplies the space by 2**k instead of k.
A code is standard equipment on a model when standard_equipment_filter selects every
orderable row of the code.
U is an upper bound. It treats the groups as independent and ignores the OEM's
compatibility rules, so that every OEM's space is measured the same way. It does not bound
the number of candidates: candidates that differ only in codes without an attribute type
are one configuration of U.
Baseline checks
Per model
baseline(model) = max(min_candidates_per_model,
min(reachable_fraction x U, ceil(factor x U ** exponent)))
models_meet_candidate_baseline compares each model's candidate count with this baseline.
The baseline comes from the catalog, not from candidate counts, because the generator's
max_optional_depth and max_per_base bound the counts. A model whose orderable codes
have no attribute type has no space to scale by, and the check raises on it. That gap is
in featurization/features.
Per feature
U_F = U / states(group(F))
baseline(model, F) = min(reachable_fraction x U_F, ceil(factor x U_F ** exponent))
features_covered_on_models_carrying_candidates compares the candidate count of each
orderable option code with this baseline, which scales with the space that remains once the
code is chosen. It does not measure a code without an
attribute type, or a model with no candidates, which the model check reports.
The check counts candidates per model and option code. A candidate has a code when its
feature_ids include an orderable row of the code or a row of the code inside a package.
With an orderable_features_filter such as Stellantis's is_orderable_standalone, only
the standalone row is orderable, so a candidate that takes the code with its package has
the package's row instead. A gap row names the code by its standalone row where it has
one.
The check measures each code in two ways:
- presence: the candidates that have the code, against the baseline;
- omission: the candidates that leave the code out, against the baseline. It applies only to a code a car can leave out, one of a mutually inclusive type that is not standard equipment, so it measures no code while no OEM declares such a type.
Findings and severity
Each baseline check reports two findings:
- a model or code on no candidate, or a code on every candidate, fails the check at ERROR, because the candidates cannot offer something at all;
- a model or code on too few candidates, or a code that too few candidates leave out, fails it at WARN, because that threshold is a calibration judgment.
Gap rows are sorted by pct_of_baseline, lowest first. The feature check lists at most
MAX_GAP_ROWS of them, so while more codes are on no candidate than that, its gap list
holds only those codes. The counts are never capped.
Baseline check metadata
Most keys name what they count. The others:
gaps_per_modellists every model below its baseline, with its candidate count, baseline andpct_of_baseline.models_carrying_candidatescounts the models in scope that have at least one candidate.coverage_gaps_per_model_featureandomission_gaps_per_model_featurehave one row per model and code, with afeature_idof the code and its name, because the ids are hashes.- The
coverage_andomission_prefixes keep each measure's keys together in Dagster's alphabetical order.
Parameters
The per-model and per-feature factor and exponent are fitted once across OEMs, on the
models each OEM allocated in its most recent cycle.
- The exponent makes the baseline grow far more slowly than
U. Covering every pair of codes acrosskgroups of sizewtakes on the order ofw**2 x ln(k)configurations, so a fixed fraction ofUwould ask for far too many. reachable_fraction, 0.5 by default, caps the baseline for a small space, which a model is expected to cover rather than sample. The cap decides the baseline whileUis below(factor / reachable_fraction) ** (1 / (1 - exponent)), about 53 configurations per model and 22 per option code.- Per model,
exponent = 0.20is bounded by the slope of log candidates against logUwithin an OEM.factor = 12.0is the largest constant that lets 90% of Jeep models with candidates pass. - Per feature, the data admits any exponent from 0 to 0.30.
exponent = 0.10keeps the baseline rising with the space without it dropping to single digits.factor = 8.0is below the tightest fit, so that one thinly supplied code does not set the threshold. This fit uses only codes on at least twelve stocked cars in the OEM's inventory export.
A baseline fitted on past candidate counts detects a regression against what already worked. It does not show that a count is enough to recommend well, which needs an ablation or a backtest against realized orders.
Candidate variety, prioritized by demand
A baseline check counts candidates. It does not say how the candidates vary across an attribute type's values: a model can meet its baseline with every candidate in its least-ordered paint. The variety checks measure that variety, and demand sets its priorities. A value that dealers order more often needs more candidates.
The checks read demand from <oem>/<market>/output/demand_take_rates, which candidate
selection scores the candidates against.
Coverage of an attribute type
Each variety check measures a set of candidates against a candidate count baseline N,
which is a model's baseline or an option code's. Demand gives each value of an attribute
type a cap, which is the number of candidates the value needs. A value's
candidates count toward the type's coverage up to the value's cap:
cap(value) = share(value) x N
coverage(type) = sum over the type's values of min(candidates(value), cap(value)) / N
Take a model with a candidate count baseline of 100 and paint demand of 50% black, 30% white and 20% blue. The caps are 50, 30 and 20. If the model has 95 black, 45 white and no blue candidates, they count for 50 + 30 + 0, a paint coverage of 80%. Candidates beyond one value's cap do not make up for another value that has too few, and candidates of a value missing from demand add nothing.
share(value) is the value's take_rate_col of demand summed over dealers, divided by
the type's total on the model. A dealer's rates over a type sum to about one, so each
dealer weighs about the same. Demand for a value that exclude_attributes names, such as
Stellantis's NO_PAINT_COLOR, is left out, because candidates_output drops every
candidate that has such a value.
A group is one set of candidates and one attribute type. A group whose coverage is below
min_coverage fails the check at WARN.
What each check measures
demand_covered_by_candidates measures each model that meets its baseline in
models_meet_candidate_baseline, with N the model's baseline.
demand_covered_given_each_feature measures each option code that meets its baseline in
features_covered_on_models_carrying_candidates. It measures the candidates that have the
code, counted as in that check, and N is the code's baseline.
A model without demand has no shares to set the caps, so neither check measures it. A model missing from demand is a gap for the demand component to report.
The per-feature check skips each attribute type that choosing the code settles:
- each of the code's own types. Every candidate that has a paint code has the same paint.
- each type
Bwhose codes on the model, orderable or not, all share one of the code's types. The baselines assume that a car has one code of each type thatmutually_inclusive_typesdoes not name. Under that assumption a car with the code has no other code ofB, and so hasB's fill value. The fill value isFALSEfor a binary type andNO_<type>for any other type. Where only metallic paints have aPAINT_FINISH, a solid paint settles the finish. Where a candidate having the code has a value ofBother than the fill value, the assumption does not hold for the code, andBis measured. The check detects such a value only when demand names it or names the fill value.
The check does not read the OEM's rules, so a value that a rule excludes from candidates having the code is reported as a gap.
attribute_types names the types both checks measure.
Variety check metadata
Most keys name what they count. The others:
models_measuredandfeatures_measuredcount the models or codes that meet their baseline and form at least one group.gaps_per_model_attribute_typeandgaps_per_model_feature_attribute_typelist at mostMAX_GAP_ROWSgroups, lowest coverage first. Each row has the group's coverage, baseline and candidate count, and inshort_valuesthe values furthest below their cap. The per-feature rows also name the code by afeature_idand its name.
min_coverage is in each check's description, because it does not change from run to
run.
Configuration
type: ai_dagster.components.CandidatesOutputComponent
attributes:
coverage_check:
orderable_models_filter: code_guides_exists AND feature_catalog_exists
active_model_years: 2
orderable_features_filter: is_configurable AND is_orderable_standalone
standard_equipment_filter: standard
min_candidates_per_model: 1
candidate_baselines: # every field at its default
mutually_inclusive_types: []
reachable_fraction: 0.5
per_model_factor: 12.0
per_model_exponent: 0.20
per_feature_factor: 8.0
per_feature_exponent: 0.10
demand_coverage:
demand: stellantis/us/output/demand_take_rates
take_rate_col: sales_hl90_my_weighted_take_rate
attribute_types: [PAINT_COLOR, UPHOLSTERY_COLOR, WHEEL_DIAMETER]
min_coverage: 0.9
coverage_check, standard_equipment_filter and demand_coverage are required, so that a
setting left out is an error rather than a silent opt-out. Set to null:
coverage_checkto turn off all checks;standard_equipment_filterto name no standard equipment;demand_coverageto turn off the variety checks.
Prefer configuring coverage_check to turning it off: while the ordering feed has no
models for the partition, nothing is in scope and every check passes.
Onboarding an OEM
The consolidated merge writes a {source_group}_exists flag for every source group an OEM
declares. To find the predicate, group the OEM's models by model_year and count each
flag. The source group whose older model years drop to zero is the ordering catalog. An
OEM with a single source group names its one flag, and active_model_years alone decides
the scope.
Audi's filters
Both filters read the order guide's is_orderable and keep a row where it is null.
- A model is out of scope once the guide's contents page lists it as closed for ordering.
- A standalone code is out of scope once the production window in effect no longer
lists it.
is_configurabledoes not follow production windows. A row with aparent_package_codeis exempt, because the guide setsis_orderablefalse on a trim's content rows.
Task sizing
Each check is a step of its own, built with the component's op_tags, and inherits no
allocation from the asset it checks. Besides the candidates, the checks read the models
and the exploded feature catalog, which the asset reads too. The variety checks also read
demand, featurization/inventory and, where the OEM has synthetic candidates,
featurization/synthetic. A failure in one check does not stop the others. The
per-feature variety check joins every code of a candidate with every value of it, so that
join runs on integer indexes (see
Performance).