Skip to main content

Candidate Coverage Checks

The asset checks on this page run over <oem>/<market>/output/candidates_output, the table that unions real inventory configurations with any synthetic ones. They live in ai_dagster.components.candidate_coverage_check.

checkasksbuilder
models_meet_candidate_baselinedoes every orderable model have its baseline of candidates?build_model_coverage_check
features_covered_on_models_carrying_candidateson a model that has candidates, does each orderable option code have its baseline of candidates, and do as many leave it out?build_feature_coverage_check
demand_covered_by_candidateson a model that meets its baseline, are the candidates spread across each attribute type's values in line with demand?build_demand_coverage_check
demand_covered_given_each_featurefor an option code that meets its baseline, are the candidates having the code spread across the values of each attribute type that choosing the code does not settle in line with demand?build_feature_demand_coverage_check

The checks form two families:

  • The baseline checks ask whether we can recommend for a model. A count per model cannot show whether each orderable code is offered, because a model can have thousands of candidates that never offer one paint. The feature check therefore counts per code.
  • The variety checks ask whether the candidates vary across the values dealers order, in proportion to how often they order each one.

None of the checks is blocking. A coverage gap is a reporting concern: the downstream assets work correctly on the models that are covered.

The models in scope​

A model with no candidates has no rows in candidates_output, so the checks read the models in scope from consolidated/models and join the candidate counts onto them. Two settings select the models:

  • orderable_models_filter is a SQL predicate over consolidated/models that says what makes a model orderable, such as code_guides_exists AND feature_catalog_exists. Each OEM's defs.yaml records which columns it picked and why.
  • active_model_years is the number of newest model years still being allocated, counted back from the newest year the predicate matches. It defaults to 2 and has no upper bound. It is separate from the predicate so that a catalog backfill that widens the years a feed covers does not widen the models in scope.

The model years never reach below the partition's own year, because candidates_output inner-joins the models on the same bound.

A model the predicate does not select is not measured and appears in no metadata. Prefer an over-wide predicate: a model wrongly included fails and gets investigated, while a model wrongly excluded goes unnoticed.

The configuration space​

Both baselines scale with a model's configuration space U, the number of configurations a dealer can order. U is computed from the model's orderable option codes and their attribute types in featurization/features. An option code that consolidated/features lists once per parent package counts once.

The orderable codes are partitioned into groups, one per attribute type. A code with several types joins the largest of its groups, and a tie goes to the group whose name sorts first. A type such as PAINT_MATERIAL, which is on exactly the codes that have a PAINT_COLOR, therefore adds nothing. A group counts option codes, not attribute values: eight paint codes that map to seven colors are eight alternatives.

states(group) = k    a group of k codes, of which a car has exactly one
= 2 a code of a mutually inclusive type, which a car has or not
= 1 such a code that the model has as standard equipment
U = product over the model's groups of states(group)

A group has k states because a car has exactly one value of every attribute type. The take_rate_sums_<label> checks on attribute_stats assert this per OEM. mutually_inclusive_types in candidate_baselines declares the exception, a type whose codes a car can have together. Each code of such a type is a group of its own. No OEM declares one, and a type listed by mistake multiplies the space by 2**k instead of k.

A code is standard equipment on a model when standard_equipment_filter selects every orderable row of the code.

U is an upper bound. It treats the groups as independent and ignores the OEM's compatibility rules, so that every OEM's space is measured the same way. It does not bound the number of candidates: candidates that differ only in codes without an attribute type are one configuration of U.

Baseline checks​

Per model​

baseline(model) = max(min_candidates_per_model,
min(reachable_fraction x U, ceil(factor x U ** exponent)))

models_meet_candidate_baseline compares each model's candidate count with this baseline. The baseline comes from the catalog, not from candidate counts, because the generator's max_optional_depth and max_per_base bound the counts. A model whose orderable codes have no attribute type has no space to scale by, and the check raises on it. That gap is in featurization/features.

Per feature​

U_F                = U / states(group(F))
baseline(model, F) = min(reachable_fraction x U_F, ceil(factor x U_F ** exponent))

features_covered_on_models_carrying_candidates compares the candidate count of each orderable option code with this baseline, which scales with the space that remains once the code is chosen. It does not measure a code without an attribute type, or a model with no candidates, which the model check reports.

The check counts candidates per model and option code. A candidate has a code when its feature_ids include an orderable row of the code or a row of the code inside a package. With an orderable_features_filter such as Stellantis's is_orderable_standalone, only the standalone row is orderable, so a candidate that takes the code with its package has the package's row instead. A gap row names the code by its standalone row where it has one.

The check measures each code in two ways:

  • presence: the candidates that have the code, against the baseline;
  • omission: the candidates that leave the code out, against the baseline. It applies only to a code a car can leave out, one of a mutually inclusive type that is not standard equipment, so it measures no code while no OEM declares such a type.

Findings and severity​

Each baseline check reports two findings:

  • a model or code on no candidate, or a code on every candidate, fails the check at ERROR, because the candidates cannot offer something at all;
  • a model or code on too few candidates, or a code that too few candidates leave out, fails it at WARN, because that threshold is a calibration judgment.

Gap rows are sorted by pct_of_baseline, lowest first. The feature check lists at most MAX_GAP_ROWS of them, so while more codes are on no candidate than that, its gap list holds only those codes. The counts are never capped.

Baseline check metadata​

Most keys name what they count. The others:

  • gaps_per_model lists every model below its baseline, with its candidate count, baseline and pct_of_baseline.
  • models_carrying_candidates counts the models in scope that have at least one candidate.
  • coverage_gaps_per_model_feature and omission_gaps_per_model_feature have one row per model and code, with a feature_id of the code and its name, because the ids are hashes.
  • The coverage_ and omission_ prefixes keep each measure's keys together in Dagster's alphabetical order.

Parameters​

The per-model and per-feature factor and exponent are fitted once across OEMs, on the models each OEM allocated in its most recent cycle.

  • The exponent makes the baseline grow far more slowly than U. Covering every pair of codes across k groups of size w takes on the order of w**2 x ln(k) configurations, so a fixed fraction of U would ask for far too many.
  • reachable_fraction, 0.5 by default, caps the baseline for a small space, which a model is expected to cover rather than sample. The cap decides the baseline while U is below (factor / reachable_fraction) ** (1 / (1 - exponent)), about 53 configurations per model and 22 per option code.
  • Per model, exponent = 0.20 is bounded by the slope of log candidates against log U within an OEM. factor = 12.0 is the largest constant that lets 90% of Jeep models with candidates pass.
  • Per feature, the data admits any exponent from 0 to 0.30. exponent = 0.10 keeps the baseline rising with the space without it dropping to single digits. factor = 8.0 is below the tightest fit, so that one thinly supplied code does not set the threshold. This fit uses only codes on at least twelve stocked cars in the OEM's inventory export.

A baseline fitted on past candidate counts detects a regression against what already worked. It does not show that a count is enough to recommend well, which needs an ablation or a backtest against realized orders.

Candidate variety, prioritized by demand​

A baseline check counts candidates. It does not say how the candidates vary across an attribute type's values: a model can meet its baseline with every candidate in its least-ordered paint. The variety checks measure that variety, and demand sets its priorities. A value that dealers order more often needs more candidates.

The checks read demand from <oem>/<market>/output/demand_take_rates, which candidate selection scores the candidates against.

Coverage of an attribute type​

Each variety check measures a set of candidates against a candidate count baseline N, which is a model's baseline or an option code's. Demand gives each value of an attribute type a cap, which is the number of candidates the value needs. A value's candidates count toward the type's coverage up to the value's cap:

cap(value)     = share(value) x N
coverage(type) = sum over the type's values of min(candidates(value), cap(value)) / N

Take a model with a candidate count baseline of 100 and paint demand of 50% black, 30% white and 20% blue. The caps are 50, 30 and 20. If the model has 95 black, 45 white and no blue candidates, they count for 50 + 30 + 0, a paint coverage of 80%. Candidates beyond one value's cap do not make up for another value that has too few, and candidates of a value missing from demand add nothing.

share(value) is the value's take_rate_col of demand summed over dealers, divided by the type's total on the model. A dealer's rates over a type sum to about one, so each dealer weighs about the same. Demand for a value that exclude_attributes names, such as Stellantis's NO_PAINT_COLOR, is left out, because candidates_output drops every candidate that has such a value.

A group is one set of candidates and one attribute type. A group whose coverage is below min_coverage fails the check at WARN.

What each check measures​

demand_covered_by_candidates measures each model that meets its baseline in models_meet_candidate_baseline, with N the model's baseline.

demand_covered_given_each_feature measures each option code that meets its baseline in features_covered_on_models_carrying_candidates. It measures the candidates that have the code, counted as in that check, and N is the code's baseline.

A model without demand has no shares to set the caps, so neither check measures it. A model missing from demand is a gap for the demand component to report.

The per-feature check skips each attribute type that choosing the code settles:

  • each of the code's own types. Every candidate that has a paint code has the same paint.
  • each type B whose codes on the model, orderable or not, all share one of the code's types. The baselines assume that a car has one code of each type that mutually_inclusive_types does not name. Under that assumption a car with the code has no other code of B, and so has B's fill value. The fill value is FALSE for a binary type and NO_<type> for any other type. Where only metallic paints have a PAINT_FINISH, a solid paint settles the finish. Where a candidate having the code has a value of B other than the fill value, the assumption does not hold for the code, and B is measured. The check detects such a value only when demand names it or names the fill value.

The check does not read the OEM's rules, so a value that a rule excludes from candidates having the code is reported as a gap.

attribute_types names the types both checks measure.

Variety check metadata​

Most keys name what they count. The others:

  • models_measured and features_measured count the models or codes that meet their baseline and form at least one group.
  • gaps_per_model_attribute_type and gaps_per_model_feature_attribute_type list at most MAX_GAP_ROWS groups, lowest coverage first. Each row has the group's coverage, baseline and candidate count, and in short_values the values furthest below their cap. The per-feature rows also name the code by a feature_id and its name.

min_coverage is in each check's description, because it does not change from run to run.

Configuration​

type: ai_dagster.components.CandidatesOutputComponent
attributes:
coverage_check:
orderable_models_filter: code_guides_exists AND feature_catalog_exists
active_model_years: 2
orderable_features_filter: is_configurable AND is_orderable_standalone
standard_equipment_filter: standard
min_candidates_per_model: 1
candidate_baselines: # every field at its default
mutually_inclusive_types: []
reachable_fraction: 0.5
per_model_factor: 12.0
per_model_exponent: 0.20
per_feature_factor: 8.0
per_feature_exponent: 0.10
demand_coverage:
demand: stellantis/us/output/demand_take_rates
take_rate_col: sales_hl90_my_weighted_take_rate
attribute_types: [PAINT_COLOR, UPHOLSTERY_COLOR, WHEEL_DIAMETER]
min_coverage: 0.9

coverage_check, standard_equipment_filter and demand_coverage are required, so that a setting left out is an error rather than a silent opt-out. Set to null:

  • coverage_check to turn off all checks;
  • standard_equipment_filter to name no standard equipment;
  • demand_coverage to turn off the variety checks.

Prefer configuring coverage_check to turning it off: while the ordering feed has no models for the partition, nothing is in scope and every check passes.

Onboarding an OEM​

The consolidated merge writes a {source_group}_exists flag for every source group an OEM declares. To find the predicate, group the OEM's models by model_year and count each flag. The source group whose older model years drop to zero is the ordering catalog. An OEM with a single source group names its one flag, and active_model_years alone decides the scope.

Audi's filters​

Both filters read the order guide's is_orderable and keep a row where it is null.

  • A model is out of scope once the guide's contents page lists it as closed for ordering.
  • A standalone code is out of scope once the production window in effect no longer lists it. is_configurable does not follow production windows. A row with a parent_package_code is exempt, because the guide sets is_orderable false on a trim's content rows.

Task sizing​

Each check is a step of its own, built with the component's op_tags, and inherits no allocation from the asset it checks. Besides the candidates, the checks read the models and the exploded feature catalog, which the asset reads too. The variety checks also read demand, featurization/inventory and, where the OEM has synthetic candidates, featurization/synthetic. A failure in one check does not stop the others. The per-feature variety check joins every code of a candidate with every value of it, so that join runs on integer indexes (see Performance).