Synthetic Candidates
SyntheticCandidatesComponent (in packages/ai_dagster) builds the
<oem>/<market>/synthetic_candidates asset. The asset's description in Dagster states what
the table holds.
Inputs are consolidated/{inventory, features, models}, and consolidated/rules when
rules_from is rules.
A cohort is a year-independent model identity (base_model_id). Its target-year
instance is one model_id. YAML rules are scoped to cohorts, and each candidate belongs to
one cohort.
For a cohort that no YAML rule is scoped to, the component generates candidates the same way as for any other cohort, with an empty rule set. The codes of the cohort's base vehicles are still kept or retired as What the target year offers describes.
Layered architecture
The component splits into three layers. OEM-specific column names (model_code,
trim_identifier, drive, …) appear only at rule authoring. From resolution onward the
pipeline, including its output, uses a single opaque identifier, model_id.
- Authoring, OEM-dependent. Every rule has a
RuleScope. Itsmodel_filtersmaps aconsolidated/modelscolumn name to a case-insensitive regex ({model_code: "^JG"},{drive: "^AWD$"}), and every entry must match for the rule to apply. Anchor a pattern (^X$) for an exact match; an empty map matches every cohort. - Resolution, the boundary.
_resolve_rule_model_idsevaluates everymodel_filtersagainst the target-yearconsolidated/modelsrows once, producingapplies: {id(rule) → frozenset[model_id]}, so the same rule covers different models as models come and go between partitions._resolve_target_cohortsmaps each base-yearmodel_idto the target-yearmodel_ids its inventory generates candidates for (see Candidates for a new trim)._validate_model_filtersrejects a filter key that names no models column. - Core generation and output, OEM-agnostic.
_generatescopes rules by set membership (tgt_model_id in applies[id(rule)]) and keys candidates bymodel_id. The output hasinventory_id,model_id,dealer_id,feature_ids,model_year,base_model_id,msrpandis_synthetic, and no OEM descriptive columns. A consumer needingmodel_codeortrimjoinsconsolidated/modelsbymodel_id. The set algebra lives inai_dagster.synthetic.generate, which imports neither Dagster nor polars.
Generation keys on the opaque model_id, so it makes no assumption about an OEM's
identity fields. It bridges model years on base_model_id, because a cohort has a
different model_id in each year.
Candidates for a new trim
Each target_combinations entry is a TargetCombination, whose model_filters
selects target-year cohorts. By default, a target's candidates come from the inventory
of the base-year cohort that shares its base_model_id. A renamed model code bridges
model years this way once base_model_id_aliases maps it onto its predecessor.
A trim that is new in the target year has no base-year cohort. Its base_filters
selects the base-year cohorts whose inventory generates its candidates instead. The
filters match as model_filters does, applied to base-year consolidated/models rows:
target_combinations:
- model_filters: {model_code: ^KMJH74$, trim_identifier: ^H$}
base_filters: {model_code: ^KMJP74$, trim_identifier: ^Q$}
Unlike an alias, base_filters leaves the base-year cohort's identity unchanged:
- The base-year cohort's inventory still generates candidates for its own target.
- The new trim's candidates have the new trim's
model_idandbase_model_id. - Rules scope by target
model_id, so rules that match only the new trim turn the base-year cohort's codes into the new trim's codes.
What the target year offers
A base vehicle's code is kept when the target-year cohort has a consolidated/features
row for it, and retired otherwise. The table keeps a row after its sources stop
listing it. With availability_sources set, a configurable target-year row counts only
when one of those source groups lists it on the partition date. Base-year rows always
count, so a base vehicle's feature IDs still resolve. Leave out a source group that lists
options a trim cannot be ordered with, such as Stellantis' equipment_categories.
A non-configurable target-year row that those source groups do not list still counts,
unless they list its code on the partition date for another model of the same
model_group_id and not for this one. For example, Stellantis' equipment_categories
lists every Wrangler Sport package under each Sport trim, but the code guides list 23S for
Sport S only, so 23S does not count on Sport B.
With package_content_sources set, a target-year row inside a package does not count when
it is replaced. A row is replaced when all of these hold:
- No source group of
consolidated/featureslists the row on the partition date. - The groups that
package_content_sourcesnames did not list the row on the last date they listed itsmodel_id. - On that date, they listed a row of the same package on the same
model_idthat they first listed after they first listed the replaced row.
While those groups list a model, the last date they listed its model_id is the
partition date. When a source stops listing a whole model, the rows of its last listing
still decide which of the model's rows are replaced.
A row that those groups stopped listing still counts when they list no such newer row,
because a source can stop listing a package's content while the package still includes
it. For example, the older Audi catalog year of the 2027 A6 Sportback e-tron lists
Adaptive cruise assist (GJ7) in the Prestige (WPT) and Premium Plus (WPS) packages, and
the newer one lists Adaptive cruise assist plus (GJ8) in WPT instead. With
package_content_sources: [catalog], the GJ7 row in WPT is replaced, and the GJ7 row in
WPS still counts.
With package_content_sources set, the asset's materialization metadata records the
number of replaced rows as replaced_package_content_rows. The run log lists each
replaced row by model_id, package and code.
optional_replaces, optional_adds and optional_add_groups skip a code that the
target-year cohort has no row for. An optional_add_groups entry with such a code is
skipped as a whole. mandatory_adds, mandatory_expands and mandatory_replaces are not
filtered. A candidate to which one of them adds such a code is dropped.
Rule types
Every rule applies to the cohorts its model_filters selects (see
Layered architecture).
Mandatory rules change every candidate of the matching cohort. An optional rule offers a choice of operations, and one of them leaves the candidate unchanged. Each optional rule therefore multiplies the number of candidates.
| Rule | Effect |
|---|---|
mandatory_replaces | 1:1 rename of a code that was relabeled in the target year. |
mandatory_expands | Replace from_codes with exactly one of to_codes. Unlike an optional rule, it cannot leave the candidate unchanged. Every candidate that has one of from_codes takes one of to_codes. |
mandatory_adds | codes are present if and only if every requires code is present. Use it for package content valid only alongside its parent. See How mandatory_adds attaches codes. |
mandatory_removes | Remove codes that the target year still offers. A code the target year does not offer retires without a rule, and a code that optional_adds or optional_add_groups offers is removed from the base vehicle before generation. With requires set, codes are removed only when every requires code is among the base vehicle's codes, configurable or not. Use it for a code that is valid only while another code is absent, such as a package that cannot be combined with a specific paint. |
standard_adds | Unconditionally attach target-year content that is standard and non-configurable. Use it for content with no configurable parent, which neither the 1:1 rename of mandatory_replaces nor the requires list of mandatory_adds can attach. Codes that do not resolve in the target catalog are skipped. A code that mandatory_adds attaches replaces a standard_adds code of the same exclusive group. A configurable base code that a package upgrade replaces therefore also belongs here. |
optional_replaces | Replace the candidate's member of an exclusive group with one of to_codes, or keep the member it has. |
optional_adds | Optionally add one of add_codes, which are mutually exclusive alternatives. Use a single-element list for an independent toggle. |
optional_add_groups | Optionally add every code in codes together, or none of them. A subset is never added, so codes that only make sense together belong here rather than in optional_adds. |
mutually_exclusive_groups | Codes of which a candidate may have at most one (see Exclusive groups). |
must_have_one_of | Require at least one of codes on every candidate of the matching cohort. A config carrying none of them is dropped. |
An entry of to_codes, in mandatory_expands and optional_replaces, is one code or a
list of codes that the replacement adds together. A list entry maps a trim package whose
content the target year splits across several packages onto combinations of those
packages. In the following example, a Q5 with the Prestige package (WPT) gets either the
Executive (WPS) and Driver assistance (WPR) packages, or those two and the Vorsprung
package (PPZ):
mandatory_expands:
- model_filters: {model_code: ^GUBAAY$}
group: WPT
from_codes: [WPT]
to_codes: [[WPS, WPR], [WPS, PPZ, WPR]]
Exclusive groups
An exclusive group is a set of codes of which a candidate may have at most one. Its members are:
- the
member_codesof themutually_exclusive_groupsentries whosegroupis the group's name; - with
rules_from: rules, the codes of the cohort'sonly_one_ofrule of that name. A rule is named by itscode, or by itsrule_idwhen it has no code. An entry whosegroupnames such a rule adds itsmember_codesto the rule's group.
A candidate is checked against an only_one_of rule as
Compatibility rules states, and against the entries of
mutually_exclusive_groups.
Each of the following rules names a group:
mandatory_expandsoptional_replacesoptional_addsoptional_add_groups
A code that such a rule adds replaces every member of the group that the candidate has, as a configurable option or as fixed equipment.
How mandatory_adds attaches codes
requires is checked on each candidate after the optional rules, mandatory_expands and
the earlier mandatory_adds entries have changed it. The content is therefore attached
when optional_adds, optional_replaces or mandatory_expands adds its package, and is
absent when one of them removes the package.
An attached code replaces the other members of each exclusive group it belongs to, including fixed equipment. For example, an upgrade that a package includes replaces the base code.
Where entries attach different members of one exclusive group, the candidate is dropped,
unless supersede_mandatory_adds is true. When it is true, an entry that another of them
supersedes does not attach its member. An entry supersedes another when its requires
codes imply every requires code of the other entry, and the other entry's requires
codes do not imply all of its own. A code implies every code it requires, directly or
through other codes. A code requires the THEN code of an inclusion whose IF part is that
code alone and whose THEN part is one code. Only an inclusion that the component checks
counts (see Compatibility rules). ordered_packages_filter does
not change which inclusions count.
On the 2027 Q5, the Vorsprung package (PPZ) requires the Executive (WPS) and Driver assistance (WPR) packages, so its OLED taillights (8VY) replace the Executive package's animated taillights (8VP).
After the first attachment, the component attaches the entries again, and withholds the
member of each entry that another entry superseded in the previous attachment. A
superseded member therefore displaces no code, and an entry whose requires codes include
it attaches nothing. The candidate is dropped when the withheld members cycle without
converging, or when two or more members of a group remain and no single entry attached all
of them.
A copy the base car has is stripped first and re-added when the requirement holds, so a
code from the base car is absent whenever its parent is absent. Set strip_carried: false
where the code is orderable under a parent the requires list does not name, and stripping
would drop valid copies from the base car.
With package_contents_as_mandatory_adds true, the component also creates entries from the
packages of the target trim (see Package contents).
Package contents
A code is package-only content on a target trim when every one of its target-trim rows
has a parent_package_code. A row outside any package makes the code not package-only,
whatever the row's is_configurable value. The reason is that consolidated/features has
no standard-equipment column that every OEM shares. The 2027 Gladiator's radio codes, such
as RFV and RTF, have a standard-equipment row outside any package with a null
is_configurable, so a Gladiator that loses their package keeps them.
After the component removes and renames the base car's codes, the base car also loses
each package-only code whose packages it no longer has. When the base car loses a package
that is itself package-only content, it also loses that package's content. A code that a
mandatory_replaces or mandatory_expands rule transforms is kept, so that the rule
still applies.
A code with several target-trim rows resolves to the feature ID of one of them. A row inside a package counts only when the candidate has the package. Of the rows that count, a configurable row comes before a non-configurable one, and a row inside a package before a row outside any package. On the 2027 Grand Cherokee L Limited, the panoramic sunroof (GWJ) resolves to its standalone row, which has its own price, unless the candidate has the Limited Altitude package (ACA). It then resolves to its row inside ACA. A candidate is dropped when one of its configurable codes has no row that counts.
With package_contents_as_mandatory_adds: true, the component creates a mandatory_adds
entry for each package of the target trim. A package is any code that is the
parent_package_code of a target-trim row of consolidated/features, including an
interior that includes other codes. The entry's requires is the package, and its codes
are the codes of the rows whose parent_package_code is the package.
A package whose rows on the target trim are all non-configurable is fixed equipment, and
the component creates no entry for it. An entry's requires codes are checked only against
configurable codes, so such an entry would never attach the package's contents. The base
car's copies of those contents are kept.
The component does create an entry for a package without a row of its own. It also creates
one for a package whose own row does not count because no availability_sources group
lists it. The requirement of such an entry never holds. The entry still strips the
package's contents from the base car, so a candidate has them only when another rule adds
them.
These entries come before the configured mandatory_adds entries. A package's entry comes
before the entries of the packages among its contents, so the contents of an included
package are attached too. When a configured entry attaches a package, the candidate gets
none of that package's contents.
Each entry strips the base car's copies of its codes, except for these codes:
- A code that the target trim has as standard equipment, on a row whose
is_standardis true and whoseparent_package_codeis null. A car without the package can have such a code. On the 2027 Q5, the FW and UO interiors include the black cloth headliner (6NQ), which is also standard. A car with the Pearl Beige interior (II) therefore keeps 6NQ when the base car has it. - A code that a configured entry with
strip_carried: falselists. - A code that a
mandatory_replacesormandatory_expandsrule transforms.
The materialization fails when a package's contents include two members of one
mutually_exclusive_groups entry or of one only_one_of rule. Every candidate with the
package would violate that group or rule.
Compatibility rules
A candidate must also satisfy the compatibility rules in the OEM's source data. A rule
listed under option codes is checked only when one of those codes is among the
candidate's configurable codes. ordered_packages_filter changes which codes some of them
read (see Codes a package includes). A rule listed under no
code is checked for every candidate of its model. rules_from selects where the component
reads the rules:
rules_from | Rules read |
|---|---|
features (default) | The rules column of the target-year consolidated/features rows. Each rule is listed under the row's option code. excludes, requires_one_of and only_one_of map to exclusion, inclusion and only_one_of. The component raises on any other type. |
rules | consolidated/rules, for the target-year cohorts. For Incomplete base configs, the component also reads the base-year cohorts' rules. |
From consolidated/rules the component keeps a rule only when some
<group>_last_seen column (source groups)
equals the partition date and a validity slice covers that date. The component raises
when the target cohorts have rules but none is current. That happens when the rules
assets did not run for the partition date. For a partition before the first
consolidated/rules partition, the component reads no rules.
An exclusion or inclusion is listed under each of its non-negated IF codes, and
applies when its IF part holds as the rules contract
defines. One whose IF part has no non-negated code, because it is empty or holds only
negated codes, is listed under no code. An only_one_of is listed under each of its
THEN codes.
Whichever source is read, the checks are:
- An
exclusionfails when a code it names is present, as a configurable option or as fixed equipment. - An
only_one_offails when more than one of its codes is present, counted the same way. - An
inclusionfails when none of its THEN codes is present, as a configurable option or as fixed equipment. Aninclusionis not checked when none of its THEN codes is a configurable option of the target trim. A warning gives the number of distinct pairs of target trim and THEN codes among such inclusions, and lists the first ten.
Configurability is the target year's. A code the base car has as fixed equipment is a configurable code of the candidate when the target trim offers it as an option.
Before these checks, the component meets each inclusion a candidate fails by adding a
THEN code the target trim offers as an option. An inclusion that names several such
codes gives one candidate per code. A package-only code (see
Package contents) is added together with its package, one candidate
per package the trim offers as an option. A code added this way can bring in a further
inclusion, which is met the same way, and a mandatory_adds entry that requires it
attaches its content. A candidate still fails when no offered code meets an inclusion,
or when an exclusion or only_one_of fails.
Order type
Some rules name the order type of a car rather than an option. order_type sets the order
type of every candidate. Its code is the code of that order type. Its other_codes are
the codes of the order types a candidate is not ordered as. The rules are checked as
though each candidate had code as fixed equipment and none of other_codes, so a
candidate fails:
- an
inclusionwhose THEN codes are allother_codes; - an
exclusionthat namescode.
An inclusion whose THEN codes include an order type code is checked even when none of
its THEN codes is a configurable option of the target trim. A rule with code in its IF
part applies to every candidate that has the other codes of its IF part. A rule with one
of other_codes in its IF part applies to no candidate. code is not a feature, so it is
left out of feature_ids and of the signature that inventory_id is built from.
Stellantis candidates are stock orders, 4EX in the
code guides. The 2027 Grand Cherokee guide prints
N/A W/4EX under Night Vision (AAN), so no 2027 Grand Cherokee candidate has AAN.
Codes a package includes
Some OEMs order a package by its own code and leave the codes of its contents off the order.
ordered_packages_filter names these packages, the ordered packages. Each exclusion and
inclusion then reads only the codes of the candidate's order, which leave out the codes
its ordered packages include.
On the MY2027 SQ8, the rule "Power soft-closing doors (GZ2) excludes the Prestige trim
(WPT)" stops an order from listing GZ2 on a Prestige, which has the doors as standard.
Audi orders WPT without GZ2, so a Prestige candidate has GZ2 on the car but not on its
order, and is not dropped.
The filter is a Polars-SQL predicate over the target trim's rows of consolidated/features
that have no parent_package_code. A package whose row matches it is an ordered package. A
package's content codes are the codes of the target-trim rows whose parent_package_code
is the package. A content code of an ordered package of the candidate is not on the order,
so:
- no
exclusionorinclusionlisted under it applies, including one also listed under another code of its IF part; - it does not break an
exclusionthat names it; - it does not meet an
inclusionthat names it. The component meets such aninclusionwith a code the order can list, or drops the candidate.
The pruning of optional choices reads the order's codes in the same way. Every
only_one_of, mutually_exclusive_groups entry and must_have_one_of entry still reads
every code of the car, because they limit the codes a car has. Supersession of
mandatory_adds does not depend on the filter. When the filter is unset, the order has
every code of the candidate.
The predicate may reference {partition_date} and {partition_year}, which are filled in
at materialization time; write a literal brace twice. A predicate that does not parse fails
when the definitions load. One that names a column
consolidated/features lacks fails the materialization.
Audi's filter
Audi treats a package as ordered when its row meets all of these conditions:
- The order guides list it on the partition date (
order_guides_last_seen).is_orderablealone can be stale (see What the target year offers). - It is orderable (
is_orderable), which means the order guide lists it in the production window in effect, outside any package (seeenriched/order_guides_features). - It is configurable and is not a paint or an interior. The cure removes only the content codes of the build's options, and the paint, the interior and standard packages are not options (see Ordering decisions).
The filter therefore leaves out a package that only the catalog lists. Its content codes count as codes of the order, although the cure removes them.
Incomplete base configs
With rules_from: rules, an inclusion whose IF part is empty states that every car
of its model has one of the THEN codes. A base config that has none of the THEN codes
of such an inclusion of its base-year model is an incomplete inventory record. The
component generates no candidates from it, and a warning gives the number of base
configs dropped. The component reads only the base-year rules current on the partition
date. When none is current, it logs a warning and drops no base config.
Audi's enriched/family_rules writes such an inclusion over each model's interiors
(see enriched/family_rules).
For example, some MY2026 Q5 (GUBAAY) records from OneGraph inventory list no interior.
Mercedes rules files
Mercedes' consolidated/rules has no only_one_of over a model's upholsteries or paints,
and no inclusion with an empty IF part. When the component meets an inclusion that
names upholsteries the car does not have, the candidate therefore keeps its upholstery and
gets a second one. On the 2027 EQS sedans, the AMG Line (9:DG3) requires one of a list
of upholsteries that leaves out Neva Grey / Sable Brown Leather (4:209).
Each Mercedes rules file lists every upholstery that the target year offers on its models,
in a mutually_exclusive_groups entry and in a must_have_one_of entry. A candidate with
more than one upholstery, or with none, is dropped. A file also lists its models' paints
in a mutually_exclusive_groups entry in either of these cases:
- An
inclusionof the models names paints, as the G-Class Night Package Magno Plus (9:DF4) does. - Base-year records list a two-tone paint as two paint codes. In the target year, each
two-tone paint has one code, such as Obsidian Black / Cirrus Silver Metallic (
2:C35) on the Maybach GLS.
The files target only target-year models that have fewer candidates from real inventory
than their baseline in models_meet_candidate_baseline. The files use the paints on at
least 2% of a model's inventory in two ways:
- When the model's base-year inventory is too small to reach the baseline, an
optional_replacesentry also offers each of these paints on a car whose paint is one of them. - A
mandatory_expandsentry replaces a paint that the target year does not offer with each of these paints.
MSRP
With msrp: true, a candidate's msrp is the sum of:
- the target model's
msrp; - the target model's
destination_charge; - the
priceof each of the candidate's option codes.
A code's price is the price of its feature row outside any package. A feature row inside a package repeats the package's price, so the component ignores that row's price. The package code's own row has the price instead. The price of a code that is content of a package on the candidate does not count, even if the code has a price outside the package.
A candidate's msrp is null in two cases:
- its target model has a null
msrpor a nulldestination_charge; - the component's
msrpsetting is false (the default).
A warning names each target model whose msrp or destination_charge is null. The
materialization fails when a code has two different prices outside a package on one model.
The setting is a boolean because the component reads only columns that every OEM's
consolidated tier has.
The asset's materialization metadata records the number of candidates with a non-null
msrp, and the 10th, 50th and 90th percentile of their msrp.
featurization/synthetic_measures does not read a candidate's msrp.
VehicleFeaturizationComponent computes a candidate's all_features_price from its features, as
it does an inventory vehicle's (see All features price).
Determinism and the cap
inventory_id and the max_per_base cap both derive from _sig, the per-candidate
signature that embeds the cohort's model_id. The signature holds only the candidate's
configurable codes, including the package contents attached to them. Two candidates with
the same configurable codes are one candidate, even where they inherit different fixed
equipment from their base cars. The component keeps the first of these candidates that
resolves to feature_ids. The fixed equipment of the kept candidate, and with it the
feature_ids and msrp of its inventory_id, therefore depends on the order in which
the component reads the base configs. When a base config expands to more than
max_per_base candidates, the cap keeps a subset ordered by a stable hash of _sig,
which is arbitrary but reproducible for a given model_id. The inputs and rules
determine the configurable codes of every candidate in a cohort.
Checks
The component asserts nothing about its own output. What a model can be recommended from
is the union of real inventory and these candidates, so the checks run downstream on
output/candidates_output (see Candidate Coverage Checks).