Skip to main content

Synthetic Candidates

SyntheticCandidatesComponent (in packages/ai_dagster) builds the <oem>/<market>/synthetic_candidates asset. The asset's description in Dagster states what the table holds.

Inputs are consolidated/{inventory, features, models}, and consolidated/rules when rules_from is rules.

A cohort is a year-independent model identity (base_model_id). Its target-year instance is one model_id. YAML rules are scoped to cohorts, and each candidate belongs to one cohort.

For a cohort that no YAML rule is scoped to, the component generates candidates the same way as for any other cohort, with an empty rule set. The codes of the cohort's base vehicles are still kept or retired as What the target year offers describes.

Layered architecture​

The component splits into three layers. OEM-specific column names (model_code, trim_identifier, drive, …) appear only at rule authoring. From resolution onward the pipeline, including its output, uses a single opaque identifier, model_id.

  1. Authoring, OEM-dependent. Every rule has a RuleScope. Its model_filters maps a consolidated/models column name to a case-insensitive regex ({model_code: "^JG"}, {drive: "^AWD$"}), and every entry must match for the rule to apply. Anchor a pattern (^X$) for an exact match; an empty map matches every cohort.
  2. Resolution, the boundary. _resolve_rule_model_ids evaluates every model_filters against the target-year consolidated/models rows once, producing applies: {id(rule) → frozenset[model_id]}, so the same rule covers different models as models come and go between partitions. _resolve_target_cohorts maps each base-year model_id to the target-year model_ids its inventory generates candidates for (see Candidates for a new trim). _validate_model_filters rejects a filter key that names no models column.
  3. Core generation and output, OEM-agnostic. _generate scopes rules by set membership (tgt_model_id in applies[id(rule)]) and keys candidates by model_id. The output has inventory_id, model_id, dealer_id, feature_ids, model_year, base_model_id, msrp and is_synthetic, and no OEM descriptive columns. A consumer needing model_code or trim joins consolidated/models by model_id. The set algebra lives in ai_dagster.synthetic.generate, which imports neither Dagster nor polars.

Generation keys on the opaque model_id, so it makes no assumption about an OEM's identity fields. It bridges model years on base_model_id, because a cohort has a different model_id in each year.

Candidates for a new trim​

Each target_combinations entry is a TargetCombination, whose model_filters selects target-year cohorts. By default, a target's candidates come from the inventory of the base-year cohort that shares its base_model_id. A renamed model code bridges model years this way once base_model_id_aliases maps it onto its predecessor.

A trim that is new in the target year has no base-year cohort. Its base_filters selects the base-year cohorts whose inventory generates its candidates instead. The filters match as model_filters does, applied to base-year consolidated/models rows:

target_combinations:
- model_filters: {model_code: ^KMJH74$, trim_identifier: ^H$}
base_filters: {model_code: ^KMJP74$, trim_identifier: ^Q$}

Unlike an alias, base_filters leaves the base-year cohort's identity unchanged:

  • The base-year cohort's inventory still generates candidates for its own target.
  • The new trim's candidates have the new trim's model_id and base_model_id.
  • Rules scope by target model_id, so rules that match only the new trim turn the base-year cohort's codes into the new trim's codes.

What the target year offers​

A base vehicle's code is kept when the target-year cohort has a consolidated/features row for it, and retired otherwise. The table keeps a row after its sources stop listing it. With availability_sources set, a configurable target-year row counts only when one of those source groups lists it on the partition date. Base-year rows always count, so a base vehicle's feature IDs still resolve. Leave out a source group that lists options a trim cannot be ordered with, such as Stellantis' equipment_categories.

A non-configurable target-year row that those source groups do not list still counts, unless they list its code on the partition date for another model of the same model_group_id and not for this one. For example, Stellantis' equipment_categories lists every Wrangler Sport package under each Sport trim, but the code guides list 23S for Sport S only, so 23S does not count on Sport B.

With package_content_sources set, a target-year row inside a package does not count when it is replaced. A row is replaced when all of these hold:

  • No source group of consolidated/features lists the row on the partition date.
  • The groups that package_content_sources names did not list the row on the last date they listed its model_id.
  • On that date, they listed a row of the same package on the same model_id that they first listed after they first listed the replaced row.

While those groups list a model, the last date they listed its model_id is the partition date. When a source stops listing a whole model, the rows of its last listing still decide which of the model's rows are replaced.

A row that those groups stopped listing still counts when they list no such newer row, because a source can stop listing a package's content while the package still includes it. For example, the older Audi catalog year of the 2027 A6 Sportback e-tron lists Adaptive cruise assist (GJ7) in the Prestige (WPT) and Premium Plus (WPS) packages, and the newer one lists Adaptive cruise assist plus (GJ8) in WPT instead. With package_content_sources: [catalog], the GJ7 row in WPT is replaced, and the GJ7 row in WPS still counts.

With package_content_sources set, the asset's materialization metadata records the number of replaced rows as replaced_package_content_rows. The run log lists each replaced row by model_id, package and code.

optional_replaces, optional_adds and optional_add_groups skip a code that the target-year cohort has no row for. An optional_add_groups entry with such a code is skipped as a whole. mandatory_adds, mandatory_expands and mandatory_replaces are not filtered. A candidate to which one of them adds such a code is dropped.

Rule types​

Every rule applies to the cohorts its model_filters selects (see Layered architecture).

Mandatory rules change every candidate of the matching cohort. An optional rule offers a choice of operations, and one of them leaves the candidate unchanged. Each optional rule therefore multiplies the number of candidates.

RuleEffect
mandatory_replaces1:1 rename of a code that was relabeled in the target year.
mandatory_expandsReplace from_codes with exactly one of to_codes. Unlike an optional rule, it cannot leave the candidate unchanged. Every candidate that has one of from_codes takes one of to_codes.
mandatory_addscodes are present if and only if every requires code is present. Use it for package content valid only alongside its parent. See How mandatory_adds attaches codes.
mandatory_removesRemove codes that the target year still offers. A code the target year does not offer retires without a rule, and a code that optional_adds or optional_add_groups offers is removed from the base vehicle before generation. With requires set, codes are removed only when every requires code is among the base vehicle's codes, configurable or not. Use it for a code that is valid only while another code is absent, such as a package that cannot be combined with a specific paint.
standard_addsUnconditionally attach target-year content that is standard and non-configurable. Use it for content with no configurable parent, which neither the 1:1 rename of mandatory_replaces nor the requires list of mandatory_adds can attach. Codes that do not resolve in the target catalog are skipped. A code that mandatory_adds attaches replaces a standard_adds code of the same exclusive group. A configurable base code that a package upgrade replaces therefore also belongs here.
optional_replacesReplace the candidate's member of an exclusive group with one of to_codes, or keep the member it has.
optional_addsOptionally add one of add_codes, which are mutually exclusive alternatives. Use a single-element list for an independent toggle.
optional_add_groupsOptionally add every code in codes together, or none of them. A subset is never added, so codes that only make sense together belong here rather than in optional_adds.
mutually_exclusive_groupsCodes of which a candidate may have at most one (see Exclusive groups).
must_have_one_ofRequire at least one of codes on every candidate of the matching cohort. A config carrying none of them is dropped.

An entry of to_codes, in mandatory_expands and optional_replaces, is one code or a list of codes that the replacement adds together. A list entry maps a trim package whose content the target year splits across several packages onto combinations of those packages. In the following example, a Q5 with the Prestige package (WPT) gets either the Executive (WPS) and Driver assistance (WPR) packages, or those two and the Vorsprung package (PPZ):

mandatory_expands:
- model_filters: {model_code: ^GUBAAY$}
group: WPT
from_codes: [WPT]
to_codes: [[WPS, WPR], [WPS, PPZ, WPR]]

Exclusive groups​

An exclusive group is a set of codes of which a candidate may have at most one. Its members are:

  • the member_codes of the mutually_exclusive_groups entries whose group is the group's name;
  • with rules_from: rules, the codes of the cohort's only_one_of rule of that name. A rule is named by its code, or by its rule_id when it has no code. An entry whose group names such a rule adds its member_codes to the rule's group.

A candidate is checked against an only_one_of rule as Compatibility rules states, and against the entries of mutually_exclusive_groups.

Each of the following rules names a group:

  • mandatory_expands
  • optional_replaces
  • optional_adds
  • optional_add_groups

A code that such a rule adds replaces every member of the group that the candidate has, as a configurable option or as fixed equipment.

How mandatory_adds attaches codes​

requires is checked on each candidate after the optional rules, mandatory_expands and the earlier mandatory_adds entries have changed it. The content is therefore attached when optional_adds, optional_replaces or mandatory_expands adds its package, and is absent when one of them removes the package.

An attached code replaces the other members of each exclusive group it belongs to, including fixed equipment. For example, an upgrade that a package includes replaces the base code.

Where entries attach different members of one exclusive group, the candidate is dropped, unless supersede_mandatory_adds is true. When it is true, an entry that another of them supersedes does not attach its member. An entry supersedes another when its requires codes imply every requires code of the other entry, and the other entry's requires codes do not imply all of its own. A code implies every code it requires, directly or through other codes. A code requires the THEN code of an inclusion whose IF part is that code alone and whose THEN part is one code. Only an inclusion that the component checks counts (see Compatibility rules). ordered_packages_filter does not change which inclusions count.

On the 2027 Q5, the Vorsprung package (PPZ) requires the Executive (WPS) and Driver assistance (WPR) packages, so its OLED taillights (8VY) replace the Executive package's animated taillights (8VP).

After the first attachment, the component attaches the entries again, and withholds the member of each entry that another entry superseded in the previous attachment. A superseded member therefore displaces no code, and an entry whose requires codes include it attaches nothing. The candidate is dropped when the withheld members cycle without converging, or when two or more members of a group remain and no single entry attached all of them.

A copy the base car has is stripped first and re-added when the requirement holds, so a code from the base car is absent whenever its parent is absent. Set strip_carried: false where the code is orderable under a parent the requires list does not name, and stripping would drop valid copies from the base car.

With package_contents_as_mandatory_adds true, the component also creates entries from the packages of the target trim (see Package contents).

Package contents​

A code is package-only content on a target trim when every one of its target-trim rows has a parent_package_code. A row outside any package makes the code not package-only, whatever the row's is_configurable value. The reason is that consolidated/features has no standard-equipment column that every OEM shares. The 2027 Gladiator's radio codes, such as RFV and RTF, have a standard-equipment row outside any package with a null is_configurable, so a Gladiator that loses their package keeps them.

After the component removes and renames the base car's codes, the base car also loses each package-only code whose packages it no longer has. When the base car loses a package that is itself package-only content, it also loses that package's content. A code that a mandatory_replaces or mandatory_expands rule transforms is kept, so that the rule still applies.

A code with several target-trim rows resolves to the feature ID of one of them. A row inside a package counts only when the candidate has the package. Of the rows that count, a configurable row comes before a non-configurable one, and a row inside a package before a row outside any package. On the 2027 Grand Cherokee L Limited, the panoramic sunroof (GWJ) resolves to its standalone row, which has its own price, unless the candidate has the Limited Altitude package (ACA). It then resolves to its row inside ACA. A candidate is dropped when one of its configurable codes has no row that counts.

With package_contents_as_mandatory_adds: true, the component creates a mandatory_adds entry for each package of the target trim. A package is any code that is the parent_package_code of a target-trim row of consolidated/features, including an interior that includes other codes. The entry's requires is the package, and its codes are the codes of the rows whose parent_package_code is the package.

A package whose rows on the target trim are all non-configurable is fixed equipment, and the component creates no entry for it. An entry's requires codes are checked only against configurable codes, so such an entry would never attach the package's contents. The base car's copies of those contents are kept.

The component does create an entry for a package without a row of its own. It also creates one for a package whose own row does not count because no availability_sources group lists it. The requirement of such an entry never holds. The entry still strips the package's contents from the base car, so a candidate has them only when another rule adds them.

These entries come before the configured mandatory_adds entries. A package's entry comes before the entries of the packages among its contents, so the contents of an included package are attached too. When a configured entry attaches a package, the candidate gets none of that package's contents.

Each entry strips the base car's copies of its codes, except for these codes:

  • A code that the target trim has as standard equipment, on a row whose is_standard is true and whose parent_package_code is null. A car without the package can have such a code. On the 2027 Q5, the FW and UO interiors include the black cloth headliner (6NQ), which is also standard. A car with the Pearl Beige interior (II) therefore keeps 6NQ when the base car has it.
  • A code that a configured entry with strip_carried: false lists.
  • A code that a mandatory_replaces or mandatory_expands rule transforms.

The materialization fails when a package's contents include two members of one mutually_exclusive_groups entry or of one only_one_of rule. Every candidate with the package would violate that group or rule.

Compatibility rules​

A candidate must also satisfy the compatibility rules in the OEM's source data. A rule listed under option codes is checked only when one of those codes is among the candidate's configurable codes. ordered_packages_filter changes which codes some of them read (see Codes a package includes). A rule listed under no code is checked for every candidate of its model. rules_from selects where the component reads the rules:

rules_fromRules read
features (default)The rules column of the target-year consolidated/features rows. Each rule is listed under the row's option code. excludes, requires_one_of and only_one_of map to exclusion, inclusion and only_one_of. The component raises on any other type.
rulesconsolidated/rules, for the target-year cohorts. For Incomplete base configs, the component also reads the base-year cohorts' rules.

From consolidated/rules the component keeps a rule only when some <group>_last_seen column (source groups) equals the partition date and a validity slice covers that date. The component raises when the target cohorts have rules but none is current. That happens when the rules assets did not run for the partition date. For a partition before the first consolidated/rules partition, the component reads no rules.

An exclusion or inclusion is listed under each of its non-negated IF codes, and applies when its IF part holds as the rules contract defines. One whose IF part has no non-negated code, because it is empty or holds only negated codes, is listed under no code. An only_one_of is listed under each of its THEN codes.

Whichever source is read, the checks are:

  • An exclusion fails when a code it names is present, as a configurable option or as fixed equipment.
  • An only_one_of fails when more than one of its codes is present, counted the same way.
  • An inclusion fails when none of its THEN codes is present, as a configurable option or as fixed equipment. An inclusion is not checked when none of its THEN codes is a configurable option of the target trim. A warning gives the number of distinct pairs of target trim and THEN codes among such inclusions, and lists the first ten.

Configurability is the target year's. A code the base car has as fixed equipment is a configurable code of the candidate when the target trim offers it as an option.

Before these checks, the component meets each inclusion a candidate fails by adding a THEN code the target trim offers as an option. An inclusion that names several such codes gives one candidate per code. A package-only code (see Package contents) is added together with its package, one candidate per package the trim offers as an option. A code added this way can bring in a further inclusion, which is met the same way, and a mandatory_adds entry that requires it attaches its content. A candidate still fails when no offered code meets an inclusion, or when an exclusion or only_one_of fails.

Order type​

Some rules name the order type of a car rather than an option. order_type sets the order type of every candidate. Its code is the code of that order type. Its other_codes are the codes of the order types a candidate is not ordered as. The rules are checked as though each candidate had code as fixed equipment and none of other_codes, so a candidate fails:

  • an inclusion whose THEN codes are all other_codes;
  • an exclusion that names code.

An inclusion whose THEN codes include an order type code is checked even when none of its THEN codes is a configurable option of the target trim. A rule with code in its IF part applies to every candidate that has the other codes of its IF part. A rule with one of other_codes in its IF part applies to no candidate. code is not a feature, so it is left out of feature_ids and of the signature that inventory_id is built from.

Stellantis candidates are stock orders, 4EX in the code guides. The 2027 Grand Cherokee guide prints N/A W/4EX under Night Vision (AAN), so no 2027 Grand Cherokee candidate has AAN.

Codes a package includes​

Some OEMs order a package by its own code and leave the codes of its contents off the order. ordered_packages_filter names these packages, the ordered packages. Each exclusion and inclusion then reads only the codes of the candidate's order, which leave out the codes its ordered packages include.

On the MY2027 SQ8, the rule "Power soft-closing doors (GZ2) excludes the Prestige trim (WPT)" stops an order from listing GZ2 on a Prestige, which has the doors as standard. Audi orders WPT without GZ2, so a Prestige candidate has GZ2 on the car but not on its order, and is not dropped.

The filter is a Polars-SQL predicate over the target trim's rows of consolidated/features that have no parent_package_code. A package whose row matches it is an ordered package. A package's content codes are the codes of the target-trim rows whose parent_package_code is the package. A content code of an ordered package of the candidate is not on the order, so:

  • no exclusion or inclusion listed under it applies, including one also listed under another code of its IF part;
  • it does not break an exclusion that names it;
  • it does not meet an inclusion that names it. The component meets such an inclusion with a code the order can list, or drops the candidate.

The pruning of optional choices reads the order's codes in the same way. Every only_one_of, mutually_exclusive_groups entry and must_have_one_of entry still reads every code of the car, because they limit the codes a car has. Supersession of mandatory_adds does not depend on the filter. When the filter is unset, the order has every code of the candidate.

The predicate may reference {partition_date} and {partition_year}, which are filled in at materialization time; write a literal brace twice. A predicate that does not parse fails when the definitions load. One that names a column consolidated/features lacks fails the materialization.

Audi's filter​

Audi treats a package as ordered when its row meets all of these conditions:

  • The order guides list it on the partition date (order_guides_last_seen). is_orderable alone can be stale (see What the target year offers).
  • It is orderable (is_orderable), which means the order guide lists it in the production window in effect, outside any package (see enriched/order_guides_features).
  • It is configurable and is not a paint or an interior. The cure removes only the content codes of the build's options, and the paint, the interior and standard packages are not options (see Ordering decisions).

The filter therefore leaves out a package that only the catalog lists. Its content codes count as codes of the order, although the cure removes them.

Incomplete base configs​

With rules_from: rules, an inclusion whose IF part is empty states that every car of its model has one of the THEN codes. A base config that has none of the THEN codes of such an inclusion of its base-year model is an incomplete inventory record. The component generates no candidates from it, and a warning gives the number of base configs dropped. The component reads only the base-year rules current on the partition date. When none is current, it logs a warning and drops no base config.

Audi's enriched/family_rules writes such an inclusion over each model's interiors (see enriched/family_rules). For example, some MY2026 Q5 (GUBAAY) records from OneGraph inventory list no interior.

Mercedes rules files​

Mercedes' consolidated/rules has no only_one_of over a model's upholsteries or paints, and no inclusion with an empty IF part. When the component meets an inclusion that names upholsteries the car does not have, the candidate therefore keeps its upholstery and gets a second one. On the 2027 EQS sedans, the AMG Line (9:DG3) requires one of a list of upholsteries that leaves out Neva Grey / Sable Brown Leather (4:209).

Each Mercedes rules file lists every upholstery that the target year offers on its models, in a mutually_exclusive_groups entry and in a must_have_one_of entry. A candidate with more than one upholstery, or with none, is dropped. A file also lists its models' paints in a mutually_exclusive_groups entry in either of these cases:

  • An inclusion of the models names paints, as the G-Class Night Package Magno Plus (9:DF4) does.
  • Base-year records list a two-tone paint as two paint codes. In the target year, each two-tone paint has one code, such as Obsidian Black / Cirrus Silver Metallic (2:C35) on the Maybach GLS.

The files target only target-year models that have fewer candidates from real inventory than their baseline in models_meet_candidate_baseline. The files use the paints on at least 2% of a model's inventory in two ways:

  • When the model's base-year inventory is too small to reach the baseline, an optional_replaces entry also offers each of these paints on a car whose paint is one of them.
  • A mandatory_expands entry replaces a paint that the target year does not offer with each of these paints.

MSRP​

With msrp: true, a candidate's msrp is the sum of:

  • the target model's msrp;
  • the target model's destination_charge;
  • the price of each of the candidate's option codes.

A code's price is the price of its feature row outside any package. A feature row inside a package repeats the package's price, so the component ignores that row's price. The package code's own row has the price instead. The price of a code that is content of a package on the candidate does not count, even if the code has a price outside the package.

A candidate's msrp is null in two cases:

  • its target model has a null msrp or a null destination_charge;
  • the component's msrp setting is false (the default).

A warning names each target model whose msrp or destination_charge is null. The materialization fails when a code has two different prices outside a package on one model. The setting is a boolean because the component reads only columns that every OEM's consolidated tier has.

The asset's materialization metadata records the number of candidates with a non-null msrp, and the 10th, 50th and 90th percentile of their msrp. featurization/synthetic_measures does not read a candidate's msrp. VehicleFeaturizationComponent computes a candidate's all_features_price from its features, as it does an inventory vehicle's (see All features price).

Determinism and the cap​

inventory_id and the max_per_base cap both derive from _sig, the per-candidate signature that embeds the cohort's model_id. The signature holds only the candidate's configurable codes, including the package contents attached to them. Two candidates with the same configurable codes are one candidate, even where they inherit different fixed equipment from their base cars. The component keeps the first of these candidates that resolves to feature_ids. The fixed equipment of the kept candidate, and with it the feature_ids and msrp of its inventory_id, therefore depends on the order in which the component reads the base configs. When a base config expands to more than max_per_base candidates, the cap keeps a subset ordered by a stable hash of _sig, which is arbitrary but reproducible for a given model_id. The inputs and rules determine the configurable codes of every candidate in a cohort.

Checks​

The component asserts nothing about its own output. What a model can be recommended from is the union of real inventory and these candidates, so the checks run downstream on output/candidates_output (see Candidate Coverage Checks).