Skip to main content

Synthetic Candidates

SyntheticCandidatesComponent (in packages/ai_dagster) builds the <oem>/<market>/synthetic_candidates asset: a mapping of a prior model year's real inventory configurations into the next model year's configuration space. A cohort that declares no rules is this same mapping with an empty rule set — codes with no target-year analog still retire — not a separate procedure. Each selected prior-year vehicle is reduced to its option-code set, rewritten by YAML-declared rules (renames, forced/optional swaps, adds, mutually exclusive constraints), and expanded combinatorially. The output is keyed by model_id / base_model_id and flagged is_synthetic=True. It carries no OEM descriptive columns — a consumer that needs model_code/trim joins consolidated/models by model_id.

Inputs are consolidated/{inventory, features, models}. A cohort — the unit a rule scopes to and a candidate belongs to — is a year-independent model identity (base_model_id), whose target-year instance is one model_id.

Layered architecture​

The component splits into three layers. OEM-specific column names (model_code, trim_identifier, drive, …) appear only at rule authoring. From resolution onward the pipeline — including its output — speaks a single opaque identifier, model_id.

  1. Authoring — OEM-dependent. Rules are RuleScopes whose model_filters map a models column name to a case-insensitive regex ({model_code: "^JG"}, {drive: "^AWD$"}). This is the human-facing form, expressive in OEM terms — "all trims of this model", "all all-wheel-drive models".
  2. Resolution — the boundary. _resolve_rule_model_ids evaluates every model_filters against the target-year consolidated/models rows once, producing applies: {id(rule) → frozenset[model_id]}. _resolve_target_cohorts selects which cohorts to generate (base_model_id → target model_id). _validate_model_filters rejects a filter key that names no real models column, so a typo fails loudly instead of silently matching nothing. The resolution is data-driven: the same rule covers a different set of model_ids as models come and go between partitions.
  3. Core generation + output — OEM-agnostic. _generate operates purely on model_id. Rule scoping is set membership (tgt_model_id in applies[id(rule)]), never regex. Keying (fid_by_key, signatures, Candidate) is by model_id. It emits the final table — inventory_id, model_id, dealer_id, feature_ids, model_year, base_model_id, msrp_bucket, is_synthetic — with no OEM descriptive columns. A consumer needing model_code/trim joins consolidated/models by model_id itself (its canonical home). The component does not denormalize them onto every synthetic row. The pure set-algebra _generate drives lives in ai_dagster.synthetic.generate (no Dagster, no polars).

Why model_id, not the OEM fields​

model_id is year-specific and opaque, which is exactly what the two jobs need:

  • Bridging model years requires a year-independent key — base_model_id — because a cohort has a different model_id in each year. The base→target mapping joins on base_model_id, never model_id.
  • Keying and matching inside generation want an opaque scalar. Reaching into the identity fields would re-introduce OEM-specific assumptions (e.g. "the first field is the primary key") into code that must serve every OEM.

Regex over model_code/trim stays useful for authoring rules across many models — so it lives in model_filters and is resolved to model_id sets at the boundary, never evaluated in the generation loop.

Determinism and the cap​

inventory_id and the max_per_base cap both derive from _sig, the per-candidate signature that embeds the cohort's model_id. When a base config expands to more than max_per_base candidates, the cap keeps a subset ordered by a stable hash of _sig — arbitrary but reproducible for a given cohort key. Because the key is model_id, both the retained subset and the inventory_id values are a function of model_id. A cohort that stays under the cap is unaffected. The candidate universe per cohort is fully determined by the inputs and rules — only which candidates survive an over-cap config depends on the signature hash.

Checks​

The component asserts nothing about its own output. What a model ends up able to be recommended from is the union of real inventory and these candidates, so it is asserted downstream on output/candidates_output — see Candidate Coverage Checks.