Featurization Schemes
The classification schemes are JSON files under
packages/ai_dagster/src/ai_dagster/components/featurization/schemes/. The rules for criteria cover
the binary schemes in the features/ and models/ directories. The steps under
Changing a scheme apply to every scheme.
TypeSafe answers a feature scheme's question from the name, description, category and
feature_type of one OEM option. It answers a model scheme's question from the model fields
that ModelFeaturizationComponent lists in description_fields.
When a scheme states criteria
A binary scheme's true criterion is its true_is text. Its false criterion is its false_is text.
Its examples are its true_examples.
TypeSafe often answers correctly from the question alone. Add a criterion or an example only where the question leaves the boundary between yes and no unclear.
- A true criterion that restates the question, or lists synonyms of its words, adds nothing.
- An example that repeats the question's words stays only where it is the exact name of an OEM option, such as "Remote Tailgate Release".
- A synonym in the examples stays where it matches how an OEM names the option, such as "Side Steps" for running boards.
- A false criterion that only negates the question adds nothing.
False criteria
A false criterion names the options that resemble the yes options but have the answer no. It ends with a clause that negates the question:
A cargo liner, or not a cargo tray.
The negating clause makes no the answer for every option the question does not cover. In some schemes, the clause also states what the question covers. Removing the clause from such a scheme removes that statement too.
The clause comes last. In "Not a cargo tray, or a cargo liner", TypeSafe can read "not" as applying to the cargo liner too.
test_binary_false_criterion_negates_question requires every binary false criterion to contain one
of these clauses, except in the schemes that the test lists:
- ", or not"
- ", or unrelated to"
- ", or neither"
The attribute hierarchy
The attribute hierarchy, under schemes/attributes/, classifies the attributes of the feature
hierarchy rather than OEM options. Its root holds the facet scheme, which asks whether an
attribute is mainly technology or mainly design.
attribute_definitions gives the state that defines each attribute the feature hierarchy can
assign:
- a binary attribute's definition is its scheme's question, focus, true answer and true examples;
- a multiclass value's definition is its scheme's question and the value's name, description and examples.
classify_attributes classifies each definition through the attribute hierarchy. A facet is
therefore a property of the attribute, not of a feature: adaptive cruise control gets the same
facet on every feature that has it, including a package. No asset classifies attributes. The
attribute_facets eval checks the facet scheme:
cd packages/ai_dagster
PYTHONPATH=. uv run python -m evals.run attribute_facets --repeat 3
Changing a scheme
Every edit to a scheme, except to exclude_from_set, changes its
scheme_version. The next featurization run therefore asks TypeSafe that scheme's question again
for every OEM, instead of reading the cached answer.
A change of one word can move an option's probability across MIN_TRUE in
ai_featurize.classification.classify. That threshold decides each binary answer and which child
categories the hierarchy traversal descends into. Check every change with the feature_hierarchy
eval in packages/ai_dagster/evals/:
- Run the eval three times on the branch and three times on
main:The eval needscd packages/ai_dagster
PYTHONPATH=. uv run python -m evals.run feature_hierarchy --repeat 3TYPESAFE_API_KEYin the environment. - Compare the assertion rate and the result of each case.
- Before you call a case a regression, run it five to ten times on the branch and on
main. Pass--cases <file>, where<file>holds only that case.
Model-year consistency
The model_year_consistency eval checks that a scheme gives one option the same answer in two model
years. An OEM often renames an option or adds an item to a package between model years. An answer
that changes with the wording splits the option's demand between two attributes. A case holds the
option's previous and current rows. It passes when the classification of each row makes every
assignment in assignments, and when the two classifications make the same assignments to each
scheme in schemes.
cd packages/ai_dagster
PYTHONPATH=. uv run python -m evals.run model_year_consistency --repeat 5
Add a case when a model year changes an option's answer, and change the scheme until the case passes
on every repeat. A case lists in schemes only the schemes that should agree. For example, Audi's
MY2027 Black optic plus package adds S line, so sport_package may differ between its two years,
and blackout_appearance_package may not.
Model schemes
The model_hierarchy eval checks the schemes under models/ the way feature_hierarchy checks the
feature schemes. A case's fields are the model fields the OEM's ModelFeaturizationComponent
lists in description_fields, so a case sends what that OEM's asset sends.
cd packages/ai_dagster
PYTHONPATH=. uv run python -m evals.run model_hierarchy --repeat 3
Spelling consistency
The spelling_consistency eval checks that a scheme gives one option the same answer under two
names that the sources give it, such as 18" ALUMINUM SPARE WHEEL (W1F) and 18" Aluminum Spare Wheel. Its cases have the same fields as model_year_consistency cases, with the two names in
previous and current. For many options whose answer differs between the two names, both
probabilities are close to MIN_TRUE. A small change to a scheme's wording can then put both
probabilities on the same side of MIN_TRUE.
The cases come from production rows of one option code that got different attributes. cases.yaml
holds the cases a scheme change is tuned against, and holdout.yaml holds the rest. A change that
improves the results on cases.yaml and not on holdout.yaml is overfitted to the wording of the
cases in cases.yaml. A scheme change written with a holdout case in view moves that case to
cases.yaml, so the holdout keeps measuring only cases no change was tuned against.
cd packages/ai_dagster
PYTHONPATH=. uv run python -m evals.run spelling_consistency --repeat 3
PYTHONPATH=. uv run python -m evals.run spelling_consistency --repeat 3 \
--cases evals/spelling_consistency/holdout.yaml