Code guide parser
Deterministic parser for Stellantis Customer Preferred Code Guide PDFs. No LLM in the loop — tables are recovered from the PDF's own ruling lines and font weights, so the same PDF always yields the same records.
The parser reads what a guide says; ai_stellantis.sources.code_guides turns that into
these records:
CodeGuideModelRecord, one per trim variant;CodeGuideFeatureRecord, one per option × model × trim × parent package;CodeGuideRuleRecord, one per distinct rule per trim.
from ai_dealer_connect.code_guides.parser import parse
guide = parse("packages/ai_dealer_connect/tests/data/code_guides/parser/fixtures/jeep_grand_cherokee_2row_2027_2026-05-05_2026-06-23/code-guides_2026-06-24_277944.pdf")
guide.trim_specs # what each trim carries
guide.page_grids(3) # the STEP #3 / STEP #4 option tables on page 3
Or the whole document as one envelope, and the records that follow from it:
from ai_stellantis.sources.code_guides import (
derive_features,
derive_models,
derive_rules,
)
from ai_dealer_connect.code_guides.parser import parse_document
doc = parse_document("…/code-guides_2026-08-19_279402.pdf")
doc.identity # brand, country, model_year, model_family, issued/reissued dates
doc.section_grids # every STEP #3 / #4 table, with its price bands
doc.standard_equipment # and GROUP DETAIL, PACKAGE CONTENT, the powertrain matrix…
doc.annotations # every distinct annotation block, paired with its rules
doc.lineage_key # 'ram|United States|2027|2500 HEAVY DUTY CREW CAB|2026-05-22'
derive_models(doc) # list[CodeGuideModelRecord]
features = derive_features(doc) # list[CodeGuideFeatureRecord]
derive_rules(features) # list[CodeGuideRuleRecord]
The envelope holds what the document says; the records follow from it. Nothing derived
lives on CodeGuideDocument — a field cannot be both the derivation's output and its
input — so derive_models and derive_features take an envelope and open nothing. They
run on one rebuilt from JSON, which is how the rulings in
features-ground-truth.md can be exercised without a PDF at all. derive_rules takes
the features they produce.
Records are reached one way only. parse() returns a guide with no models or
features on it, and the parser package exports neither the record classes nor the
derivations — a caller wanting records goes to extract and hands it an envelope. The
lazy views stay lazy so page_grids(3) need not parse 175 pages, while parse_document
parses everything and is the serialization boundary.
The Dagster assets
StellantisCodeGuidesRawComponent owns the stored PDFs and the
dynamic partitions, and StellantisCodeGuidesComponent owns the transformed assets
derived from them. StellantisCodeGuideRulesComponent restricts the rules to each
trim's codes.
| asset | partition | rows |
|---|---|---|
raw/code_guides | one per guide | one — the PDF's own bytes |
transformed/code_guides_models_by_edition | one per guide | that guide's trim variants |
transformed/code_guides_features_by_edition | one per guide | that guide's (option, model code, trim, parent package) rows |
transformed/code_guides_rules_by_edition | one per guide | that guide's rules |
transformed/code_guides_models | daily | the current edition's trim variants |
transformed/code_guides_features | daily | the current edition's options |
transformed/code_guides_rules | daily | the current edition's rules |
enriched/code_guides_rules | daily | those rules, restricted to the codes on each trim |
A guide is a document, so the partition is the blob id its filename carries —
code-guides/2026-07-25/278922.pdf is partition 278922. The raw and transformed
components derive the
partitions-definition name the same way, so they address one set.
The raw asset
raw/code_guides reads one PDF from GCS and stores its bytes:
| column | |
|---|---|
partition | the blob id, and the Iceberg partition column |
content | the PDF, as Binary |
content_type | application/pdf |
fetched_at | when it was read |
A stellantis_us_code_guides sensor reads the code_guides registry every five
minutes and adds a partition for every full-pricing blob id it has not seen. The registry
is the record of what the collection pipeline stored, so the sensor never lists the
bucket; it requests no runs of its own either — the asset's on_missing condition fills
whichever partitions have never been materialized.
The asset resolves the bytes when it runs, matching the blob id server-side. One id can appear under several date directories: the collection pipeline re-files a guide it collects again, byte for byte. Those copies are one document, so the newest is taken to stand for all of them.
Pricing visibility
DealerConnect publishes one guide as three PDFs — full pricing, MSRP only, no
pricing — under three different blob ids, and the bucket holds all three. The blob id
says nothing about which is which. The dealerconnect.code_guides BigQuery table does:
one row per collected PDF, carrying the var_name of the DealerConnect listing it came
from, whose third underscore-delimited field is the visibility.
dates2026_3_1_3500HEAVYDUTYREGCABArray full pricing — $MSRP and $FWP
dates2026_3_2_3500HEAVYDUTYREGCABArray MSRP only — $FWP blank
dates2026_3_3_3500HEAVYDUTYREGCABArray no pricing — neither column printed
Only the full-pricing PDF is partitioned. The other two are not merely price-free:
absent the $FWP/$MSRP pair, grids._layout_of reads the description column as
followed straight away by the group/trim columns, so the option tables shift their
column roles. Measured across the 1,038 same-edition triplets in the corpus, the two
priced siblings agree exactly on their non-price columns in 1,036, while the no-pricing
sibling disagrees in 531 and yields fewer rows in 389 — 2,796,638 feature rows against
the priced 2,885,405.
ai_dealer_connect.code_guides.visibility reads the visibility out of a listing name;
fetch_registry in the raw component queries the table. A blob id the registry
places at another visibility, does not place at all, or places inconsistently gets no
partition — an unplaced guide is not one to keep. An empty registry raises rather than
retaining nothing.
The collection pipeline reads only the full-pricing arrays, so every listing the
registry holds is a full-pricing one: all 1,412 rows encode 1. The first field varies
across 1–4 and is the brand, not the visibility. The blob ids within one edition
usually count up from the full-pricing PDF, but not reliably — 99 of the 1,412 listed
guides are not the lowest id of their edition, so the listing is the only sound
discriminator.
Removing what was partitioned before
The bucket predates the registry, so raw/code_guides holds partitions the registry
never listed. scripts/migrations/list_code_guide_partitions_to_remove.py sorts every
materialized partition into four groups and lists the two to remove:
| group | ||
|---|---|---|
keep | the registry lists it | 1,412 |
suppressed | unlisted, but the registry lists another PDF of the same edition | 2,423 |
unlisted_unpriced | the registry lists no PDF of this edition and it prints no prices | 25 |
unlisted_priced | the registry lists no PDF of this edition and it prints prices | 13 |
unlisted_priced is a registry coverage gap rather than a suppressed variant, so those
are kept and reported. Nineteen listed guides print no prices at all — pre-release
editions published before pricing is set — and the daily view's recency rule supersedes
them as soon as a priced edition lands.
Each key names a partition of raw/code_guides and of every _by_edition table, plus a
Dagster dynamic partition in the stellantis_us_code_guides set — delete the dynamic
partition alongside the rows, since on_missing re-materializes any key that keeps one.
The parse
transformed/code_guides_{models,features,rules}_by_edition read the stored bytes for their
partition and parse them. They share one op because derive_models and derive_features
read the same parse_document envelope, and the envelope is the expensive part.
derive_rules reads the features that op derives.
A run may select any subset of the three, and writes only the tables it selects. Every run still parses the whole guide.
is_configurable_overrides is a list of SQL WHEN … THEN false arms composed into one
CASE over the feature rows — for codes a guide prints as orderable that are
administrative or standard equipment. Forced-false only: a forced-true belongs in
the consolidated derived_fields, after the subcategory visibility guard that would
undo it here.
A GROUP DETAIL item is not configurable under its group when the trim has the code whichever customer preferred package (CPP) is ordered. That holds in two cases:
- STANDARD EQUIPMENT marks the code for the model code.
- The code is unconditional PACKAGE CONTENT of every CPP the trim offers.
GROUP DETAIL lists a group's contents once per guide, not per trim, so a group can restate
equipment that a trim already has. The 2027 Compass guide lists heated steering wheel
NHS under AHM CONVENIENCE GROUP, which only Latitude Altitude can order, and marks
NHS standard on the Latitude. A configurable record for that restated item presents the
code as a choice. acceptance/dealer_rules then infers a must-have for the code from
every vehicle that has it as standard. A STEP #3 "packaged into" mark on the same item
does not change this, because the mark records group membership.
is_fleet_only says why a feature is not a choice, which is_configurable alone
cannot: package content, standard equipment and codes never printed as orderable are
all equally unconfigurable. A code carries it when a FLEET ONLY banner or variant
qualifier named it for that trim and nothing offered it as a retail choice. This applies
to 2,162 of the corpus's 67,056 feature records, overwhelmingly the X9* and 5U*
administrative codes that feature_catalog independently classes as non-selectable.
consolidated/features reads is_fleet_only from the code_guides source group.
The current view
transformed/code_guides_{models,features,rules} are daily and read every guide
partition through an AllPartitionMapping. For their partition date they:
- drop model years before the partition date's calendar year minus
prior_model_years. Stellantis sets it to 2, so the 2026-10-19 partition keeps MY2024 and later.consolidated/featuresreads a year of these partitions, and every model year a guide still prints would otherwise be in each of them; - date each edition
coalesce(reissue_date, issue_date)— a reissue never predates its issue, so the coalesced date alone bounds both; - drop editions published after the partition date, which is what makes a past partition reproducible: it sees the guides that existed then, not the ones that exist now;
- keep, per
(brand, model_year, model_family)— the scopelineage_keyuses — every row of the newest surviving edition, breaking a tie on the blob id:ISSUEDis a season-open date that several guides print at once, so two editions of one vehicle can share a date, and the id counts up as DealerConnect publishes.
The features and rules views choose the newest edition among the guides in
transformed/code_guides_models_by_edition, not among their own rows. A reissue that
prints no rules has no rule rows, but it still replaces the previous edition's rules.
A guide with no model rows has no trims, so it has no feature or rule rows either.
model_family, issue_date, reissue_date and _blob_id ride alongside the entity
columns on the per-guide tables and leave with the edition they selected, so the daily
output carries exactly the entity's own columns. brand is one of those entity columns,
so it scopes the edition and stays.
The rules
derive_rules reads the rules each feature record holds, and the component writes
each rule as an ai_core.entities.RuleEntity
row. The parser has already put each rule on the feature records of the trims it
applies to, as rule-grammar.md describes. Those rules come from
several parts of a guide:
- the annotations printed under an option's description;
- the clause printed under a CPP in the powertrain matrix;
- the clauses printed on PACKAGE CONTENT and GROUP DETAIL items;
- the paint sections, whose paint-type boxes print rules. From each section the parser
also derives a
requires_one_offrom each color to its paint type, and anonly_one_ofover the paint types.
The rules contract defines what each
rule_type means. The parser's rule types map onto it as follows:
OptionRule.type | rule_type | rule_if | rule_then |
|---|---|---|---|
excludes | exclusion | the record's option_code, then if_codes | the codes named |
requires_one_of | inclusion | the record's option_code, then if_codes | the codes named |
only_one_of | only_one_of | empty | the codes named |
A code of the form 2T<trim> names a trim, not an option. A trim satisfies a
requires_one_of that names it, so derive_rules writes no rule on that trim. On any
other trim it leaves the code out of the rule. On trim J, for example,
M/H AAW OR ACY OR 2TB becomes requires_one_of [AAW, ACY].
A guide names order types as codes:
4EX, a stock order;4EA, a custom order;4FT, a fleet order.
None of them is an option. derive_rules writes them into rule_if and rule_then like
any other code. M/H 4EA OR 4FT on an option is an
inclusion of [4EA, 4FT], and W/4EX M/H MHV under package 27X is an inclusion of
MHV with rule_if [27X, 4EX]. Synthetic candidates are checked against these rules as
stock orders (see Order type).
The record that holds an only_one_of need not be one of its members. Every color in a
paint-type box holds the only_one_of over the section's paint types, for example.
An option has one feature record per package that installs it, and an only_one_of is
held by several records, so derive_rules reads the same rule many times. It keeps
each rule once per (model_code, model_year, trim_identifier, subject_code), and each
only_one_of rule once per trim and set of members.
The guides give their rules no identifier, so code is null. rule_key is built from
the rule's own codes, so a rule keeps its rule_id when a reissued guide adds or
removes other rules.
Each rule's validity is unbounded. A guide does not date its rules, and
current_edition selects the edition in effect on the partition date.
Restricting the rules to the trim
A rule can name a code its model and trim do not offer:
- an option sold on a sibling model code in the same section;
- an option sold on another trim;
- a code no trim offers.
transformed/code_guides_rules keeps the codes as printed. enriched/code_guides_rules
keeps only the codes that consolidated/features lists for the rule's model and trim,
from any feature feed:
| where the missing code is | effect |
|---|---|
| the IF part | the condition never holds, so the rule is dropped |
a THEN code of an exclusion or inclusion | the code is dropped, and the rule too if none is left |
a member of an only_one_of | the member is dropped, and the rule too if at most one is left |
After the restriction, rule_key is rebuilt from the codes that remain. For an
only_one_of, subject_code is rebuilt too. Two rules left with the same codes are
kept once. Consolidated reads the enriched rules, so every code a rule names resolves
to a feature on the rule's trim.
The catalog_restriction check fails when the catalog is empty or no rule remains after
the restriction, which is what a broken catalog join produces. Its metadata counts the
rules and THEN codes dropped, and names the THEN codes dropped most often.
Task sizing
Fargate sizing is read per step, so the two layers are sized apart:
by_edition_ecs covers the per-guide parse and daily_ecs each daily view. Either
unset falls back to the component's own ecs_vcpu/ecs_memory_gb, and that to the
deployment default. The daily views run at 8 GB; the parse holds one guide's records at
a time and takes the default.
The _by_edition assets are outputs of one op, so no configuration sizes them apart —
splitting them would parse every PDF more than once.
When a guide does not parse
The per-guide parse hard-fails rather than writing what it could not read. Bytes that
are not a PDF raise, and so does a guide the parser read but drew nothing from — no
models, or no features, or no ISSUED date to place the edition in time. Every guide in
the corpus yields both kinds, so an empty side is a parse that failed. Writing it
would drop a document out of the tables behind a successful materialization.
The materialization fails and names the blob id. The daily views are eager, so they wait for every guide partition and hold until that guide is fixed and parsed: a daily partition is the whole catalog or it is nothing.
Linking editions across time
The running page header prints six identity fields on every page:
RAM United States REISSUED: 08/19/2026 PAGE: 2
2027 2500 HEAVY DUTY ISSUED: 05/22/2026 REVISED
ISSUED does not move when a guide is reissued; REISSUED is the edition. Both
revision pairs in the fixture set confirm it: RAM 2500 279402 and 278388 share
ISSUED 2026-05-22, and Grand Cherokee 279384 and 277944 share 2026-05-05. So
lineage_key groups editions, and edition_key distinguishes them.
Two things worth knowing before you rely on it:
model_familyis load-bearing in the key.ISSUED: 2026-05-05is shared by five of the ten guides — it is a season-open date, not a per-guide one — so a key without the name would merge four unrelated guides into one lineage.- The document number is not in the document.
279402,278388,279384appear only in the filename.edition_keytherefore uses the reissue date, and two editions reissued the same day would collide.manifest.jsonrecords the number per fixture, from the filename.
Read once, from page 2 rather than the cover: the cover sets the brand as vertically
stacked letters, so word extraction there gives R / A / M on three lines.
It reproduces every one of the 73,641 hand-read feature records on all ten complete guides exactly — all twelve fields, including record order: nothing missing, nothing invented, no field wrong.
pants test projects/ai_stellantis/tests/code_guides::
Layout
The parser reads a DealerConnect artifact and knows nothing of what we make of one, so it lives with the other DealerConnect clients:
packages/ai_dealer_connect/src/ai_dealer_connect/code_guides/
parser/
api.py parse(), parse_document(), and the contract calls
read.py the reading: a path in, dicts out
models.py OptionRule, the CodeGuideDocument envelope, its raw material
parser.py document identity, cover index, sections, trims, powertrain
grids.py STEP #3 / STEP #4 tables
shared_tables.py GROUP DETAIL and STANDARD EQUIPMENT
annotations.py clause text -> OptionRule
tables.py ruled-table recovery from the page's own lines
layout.py spans, words and rules
text.py money, sales codes, band membership
The records are ours, not the guide's, so they stay with the pipeline that wants them:
projects/ai_stellantis/src/ai_stellantis/sources/
code_guides.py CodeGuideModelRecord, CodeGuideFeatureRecord, and the
extraction that builds them from an envelope. Imports
`parser.models` and nothing else — no reader, no PDF library
One layer, no dispatch, and one direction: read.py says what a document contains,
sources/code_guides.py says what follows from it. Nothing in it opens a file — a
test asserts that import boundary rather than trusting it. The package boundary
enforces the other direction: ai_dealer_connect cannot reach the records at all.
How the reader works
packages/ai_dealer_connect/src/ai_dealer_connect/code_guides/parser/tables.py does the heavy lifting: a contiguous run of a page's left
border is one table, and the rules crossing that run are its row and column bands.
Everything above it addresses cells by index wherever it can: features,
annotations and text never see a coordinate, though parser, grids and
group_detail still do their own geometry alongside tables and layout.
| stage | module | what it reads |
|---|---|---|
| document identity | parser.parse_doc_meta | running page header — brand left of United States, model year from the bold title |
| cover index | parser.parse_index | page 1's MODEL / CODE / PAGES table, read outwards from the bold model codes |
| section split | parser.find_section_pages | pages carrying STEP #1; the last section stops where the shared tables start |
| model code, vehicle, price | parser._parse_step1 | the only bold six-character code on the page, and its row |
| trims | parser._trim_columns | POWERTRAIN AVAILABILITY columns right of the matrix's own scaffolding |
| per-trim price | parser._msrp_by_prefix | MANUFACTURER SUGGESTED RETAIL PRICE, matched by where each number is printed |
| options | grids.parse_grids | STEP #3 / STEP #4 tables — description, annotation, price variants, cells |
| annotations → rules | annotations.parse_annotation | M/H / N/A W/ / ONLY 1: clauses → OptionRule |
| package contents | group_detail, grids.parse_package_items | GROUP DETAIL, STANDARD EQUIPMENT, and the "Included in Equipment Groups" list |
| derivation | extract.derive_features | merges every source and decides is_configurable |
Testing
pants test packages/ai_dealer_connect/tests/code_guides:: # the parser
pants test projects/ai_stellantis/tests/code_guides:: # the derivation
Every test taking a case argument runs once per fixture. Skips are fixtures lacking the
expectation a test needs — a partial fixture, or a guide with no hand-read GROUP DETAIL.
| file | covers |
|---|---|
test_contract.py | the parser's public API and the derivation over its envelope |
test_api.py | the public surface and input handling |
test_extract.py | record definitions and the extraction, synthetic and against fixtures |
test_parser.py test_grids.py test_rules.py | the reader's internals |
The feature fixtures are complete: expected_features.json holds every record of every
guide — 73,641 — each read off the page, with expected_features_provenance.json naming
the pages and source behind each. feature_counts in each manifest.json is a cheap drift
tripwire over the same records, not ground truth itself.
is_configurable on a restated GROUP DETAIL item is set by the rule in
The parse, applied to each fixture's own hand-read is_standard and
PACKAGE CONTENT records rather than to parser output.
Two things are asserted by rule rather than read, and are recorded as underdetermined:
short_description follows a precedence rule because 88 of 110 disagreements have both
candidate strings printed on the page (judgment call 10); and ASQ on DJ7X91 is taken
from GROUP DETAIL's E mark over a blank STEP #3 row that contradicts it.
Fixtures live one directory per PDF under packages/ai_dealer_connect/tests/data/code_guides/parser/fixtures/, named
<brand>_<model>_<model_year>_<issued>[_<reissued>] — the identity that makes a guide
unique, so two editions of one guide cannot collide. Each holds a manifest.json
plus whichever expectations have been hand-built. Tests parameterise over the discovered
directories and skip a case when a fixture is absent, so a new PDF can be onboarded one
fixture at a time.
Ground truth was read off rendered page images by eye, before the parser existed, and independently re-derived by separate reviewers:
- model records — every guide's records were re-extracted by a second reviewer working only from the PDF; the RAM guide got three independent passes
- the annotation grammar — reviewers classified each guide's whole corpus from a written grammar and agreed with each other and the parser on every block
- at least one whole STEP #3 or STEP #4 page per PDF, transcribed cell by cell — some guides have three
Regenerate a fixture only by re-reading the PDF — never from parser output. Where a fixture and a page disagree, the page is right; where the parser disagrees with a fixture, measure the page before concluding either is wrong.
Two reasons a fixture can miss a real defect: a record count is equally stable whether an
option is coded UBQ or USA. The annotation corpus is also regenerated from the parser's
own input, so text the parser never looks at can never fail an exhaustiveness test. That is
why the hand read is cell-level and complete rather than a set of totals.
What the fixture PDFs disagree about
The set is deliberate: nearly every structural assumption one guide supports, another
breaks. Ten guides are onboarded, including three that are later revisions of another in
the set — a revision is the cheapest source of a structure the first pass never saw
(RAM 279402 grew a trim column, and with it the guide's only trim whose 24- price cell
is blank).
| RAM 2500 | Grand Wagoneer | Grand Cherokee | Pacifica | |
|---|---|---|---|---|
| STEP #3 columns | equipment tiers (BASE, A7B) | LLP (2_E) | trim name over LLP | LLP, one column |
| option code | in the description … (DSA) | in the cell | in the cell | in the cell |
| annotation face | bold oblique | bold upright | bold upright | bold upright |
FLEET ONLY | per price sub-row | banner | banner | banner |
| CPPs per trim | two (a real choice) | one | one | one |
| trim column heading | trim name | trim name | name + LLP | none at all |
| STEP #3 / #4 pages | separate | separate | same page | same page |
| prices | all printed | one section prints none | all printed | all printed |
Every fixture but one is the full-pricing listing of its edition, so the corpus reads the
same documents the pipeline partitions. The exception is Grand Wagoneer SWB 2026
(277696), a no-pricing listing kept deliberately: it prints no $FWP/$MSRP pair on
any page and is the only fixture exercising the branch of grids._layout_of that reads a
table whose description column runs straight into the group/trim columns.
A whole-guide blank $FWP is an MSRP-only listing, and a guide printing neither price
column is a no-pricing one — three of the fixtures are listings the sensor does not
partition (Grand Cherokee 277946, Pacifica 275927, Grand Wagoneer 277696). They stay in
the corpus: the parser reads what a guide says regardless of which listing published it,
and the no-pricing layout is the only fixture exercising the absent-price-pair branch of
grids._layout_of.
Traps worth knowing about
- A sales code is the last parenthesis, not the first.
UCONNECT 5 NAV W 12.0" DISPLAY (USA) (UBQ)isUBQ, notUSA;8-SPD AUTO 880RE TRANS (MAKE) (DC1)isDC1, notMAKE. Both wrong answers validate as option codes. DJ7X91has three trim columns, two of them both headedPOWER WAGON(trimsPandW), and a sparse price table where each row leaves a different column blank. Packing values left-to-right silently mis-prices it.- A cell's border is drawn per row and can be split mid-cell, so segments must be joined before asking whether a rule spans a column — that is what separates an option row from its own price variants.
- Fixed-grid line bucketing splits a row whose words differ by 0.2 pt; the word grouper clusters on a running baseline instead.
- Decide "is this a color table?" per table, not per page — two of the four guides print STEP #3 and STEP #4 on one page, and a page-level test marks every option on such a page as an always-configurable color.
- A
Punder a trim column is not a parent package. Under an equipment-group tier it is; under a trim column the column is the trim, and treating it as a parent invents a package named2_J. - A cell spanning several price sub-rows repeats its text once per row, so a
packaged cell can read
"P P"rather than"P".
The repeated-cell case is the most recent: a cell's text
is read one line per ruled band with each distinct line kept once, so "P P" reads "P".
The last of them is the trap that has cost the most, and the one place where reading the page by eye is not good enough: at page scale a rule dividing one row into bands looks exactly like a rule between two rows. Six tables across four guides were recorded wrongly, in both directions, before the rule was measured instead of argued. The test is arithmetic, not judgement — read the page's ruling lines and ask which ones cross the description column:
290.70 ROW 50.6 → 590.2 <- crosses it: a row boundary
302.10 band 356.6 → 590.2 <- starts at the price columns: one row, two bands
313.40 ROW 50.6 → 590.2
Three patterns band rather than split, and all three are one row:
- a code per trim column: the row holds the union of the bands' cells.
- a price per variant: the row holds the first price — the schema has one
msrp, so the variant is lost. - the same cell repeated once per band.
When the parser disagrees with a fixture here, measure the ruling lines before assuming either is wrong.