Code guide parser

Deterministic parser for Stellantis Customer Preferred Code Guide PDFs. No LLM in the loop — tables are recovered from the PDF's own ruling lines and font weights, so the same PDF always yields the same records.

The parser reads what a guide says; ai_stellantis.sources.code_guides turns that into CodeGuideModelRecord (one per trim variant) and CodeGuideFeatureRecord (one per option × model × trim × parent package).

from ai_dealer_connect.code_guides.parser import parse

guide = parse("packages/ai_dealer_connect/tests/data/code_guides/parser/fixtures/jeep_grand_cherokee_2row_2027_2026-05-05_2026-06-23/code-guides_2026-06-24_277944.pdf")
guide.trim_specs        # what each trim carries
guide.page_grids(3)     # the STEP #3 / STEP #4 option tables on page 3

Or the whole document as one envelope, and the records that follow from it:

from ai_stellantis.sources.code_guides import derive_features, derive_models
from ai_dealer_connect.code_guides.parser import parse_document

doc = parse_document("…/code-guides_2026-08-19_279402.pdf")
doc.identity            # brand, country, model_year, model_family, issued/reissued dates
doc.section_grids       # every STEP #3 / #4 table, with its price bands
doc.standard_equipment  # and GROUP DETAIL, PACKAGE CONTENT, the powertrain matrix…
doc.annotations         # every distinct annotation block, paired with its rules
doc.lineage_key         # 'ram|United States|2027|2500 HEAVY DUTY CREW CAB|2026-05-22'

derive_models(doc)      # list[CodeGuideModelRecord]
derive_features(doc)    # list[CodeGuideFeatureRecord]

The envelope holds what the document says; the records follow from it. Nothing derived lives on CodeGuideDocument — a field cannot be both the derivation's output and its input — so derive_models and derive_features take an envelope and open nothing. They run on one rebuilt from JSON, which is how the rulings in features-ground-truth.md can be exercised without a PDF at all.

Records are reached one way only. parse() returns a guide with no models or features on it, and the parser package exports neither the record classes nor the derivations — a caller wanting records goes to extract and hands it an envelope. The lazy views stay lazy so page_grids(3) need not parse 175 pages, while parse_document parses everything and is the serialization boundary.

The Dagster assets

Two components. StellantisCodeGuidesRawComponent owns the stored PDFs and the dynamic partitions; StellantisCodeGuidesComponent owns everything derived from them.

asset	partition	rows
raw/code_guides	one per guide	one — the PDF's own bytes
transformed/code_guides_models_by_edition	one per guide	that guide's trim variants
transformed/code_guides_features_by_edition	one per guide	that guide's (option, model code, trim, parent package) rows
transformed/code_guides_models	daily	the current edition's trim variants
transformed/code_guides_features	daily	the current edition's options

A guide is a document, so the partition is the blob id its filename carries — code-guides/2026-07-25/278922.pdf is partition 278922. Both components derive the partitions-definition name the same way, so they address one set.

The raw asset

raw/code_guides reads one PDF from GCS and stores its bytes:

column
partition	the blob id, and the Iceberg partition column
content	the PDF, as Binary
content_type	application/pdf
fetched_at	when it was read

A stellantis_us_code_guides sensor reads the code_guides registry every five minutes and adds a partition for every full-pricing blob id it has not seen. The registry is the record of what the collection pipeline stored, so the sensor never lists the bucket; it requests no runs of its own either — the asset's on_missing condition fills whichever partitions have never been materialized.

The asset resolves the bytes when it runs, matching the blob id server-side. One id can appear under several date directories: the collection pipeline re-files a guide it collects again, byte for byte. Those copies are one document, so the newest is taken to stand for all of them.

Pricing visibility

DealerConnect publishes one guide as three PDFs — full pricing, MSRP only, no pricing — under three different blob ids, and the bucket holds all three. The blob id says nothing about which is which. The dealerconnect.code_guides BigQuery table does: one row per collected PDF, carrying the var_name of the DealerConnect listing it came from, whose third underscore-delimited field is the visibility.

dates2026_3_1_3500HEAVYDUTYREGCABArray   full pricing — $MSRP and $FWP
dates2026_3_2_3500HEAVYDUTYREGCABArray   MSRP only    — $FWP blank
dates2026_3_3_3500HEAVYDUTYREGCABArray   no pricing   — neither column printed

Only the full-pricing PDF is partitioned. The other two are not merely price-free: absent the $FWP/$MSRP pair, grids._layout_of reads the description column as followed straight away by the group/trim columns, so the option tables shift their column roles. Measured across the 1,038 same-edition triplets in the corpus, the two priced siblings agree exactly on their non-price columns in 1,036, while the no-pricing sibling disagrees in 531 and yields fewer rows in 389 — 2,796,638 feature rows against the priced 2,885,405.

ai_dealer_connect.code_guides.visibility reads the visibility out of a listing name; fetch_registry in the raw component queries the table. A blob id the registry places at another visibility, does not place at all, or places inconsistently gets no partition — an unplaced guide is not one to keep. An empty registry raises rather than retaining nothing.

The collection pipeline reads only the full-pricing arrays, so every listing the registry holds is a full-pricing one: all 1,412 rows encode 1. The first field varies across 1–4 and is the brand, not the visibility. The blob ids within one edition usually count up from the full-pricing PDF, but not reliably — 99 of the 1,412 listed guides are not the lowest id of their edition, so the listing is the only sound discriminator.

Removing what was partitioned before

The bucket predates the registry, so raw/code_guides holds partitions the registry never listed. scripts/migrations/list_code_guide_partitions_to_remove.py sorts every materialized partition into four groups and lists the two to remove:

group
keep	the registry lists it	1,412
suppressed	unlisted, but the registry lists another PDF of the same edition	2,423
unlisted_unpriced	the registry lists no PDF of this edition and it prints no prices	25
unlisted_priced	the registry lists no PDF of this edition and it prints prices	13

unlisted_priced is a registry coverage gap rather than a suppressed variant, so those are kept and reported. Nineteen listed guides print no prices at all — pre-release editions published before pricing is set — and the daily view's recency rule supersedes them as soon as a priced edition lands.

Each key names a partition of raw/code_guides and of both _by_edition tables, plus a Dagster dynamic partition in the stellantis_us_code_guides set — delete the dynamic partition alongside the rows, since on_missing re-materializes any key that keeps one.

The parse

transformed/code_guides_{models,features}_by_edition read the stored bytes for their partition and parse them. They share one op because derive_models and derive_features read the same parse_document envelope, and the envelope is the expensive part.

is_configurable_overrides is a list of SQL WHEN … THEN false arms composed into one CASE over the feature rows — for codes a guide prints as orderable that are administrative or standard equipment. Forced-false only: a forced-true belongs in the consolidated derived_fields, after the subcategory visibility guard that would undo it here.

is_fleet_only says why a feature is not a choice, which is_configurable alone cannot: package content, standard equipment and codes never printed as orderable are all equally unconfigurable. A code carries it when a FLEET ONLY banner or variant qualifier named it for that trim and nothing offered it as a retail choice. This applies to 2,162 of the corpus's 67,056 feature records, overwhelmingly the X9* and 5U* administrative codes that feature_catalog independently classes as non-selectable. It reaches consolidated/features from the code_guides source group, and sits in discrepancy_ignore_fields because the other feature feeds carry the schema default rather than a reading of their own.

The current view

transformed/code_guides_{models,features} are daily and read every guide partition through an AllPartitionMapping. For their partition date they:

1. date each edition coalesce(reissue_date, issue_date) — a reissue never predates its issue, so the coalesced date alone bounds both;
2. drop editions published after the partition date, which is what makes a past partition reproducible: it sees the guides that existed then, not the ones that exist now;
3. keep, per (brand, model_year, model_family) — the scope lineage_key uses — every row of the newest surviving edition, breaking a tie on the blob id: ISSUED is a season-open date that several guides print at once, so two editions of one vehicle can share a date, and the id counts up as DealerConnect publishes.

model_family, issue_date, reissue_date and _blob_id ride alongside the entity columns on the per-guide tables and leave with the edition they selected, so the daily output carries exactly the entity's own columns. brand is one of those entity columns, so it scopes the edition and stays.

Task sizing

Fargate sizing is read per step, so the two layers are sized apart: by_edition_ecs covers the per-guide parse and daily_ecs each daily view. Either unset falls back to the component's own ecs_vcpu/ecs_memory_gb, and that to the deployment default. The daily views run at 8 GB; the parse holds one guide's records at a time and takes the default.

The two _by_edition assets are outputs of one op, so no configuration sizes them apart — splitting them would parse every PDF twice.

When a guide does not parse

The per-guide parse hard-fails rather than writing what it could not read. Bytes that are not a PDF raise, and so does a guide the parser read but drew nothing from — no models, or no features, or no ISSUED date to place the edition in time. Every guide in the corpus yields both kinds, so an empty side is a parse that failed. Writing it would drop a document out of the tables behind a successful materialization.

The materialization fails and names the blob id. The daily views are eager, so they wait for every guide partition and hold until that guide is fixed and parsed: a daily partition is the whole catalog or it is nothing.

Linking editions across time

The running page header prints six identity fields on every page:

RAM United States        REISSUED: 08/19/2026    PAGE: 2
2027 2500 HEAVY DUTY     ISSUED:   05/22/2026    REVISED

ISSUED does not move when a guide is reissued; REISSUED is the edition. Both revision pairs in the fixture set confirm it: RAM 2500 279402 and 278388 share ISSUED 2026-05-22, and Grand Cherokee 279384 and 277944 share 2026-05-05. So lineage_key groups editions, and edition_key distinguishes them.

Two things worth knowing before you rely on it:

• model_family is load-bearing in the key. ISSUED: 2026-05-05 is shared by five of the ten guides — it is a season-open date, not a per-guide one — so a key without the name would merge four unrelated guides into one lineage.
• The document number is not in the document. 279402, 278388, 279384 appear only in the filename. edition_key therefore uses the reissue date, and two editions reissued the same day would collide. manifest.json records the number per fixture, from the filename.

Read once, from page 2 rather than the cover: the cover sets the brand as vertically stacked letters, so word extraction there gives R / A / M on three lines.

It reproduces every one of the 73,641 hand-read feature records on all ten complete guides exactly — all twelve fields, including record order: nothing missing, nothing invented, no field wrong.

pants test projects/ai_stellantis/tests/code_guides::

Layout

The parser reads a DealerConnect artifact and knows nothing of what we make of one, so it lives with the other DealerConnect clients:

packages/ai_dealer_connect/src/ai_dealer_connect/code_guides/
  parser/
    api.py          parse(), parse_document(), and the contract calls
    read.py         the reading: a path in, dicts out
    models.py       OptionRule, the CodeGuideDocument envelope, its raw material
    parser.py       document identity, cover index, sections, trims, powertrain
    grids.py        STEP #3 / STEP #4 tables
    back_matter.py  GROUP DETAIL and STANDARD EQUIPMENT
    annotations.py  clause text -> OptionRule
    tables.py       ruled-table recovery from the page's own lines
    layout.py       spans, words and rules
    text.py         money, sales codes, band membership

The records are ours, not the guide's, so they stay with the pipeline that wants them:

projects/ai_stellantis/src/ai_stellantis/sources/
  code_guides.py    CodeGuideModelRecord, CodeGuideFeatureRecord, and the
                    extraction that builds them from an envelope. Imports
                    `parser.models` and nothing else — no reader, no PDF library

One layer, no dispatch, and one direction: read.py says what a document contains, sources/code_guides.py says what follows from it. Nothing in it opens a file — a test asserts that import boundary rather than trusting it. The package boundary enforces the other direction: ai_dealer_connect cannot reach the records at all.

How the reader works

packages/ai_dealer_connect/src/ai_dealer_connect/code_guides/parser/tables.py does the heavy lifting: a contiguous run of a page's left border is one table, and the rules crossing that run are its row and column bands. Everything above it addresses cells by index wherever it can: features, annotations and text never see a coordinate, though parser, grids and group_detail still do their own geometry alongside tables and layout.

stage	module	what it reads
document identity	parser.parse_doc_meta	running page header — brand left of United States, model year from the bold title
cover index	parser.parse_index	page 1's MODEL / CODE / PAGES table, read outwards from the bold model codes
section split	parser.find_section_pages	pages carrying STEP #1; the last section stops where the shared back matter starts
model code, vehicle, price	parser._parse_step1	the only bold six-character code on the page, and its row
trims	parser._trim_columns	POWERTRAIN AVAILABILITY columns right of the matrix's own scaffolding
per-trim price	parser._msrp_by_prefix	MANUFACTURER SUGGESTED RETAIL PRICE, matched by where each number is printed
options	grids.parse_grids	STEP #3 / STEP #4 tables — description, annotation, price variants, cells
annotations → rules	annotations.parse_annotation	M/H / N/A W/ / ONLY 1: clauses → OptionRule
package contents	group_detail, grids.parse_package_items	GROUP DETAIL, STANDARD EQUIPMENT, and the "Included in Equipment Groups" list
derivation	extract.derive_features	merges every source and decides is_configurable

Testing

pants test packages/ai_dealer_connect/tests/code_guides::  # the parser
pants test projects/ai_stellantis/tests/code_guides::                  # the derivation

Every test taking a case argument runs once per fixture. Skips are fixtures lacking the expectation a test needs — a partial fixture, or a guide with no hand-read GROUP DETAIL.

file	covers
test_contract.py	the parser's public API and the derivation over its envelope
test_api.py	the public surface and input handling
test_extract.py	record definitions and the extraction, synthetic and against fixtures
test_parser.py test_grids.py test_rules.py	the reader's internals

The feature fixtures are complete: expected_features.json holds every record of every guide — 73,641 — each read off the page, with expected_features_provenance.json naming the pages and source behind each. feature_counts in each manifest.json is a cheap drift tripwire over the same records, not ground truth itself.

Two things are asserted by rule rather than read, and are recorded as underdetermined: short_description follows a precedence rule because 88 of 110 disagreements have both candidate strings printed on the page (judgment call 10); and ASQ on DJ7X91 is taken from GROUP DETAIL's E mark over a blank STEP #3 row that contradicts it.

Fixtures live one directory per PDF under packages/ai_dealer_connect/tests/data/code_guides/parser/fixtures/, named <brand>_<model>_<model_year>_<issued>[_<reissued>] — the identity that makes a guide unique, so two editions of one guide cannot collide. Each holds a manifest.json plus whichever expectations have been hand-built. Tests parameterise over the discovered directories and skip a case when a fixture is absent, so a new PDF can be onboarded one fixture at a time.

Ground truth was read off rendered page images by eye, before the parser existed, and independently re-derived by separate reviewers:

• model records — every guide's records were re-extracted by a second reviewer working only from the PDF; the RAM guide got three independent passes
• the annotation grammar — reviewers classified each guide's whole corpus from a written grammar and agreed with each other and the parser on every block
• at least one whole STEP #3 or STEP #4 page per PDF, transcribed cell by cell — some guides have three

Regenerate a fixture only by re-reading the PDF — never from parser output. Where a fixture and a page disagree, the page is right; where the parser disagrees with a fixture, measure the page before concluding either is wrong.

Two reasons a fixture can miss a real defect: a record count is equally stable whether an option is coded UBQ or USA. The annotation corpus is also regenerated from the parser's own input, so text the parser never looks at can never fail an exhaustiveness test. That is why the hand read is cell-level and complete rather than a set of totals.

What the fixture PDFs disagree about

The set is deliberate: nearly every structural assumption one guide supports, another breaks. Ten guides are onboarded, including three that are later revisions of another in the set — a revision is the cheapest source of a structure the first pass never saw (RAM 279402 grew a trim column, and with it the guide's only trim whose 24- price cell is blank).

	RAM 2500	Grand Wagoneer	Grand Cherokee	Pacifica
STEP #3 columns	equipment tiers (BASE, A7B)	LLP (2_E)	trim name over LLP	LLP, one column
option code	in the description … (DSA)	in the cell	in the cell	in the cell
annotation face	bold oblique	bold upright	bold upright	bold upright
FLEET ONLY	per price sub-row	banner	banner	banner
CPPs per trim	two (a real choice)	one	one	one
trim column heading	trim name	trim name	name + LLP	none at all
STEP #3 / #4 pages	separate	separate	same page	same page
prices	all printed	one section prints none	all printed	all printed

Every fixture but one is the full-pricing listing of its edition, so the corpus reads the same documents the pipeline partitions. The exception is Grand Wagoneer SWB 2026 (277696), a no-pricing listing kept deliberately: it prints no $FWP/$MSRP pair on any page and is the only fixture exercising the branch of grids._layout_of that reads a table whose description column runs straight into the group/trim columns.

A whole-guide blank $FWP is an MSRP-only listing, and a guide printing neither price column is a no-pricing one — three of the fixtures are listings the sensor does not partition (Grand Cherokee 277946, Pacifica 275927, Grand Wagoneer 277696). They stay in the corpus: the parser reads what a guide says regardless of which listing published it, and the no-pricing layout is the only fixture exercising the absent-price-pair branch of grids._layout_of.

Traps worth knowing about

• A sales code is the last parenthesis, not the first. UCONNECT 5 NAV W 12.0" DISPLAY (USA) (UBQ) is UBQ, not USA; 8-SPD AUTO 880RE TRANS (MAKE) (DC1) is DC1, not MAKE. Both wrong answers validate as option codes.
• DJ7X91 has three trim columns, two of them both headed POWER WAGON (trims P and W), and a sparse price table where each row leaves a different column blank. Packing values left-to-right silently mis-prices it.
• A cell's border is drawn per row and can be split mid-cell, so segments must be joined before asking whether a rule spans a column — that is what separates an option row from its own price variants.
• Fixed-grid line bucketing splits a row whose words differ by 0.2 pt; the word grouper clusters on a running baseline instead.
• Decide "is this a color table?" per table, not per page — two of the four guides print STEP #3 and STEP #4 on one page, and a page-level test marks every option on such a page as an always-configurable color.
• A P under a trim column is not a parent package. Under an equipment-group tier it is; under a trim column the column is the trim, and treating it as a parent invents a package named 2_J.
• A cell spanning several price sub-rows repeats its text once per row, so a packaged cell can read "P P" rather than "P".

The repeated-cell case is the most recent: a cell's text is read one line per ruled band with each distinct line kept once, so "P P" reads "P".

The last of them is the trap that has cost the most, and the one place where reading the page by eye is not good enough: at page scale a rule dividing one row into bands looks exactly like a rule between two rows. Six tables across four guides were recorded wrongly, in both directions, before the rule was measured instead of argued. The test is arithmetic, not judgement — read the page's ruling lines and ask which ones cross the description column:

  290.70  ROW      50.6 → 590.2     <- crosses it: a row boundary
  302.10  band     356.6 → 590.2    <- starts at the price columns: one row, two bands
  313.40  ROW      50.6 → 590.2

Three patterns band rather than split, and all three are one row:

• a code per trim column: the row holds the union of the bands' cells.
• a price per variant: the row holds the first price — the schema has one msrp, so the variant is lost.
• the same cell repeated once per band.

When the parser disagrees with a fixture here, measure the ruling lines before assuming either is wrong.