Performance Notes
Memory and wall-time considerations for the ingestion pipeline. This page records the patterns that keep peak memory bounded on wide-and-tall entities (Stellantis Jeep US inventory, ~250k rows x ~70 columns at national scope), plus benchmark measurements and deferred work.
Some benchmark sections use helper names from the benchmark scenario. For
implementation work, use the current LazyFrame modules in
ai_harmonize/transformed/{frames,discrepancies}.py and
ai_dagster/components/transformed/component.py.
Benchmarking
Local benchmarks live in benchmarks/ at the repo root. See
benchmarks/README.md for the full harness:
# Fast iteration scenario (NYC + 50 mi radius, ~90 s)
benchmarks/run.sh stellantis_jeep_inventory_fast <state-label>
# Canonical national scenario (~17 min)
benchmarks/run.sh stellantis_jeep_inventory <state-label>
# Compare two summary.json files
python benchmarks/compare.py BASELINE.json CANDIDATE.json
Stage classification uses two independent thresholds (yellow / red), both for
peak memory (default 6 / 8 GB) and wall-clock duration (default 10 / 20
min). Container cap is informational; the classification drives pass/fail. A
STAGE_TIMEOUT_S watchdog SIGKILLs runaway stages.
Numbers from a benchmark run belong in the relevant PR description, not in
committed JSON files. They are tightly coupled
to the host machine and day-to-day API volume, so a checked-in
"baseline" decays quickly. The git SHA and branch are embedded in every
saved summary.json for forensic lookup.
Watch out for variable upstream API throughput
The Stellantis public inventory endpoint shows uneven response time during evening US peak hours. Symptom: per-page fetch time climbs from ~0.3 s/page early in a run to ~0.6 s/page mid-run, and the largest models (1000+ pages) can take 9-10 minutes each instead of the ~3-4 minutes seen on faster days. No 4xx/5xx responses — the requests just slow down. This is wall-clock noise, not a code regression, but it can trip the duration watchdog (default 30 min/stage) on national runs during peak load.
The slowdown is not per-IP rate limiting. A fresh VPN egress IP saw the same slow rates immediately, with no warm-up period. The slowness is server-side, not client-targeted. The likely cause is API saturation from consumer dealer-locator traffic during evening shopping hours. Time of day correlates more strongly than any source-IP behavior we observed.
Mitigations:
- Run during off-peak hours (US morning / midday) for the cleanest duration measurements. Memory measurements are unaffected by API throughput so they're durable regardless of run time.
- Bump
STAGE_TIMEOUT_Sto 2700-3600s on national runs during peak hours so the watchdog doesn't kill an otherwise-healthy run. - The inventory client rotates its User-Agent per fetch to match the
pattern in
ai_audi.sources.scs. Seerandom_headersinai_http. - Production runs from Dagster Cloud Serverless run during a configurable cron window — schedule them outside US evening hours.
Memory profile per tier
The benchmark analysis ranked these memory contributors by peak contribution:
-
Transformed-tier snapshot hashing. The benchmarked path stringified every column and MD5-ed the concatenation. On a 70-column inventory entity that materialized a per-row string roughly the size of the whole row, twice (once for the snapshot column, once during
concat_str). It now usespl.struct(*cols).hash()— a single Polars C++ pass over Arrow buffers, native handling of List/Struct columns, no per-row string materialization. -
Per-row Polars frame construction in transformed tier. Each raw inventory page (50 vehicles) produced its own
pl.from_dictscall. For Jeep at national scope ~5000 raw pages produced ~5000 tiny DataFrames inbatch_dfs. The current path coalesces to one frame per Arrow batch (collect_batches default ~1000 rows) — single-digitbatch_dfslength, larger amortization on eachpl.from_dicts. -
Raw-tier
list[dict]intermediate. The[make_raw_row(...) for ...]+pl.from_dicts(rows, ...)pattern held the row-dict list AND the resulting Polars frame simultaneously during construction.make_raw_row_frame(inai_core.schema) builds the frame from columnar inputs — constants likepartition_date,request_url,fetched_atare broadcast into a Series of length n instead of repeated in n separate dicts. -
serialize_json_fieldsreturning new dicts. Doubled transient memory by allocatingdict(row)per row before mutating. It mutates in place because the caller owns the row object. -
Unbounded
_extract_field_diffsaggregation. The check ran a.unique()aggregation over every diff column on every conflicting eid. On wide entities with many conflicts that allocation could explode. The current cap is_MAX_DISC = 50eids; diffs beyond that are not reported in metadata anyway. -
Per-row Polars filter in consolidated field policies.
_apply_cross_source_policiesransrc_df.filter(...)once per row to look up the today-source's matching record. It now uses a pre-built dict index keyed by(key_field, ...)tuple — O(1) lookup per row, single index per source.
Streaming patterns
Raw tier — per-sub-resource yield
The Stellantis inventory raw source (ai_public_api.stellantis.inventory:inventory) yields one
RawFetchResult per model year code instead of accumulating all pages
across all models into one frame. The IO manager
(AiPolarsIcebergIOManager.handle_output) writes each yielded chunk via
table_writer separately — overwrite on the first chunk, append on
subsequent. Single Iceberg snapshot per partition; bounded peak memory
per chunk regardless of total fan-out.
This pattern remains Stellantis-public-inventory-specific. If another OEM hits
the same scale, factor it out
into a StreamingRawSourceComponent base class in ai_dagster then.
Transformed tier — chunked write
TransformedEntityComponent yields a generator of pl.DataFrame slices
via _iter_chunks(df, _WRITE_CHUNK_SIZE) (default 50k rows, override
with TRANSFORMED_WRITE_CHUNK_SIZE env var). The IO manager writes each
chunk separately. The full transformed frame stays alive while chunks
iterate, but pyiceberg's encoder state is bounded per-chunk.
Shared subplans and CACHE nodes
A CACHE node appears in an explain(engine="streaming") plan when one
LazyFrame subplan is referenced from more than one branch of the same plan
tree. assert_no_cache_nodes in ai_core.polars detects them.
A CACHE buffers its own output — not its upstream, and not its consumers' output. Downstream row counts cannot flow back into it, so the cost is fixed at the width and height of the shared node alone. That makes CACHE worth eliminating when the shared node is a large intermediate, and not worth eliminating when it is a small key frame.
A shared subplan that is also the build side of every join consuming it is
the second case: a hash join materializes its build side regardless, so the
CACHE holds bytes that would be resident anyway. Collecting it with a
collect() moves the same rows from a cache buffer into a DataFrame and
adds a second pass over the upstream.
filter_must_haves_to_candidate_universe in
ai_dagster.derived.dealer_preferences is an instance. Its max_my_models
subplan — the max-model_year model_id per base_model_id, two Utf8
columns, one row per base_model_id — is referenced twice: once to scope
the candidate universe, once to attach model_id to each rule row. It is
the build side of both joins, so the plan carries one bounded CACHE by
design and assert_no_cache_nodes does not apply to it. The
assert_streaming_compatible check on that plan covers the nodes that do
scale with rule count.
Where memory in a plan like this one actually goes:
- Per-row payload. A
List(Struct(...))column joined onto a large frame is copied per output row. A 0.11 MB build-side list column attached to 2M rows measures 543 MB collected — bounded per-morsel under the streaming engine, but it is the term that scales. - Join fan-out. A key expected to be unique but duplicated multiplies
the frame, and every duplicate carries that payload.
max_my_modelsrelies on onemodel_idper (base_model_id,model_year); the consolidated models asset enforces it with an asset check.
Check the shared node's size and role before treating a CACHE as a defect.
BigQuery partition reads — Storage Read API
stream_bq_table_partition_arrow reads a date partition through the Storage
Read API rather than through a SELECT job. The Storage Read API reads physical
storage and has no query result-size limit, so it handles multi-GB partitions
that a SELECT rejects with responseTooLarge. BQReadStrategy selects between
the two per component: QUERY keeps the query cache and the result-size limit,
STORAGE gives up the cache to lose the limit.
Candidate variety checks — counting candidates per value
The candidate variety checks
on candidates_output count the candidates of each attribute value
with pl.len(), for which the aggregation keeps only the group keys in memory. A distinct
count would keep each (key, inventory_id) pair. The row count equals the distinct count
because both of these hold:
- each featurized vehicle table has one row per
(inventory_id, attribute_type, attribute_value), which its blockingunique_attribute_rowscheck enforces; - inventory and synthetic candidates share no
inventory_id.
The per-feature check counts the candidates that have both an option code and a value.
Its join pairs every code of a candidate with every value of it.
counts_per_code_and_value replaces each measured (model_id, option_code) and each
demanded (attribute_type, attribute_value) with an integer index before the join. Each
side of the join then holds an inventory_id and one integer, and the aggregation groups
by two integers rather than four strings. Codes that are not measured and values that
demand does not name are dropped before the join. The check's memory still grows with
the number of joined rows. A candidate adds its number of measured codes times its
number of demanded values.
Universal vs OEM-specific changes
Shared patterns
_snapshot_seriesusespl.struct(*cols).hash()(transformed tier)._extract_field_diffscapped at_MAX_DISCeids (transformed tier).serialize_json_fieldsmutates in place (transformed tier).- Per-Arrow-batch coalescing in
transformed_asset(transformed tier). _apply_cross_source_policiespre-indexes source rows for O(1) lookup (consolidated tier).make_raw_row_framehelper available inai_core.schema.
OEM-specific migrations
-
make_raw_row_frameadoption: only Stellantis inventory + dealers use it today. Other Stellantis raw components and all of ai_audi, ai_mercedes, ai_nissan still use the[make_raw_row(...) for ...] + pl.from_dicts(...)pattern. Migration is mechanical for sequential producers (catalog-style: build a list of records, convert to columnar lists, callmake_raw_row_frame). For threaded producers (e.g. ai_audiscs.py), per-rowfetched_atsemantics need to be either flattened ormake_raw_row_frameextended to accept a per-row list. -
Stellantis
dc_transformed.pyshares the per-row Polars frame construction problem. It iteratespartition_raw.iter_rows(named=True)directly with no Arrow batch boundary, so the same coalescing fix needs manual chunking (e.g. flush to a frame every N rows).
Deferred enhancements
The list of changes evaluated but not implemented, with rationale.
Streaming raw-tier read in transformed_asset
partition_lazy.collect_batches() in TransformedEntityComponent
iterates Arrow batches. But the underlying Polars LazyFrame.collect_batches()
materializes the full result into a list of batches before yielding,
so the entire raw partition (~5 GB at national Jeep scope) is held
during the loop. Switching to Polars streaming engine
(scan_iceberg(...).filter(...).collect(streaming=True)) could
process the filter without materializing — projected savings: ~3-5 GB
on transformed at national. Deferred because Polars streaming has
open issues around Iceberg scans and the iteration cycle (17 min per
national run) makes debugging slow. Reconsider if memory headroom
becomes critical.
Polars-expression rewrite of _apply_cross_source_policies
The function still does df.to_dicts() → Python loop → pl.from_dicts(...).
Pre-indexing source rows (Phase 5) eliminates the worst hotspot but the
to_dicts/from_dicts round-trip remains, allocating a full Python dict
list and a fresh DataFrame on output. A full rewrite as Polars
expressions over the merged frame would eliminate this. Deferred
because the consolidated tier is not in the current benchmark scenario,
no production hotspot is documented, and the rewrite is non-trivial
(handles multi-source ranking, today/historic distinction, per-row
policy decisions).
Vectorize Pydantic extraction as Polars expressions
This is the largest remaining optimization in this benchmark area. Today every OEM source module's extractor looks like this:
def extract_vehicles(response_body, request_params):
raw = data.get("vehicles") or []
return [PublicInventoryEntity.model_validate(v).model_dump() for v in raw]
For 250k vehicles per Jeep partition this is 250k model_validate
calls + 250k model_dump calls in a Python hot loop. Pydantic v2 on a
50-field model with multiple validation_alias / AliasPath mappings
and field_validator / model_validator hooks allocates substantial
transient state per record — FieldInfo lookups, ValidatorContext
objects, deeply nested coercion calls. This can allocate hundreds of
millions of objects for work that can be expressed as column operations.
Conservative estimate of a vectorized rewrite: 5-10× faster wall time on the transformed tier, 40-60% lower transient memory.
What the per-record work is actually doing (all vectorizable in Polars):
- Field renaming (
camelCase→snake_case) - Type coercion (string → float, string → int)
- Defaults / null substitution
- Computed columns (
make_cosy_image_urls,_strip_app_char, etc.) - Nested-object flattening (
price.msrp,specs.attributes→ flat columns)
The direct replacement is hard to read: dozens of pl.col(...).alias(...),
pl.col(...).cast(...), pl.when(...).then(...).otherwise(...)
expressions per entity, repeated across every OEM source module. We
lose the readability of a single Pydantic class as the entity schema.
Proposed approach — keep the class declaration, compile it to a Polars
pipeline. The Pydantic class stays as the canonical schema source of
truth (we already extract pydantic_to_polars_schema(...) from it), but
its per-record validation is replaced with a vectorized compiler that
walks the class's field metadata once at registration time and produces
a reusable Polars expression chain. Sketch:
class PublicInventoryEntity(InventoryPublicFields):
# Field types — same as today; consumed by pydantic_to_polars_schema.
# Per-field renames declared as class metadata (today's
# validation_alias) but compiled to .rename() at extraction time.
_aliases: ClassVar[dict[str, str]] = {
"lower_level_package": "llp",
"model_year_code": "assumedModelYearCode",
# ...
}
# Nested-path extractions (today's AliasPath) compiled to
# struct.field(...) chains.
_paths: ClassVar[dict[str, list[str]]] = {
"upper_level_package": ["mackevisionData", "ulp"],
# ...
}
# Pre-flatten transforms (today's @model_validator(mode="before"))
# declared as named transform classes so they can be replayed in
# a Polars context.
_transforms: ClassVar[list[Transform]] = [
FlattenStruct("price", into=["msrp", "destination", "net_price"]),
SpecsToColumns("specs.attributes", mapping=_SPEC_NAME_MAP),
# ...
]
# Per-field coercers — same set we already have, but consumed by
# the compiler instead of running per-instance.
_coerce: ClassVar[dict[str, Coercer]] = {
"msrp": Float, "horsepower": Float, "vin": StripStr, ...
}
Then a compile_extractor(EntityClass) -> Callable[[dict], pl.DataFrame]
walks the metadata once and produces a function that does the entire
extraction over a batch of dicts via Polars expressions — the same
declarative schema as Pydantic with vectorized execution.
The migration is non-trivial because:
- ~10 OEM source modules need their extractors updated.
- Some
model_validator(mode="after")hooks do non-trivial work (cosy image URL generation, ccode-to-model-code derivation) that needs careful translation. - Tests for entity validation will need to validate against the vectorized output.
Deferred until either (a) memory headroom on Serverless becomes a sustained problem despite the current mitigations, or (b) wall-time on transformed tier becomes a bottleneck. We have not seen transformed duration go red in measurements so far. The architectural pattern (declaration on the entity class, vectorized execution at extract time) preserves the class-based schema definition that makes the codebase approachable: compile the Pydantic-style declaration to Polars.
dc_transformed coalescing migration
Apply the same per-Arrow-batch coalescing as
ai_dagster/components/transformed/component.py
to Stellantis dc_transformed.py. Needs manual chunking since the file
iterates rows directly without Arrow batch boundaries. Deferred
because DC inventory is not exercised in the current benchmark scenario.
Streaming RecordBatchReader IO manager path
For raw-tier components fanning out across many sub-resources,
delivering a pa.RecordBatchReader to the IO manager would let
pyiceberg tx.append() per batch within one transaction (single Iceberg
snapshot, bounded encoder state). Deferred — current per-chunk
streaming via AiPolarsIcebergIOManager.handle_output already keeps memory
bounded. The additional gain is marginal for the schemas we have today.
Move intensive operations off Dagster Cloud Serverless
For sustained data growth, the 16 GB Serverless tier may stop being enough even with these mitigations (especially if multiple heavy assets run in parallel). Offloading the largest materializations to dedicated compute (a separate runner / GKE cluster / fargate task) is the architecture change to consider after per-process optimizations. Deferred — out of scope for this work, but worth tracking as the next architectural lever.
Adding a benchmark scenario
Drop a JSON file in benchmarks/configs/:
{
"name": "<scenario>",
"oem": "<oem>",
"description": "...",
"env": { "OPTIONAL_ENV_VAR": "value" },
"stages": [
{"name": "raw_x", "asset_key": "<oem>/raw/<source>", "label": "..."},
{"name": "transformed_x", "asset_key": "<oem>/<market>/transformed/<name>_<entity>", "label": "..."}
]
}
The optional env block sets variables on the dagster subprocess (used
by _fast variants to constrain data scope without changing source
code). Run via benchmarks/run.sh <scenario>. See the Stellantis Jeep
configs for working examples.