Skip to main content

Candidate pool

A scope's candidates are held as columns (CandidatePool), not as list[NormalizedCandidate]. The Stellantis and BMW normalizers emit those columns directly, so no object exists between the source frame and the pool. Mercedes validates per row and assembles the pool from the objects — see Construction.

NormalizedCandidate remains the boundary type, and the boundary is the recommendation: strategies.py calls materialize for the handful of candidates a round selects, and that is what services/ai_ordering receives.

Why​

At national scope the candidates are the largest thing an ordering run holds. One Pydantic model per candidate plus one per attribute, for 50,000 candidates of 18 attributes, measures 656 MB resident. A Lambda invocation was killed at Max Memory Used: 8192 MB of 8192.

What it costs​

The same candidates both ways — ids, attributes, option codes, feature ids and metadata included:

candidatesobjects (getsizeof)objects (peak RSS)poolreduction
10,00072.9 MB130.7 MB4.3 MB17.0x / 30.5x
50,000364.6 MB655.8 MB21.4 MB17.0x / 30.6x

Two instruments, because they answer different questions. A recursive sys.getsizeof is what a test can assert. Peak RSS is what the Lambda runs out of, and it is higher because getsizeof sees neither allocator overhead nor Pydantic's internals. Plan against that column. Both scale linearly.

estimated_size() reports frame buffers. It excludes the category map the encoded columns index into, which polars shares across columns and which is ~1.6 MB for a 50,000-candidate pool. Sizing that map per column overstates it several times over: Series.dtype.categories.to_series() returns the shared map, so it reports every category in the pool for a column holding one distinct value.

Structure​

One long frame per thing a candidate has many of — attributes, options, features — against scalars, which carries one row per candidate and its metadata. Long form is what makes the reduction: the repeated ids and codes dictionary-encode.

Encoding is applied where values repeat and skipped where they do not. A column with one row per candidate is all-distinct, so encoding it costs indices on top of a map holding every value. id on scalars is therefore unencoded, and metadata shares that frame rather than carrying a second copy of the ids — holding the id map once instead of per frame is worth about 2x on its own.

Option codes have their own frame rather than joining the attributes under OPTION_CODE. An OPTION_CODE restriction matches a candidate's option codes, while a candidate may separately carry an attribute of that type which it must not match. Folding them together conflates two different questions.

A candidate's feature_ids hold every feature the vehicle has, as candidates_output writes them, including features that come with an option, such as the black roof of a two-tone paint package. Its options are only the features of <oem>/<market>/output/order_form_features, the features that go on their model's order form, so a candidate's option_ids hold the codes the ordering service submits. The recommendation event therefore holds the features that featurization reads, and an evaluation that featurizes a recommended candidate gives it the attributes its take rates and references know it by.

attribute_pairs() returns text and encoded_attribute_pairs() returns the encoded frame. Only consumers that join against frames loaded from the warehouse need the text form — a Categorical-to-Utf8 join raises rather than coercing — so pool-internal work is not decoded to be iterated.

Construction​

An OEM normalizer renames its source frame to the canonical column set — id, attributes, feature_ids, model_year, plus one column per metadata key — and calls build_pool, which explodes the nested attribute column into the long frames. normalizers/columnar.py takes no column-name parameters: reconciling each source's column names is the normalizer's job, and by the time build_pool sees a frame the names are ours.

build_pool removes repeated features within each candidate's feature_ids list before it explodes the lists into the features frame. Each candidate id appears in one row of the frame that build_pool receives, so this gives the same rows as removing repeated (id, feature_id) pairs from the exploded frame. Removing them from the exploded frame hashes every pair. feature_ids holds every feature of a vehicle, so the memory of that hash grows with the number of features of every candidate in the scope.

When the table map names a candidate_attributes table, the attributes are never nested. The assembler reads the table's flat (inventory_id, attribute_type, attribute_value) rows. Inside the scan, it keeps the rows of the scope's candidates whose attribute_type is not null, and casts them to Categorical. It then passes the frame to normalize_candidate_attributes.

When candidate_attributes and inventory_candidates name the same table, the assembler scans the table once. Every OEM's table map names featurization/inventory for both. The assembler reads the rows either frame keeps and splits each streamed batch into the candidate attribute rows and the dealer's inventory rows. A table in only one entry's extra_namespaces, such as featurization/synthetic for candidate_attributes, is scanned once for that frame alone. A single collect_all over two queries does not share the scan. Their filters differ, so Polars plans two scans. The table is not sorted by model_id, so one scope's rows are spread over a large share of the table's row groups. Each scan downloads all of those row groups.

The scan reads the columns that either frame keeps or filters on, or every column when one of the two table-map entries has no select_columns. A batch holds those columns for every candidate attribute row, so the scan casts its string columns to Categorical before the batch is split. The assembler casts the dealer's inventory rows back to the table's types.

A large scope has tens of millions of these rows. Each of these steps would hold another copy of them:

  • nesting the rows and exploding them again;
  • removing duplicate rows as strings;
  • checking exclusivity with an in-memory group-by.

Together, those copies put one scope's build above the ordering worker's memory limit. This path therefore neither nests nor removes duplicate rows. Instead, normalize_candidate_attributes raises ValueError when a row appears more than once:

  • for an exclusive type, the exclusivity check finds the repeated row;
  • for a multi-valued type, a second check counts the rows of each candidate, type and value.

Both checks run on the streaming engine.

normalize_candidate_attributes sorts the rows by candidate and attribute type. In the table's file order, one candidate's rows are spread across the table. The pickled pool that an ordering activity returns then compresses several times worse.

For 50,000 candidates of 18 attributes, building the same pool both ways:

Python allocated (peak)frames held
from objects747 MB19.8 MB
from frames~019.8 MB

tracemalloc measures Python allocations, which is exactly what the object graph is. The frames themselves are Rust-side and it does not see them, so read the first column as "the object graph" and the second from estimated_size.

from_candidates stays, for the sources that validate per row: Mercedes reads model_id off each row, parses six JSON-encoded fields, and sums attribute prices into total_price.

Sharing one pool across scopes​

The take-rate bulk path normalizes once per model_id and hands every dealer scope of that model the same pool, paired with the ids that scope draws from it (SharedCandidates). subset runs on first access to candidate_pool, so a scope that is built but never scored costs its id list and nothing more.

A scored scope of 5,000 candidates holds a 1.98 MB subset pool, against one per-model object graph removed. services/ai_ordering builds one scope per invocation, so only a multi-scope caller sees the per-scope figure at all.

Scoring on the encoded attribute rows​

AttributeTurnScorer and DemandSupplyGapScorer join encoded_attribute_pairs() to their turn, demand, supply and weight tables. Before each join, dictionary_encoded casts the string columns of the table to Categorical. The scorers cast id to Utf8 only after grouping, when the frame has one row per candidate.

A large scope has tens of millions of attribute rows. Cast to Utf8, those rows take several times the memory of the encoded rows. Each join and group-by over the Utf8 rows allocates another frame of that size. The resulting peak memory exceeds the memory of a serverless ordering worker. A scorer that needs attribute strings casts them to Utf8 only after it has reduced the rows to the values it returns.