Skip to main content

Ordering Platform — Architecture Decision Records

Settled in the August 2026 design review. The scoped services/ai_ordering/CLAUDE.md tracks which are enforced in code today.

ADRDecision
001Service interfaces over activity polymorphism
002Collection is an OEM-declared job set, executed by collection workers
003Registries removed in favor of durable run state
004Selection pipelines initialize per scope, at scope entry
005BigQuery snapshots keyed (dealer, table, date, collection_id); freshness-checks runs
006Orderable candidates: scored candidate in, OMS values out, auditable
007Three-state order reporting; cured results feed the candidate pool
008Internal model_id is product identity
009OrderingInput.extras becomes a typed RunConfig
010OEM profile is pure job config; shared contracts stay closed
011Config selects code paths only at the composition root
012One active ordering job per supply cluster; supply ≠ demand clusters
013All collectors run in the Python runner
014One image; one ECS service per (OEM, run kind); trigger-driven scaling
015A scope-level stop: SupplyExhausted ends the scope and lets the run continue

ADR-001: Service interfaces over activity polymorphism​

Status: Accepted (2026-08-05); amended (2026-08-12) — one service class per OEM serves every run flavor; the injected private API client is the only difference

Context​

Per-OEM behavior was dispatched by registering per-OEM activity implementations under shared activity names, selected by task queue. The OemOrderingActivities Protocol was documentation-only: nothing bound implementations to it. The contract drifted — it grew a fifth activity without the Protocol changing. OEM logic was tightly coupled to Temporal, unusable from other contexts (daily amendment, BMW template generation).

Decision​

A Temporal-free OrderingService interface in services/ordering.py, implemented per OEM (Strategy pattern). One generic OrderingActivities layer — the only temporalio importer — resolves the run's service from the OrderingServiceFactory injected at the worker's composition root. The live and simulated factory classes are the two compositions. Conformance is mypy-checked. One service class per OEM serves every run kind — simulated, live validation, live submission execute the same code. One live queue and one simulated queue each serve every OEM, so worker count stays constant as OEMs are added. A run names its OEM in its input, and the activity layer dispatches on (oem, market). Simulated isolation stays structural: the simulated workflow calls no private-API-touching activity, its queue registers none, and the services composed onto it hold a mock private API client. The workflow keeps the per-target loop (each iteration is a durability checkpoint); no generic orchestrator service re-implements it.

Consequences​

Adding an OEM = one profile + one service implementation + one queue registration. Services are reusable outside the durable loop. Non-durable callers (BMW template generation) compose the same services into their own loops.

ADR-002: Collection is an OEM-declared job set, executed by collection workers​

Status: Accepted (2026-08-05); amended (2026-08-12) — the collection boundary is a OEM-declared job list plus a workflow contract, not a fixed set of methods; amended (2026-08-14) — collection_id names the completed session: minted from fetch_date at read today, stamped as the child workflow's id at write after ADR-013, and resolved per freshness-checked table until then

Context​

The original design folded data pulling into the ordering service as a pull_data method. But collectors do the same pulls with no ordering involved. One shared method would leave cadence collection unable to reuse it, and it would leave ordering workers carrying private-API-pull code they never run.

Decision​

Collection is separate from ordering, and it is configuration, not a fixed contract: each OEM declares the collection jobs an ordering run requires (CollectionJob — workflow type, task queue, staleness quantities — on its profile). An OEM runs many collection jobs. Which subset an ordering run requires differs OEM by OEM, so no fixed per-data-kind methods exist. The contract is what any job must satisfy: startable with the run's user and dealer, and writing every freshness-checked table's rows as one identifiable session. collection_id names that session and is opaque to consumers. While the TypeScript collectors write snapshots, it is minted at resolve from the session's fetch_date (constant per session within a freshness-checked table). Once collectors run in Python (ADR-013), the triggering workflow mints the child's workflow id and stamps it at write. Either way, the parent constructs snapshot refs without a collection return value. Session identity is per table until the write-stamped id spans several, so an ordering run resolves one collection_id per freshness-checked table of each declared job (CollectionJob.tables). A job may write more tables than it freshness-checks. OrderingService (snapshot resolution, targets, mapping, validate/cure, submit, report data) remains the ordering interface. Collection jobs execute on the collection workers — cadence and run-triggered alike. Resolution is freshness-checked against two per-job staleness quantities:

  • data younger than refresh_after is reused
  • past it, a live run triggers the job as a child workflow
  • data older than max_staleness blocks the run

The max_staleness check is the only one a simulated run applies, since a simulated run never triggers and always resolves to the most recent data. Ordering only reads the stored snapshot. There is no monolithic "pull data" step or file: activities read what they need through the common storage interfaces.

Consequences​

Ordering workers host only OrderingService; collection workers only collection jobs — the capability split falls out of the queue split. Normalization from OEM-native structures happens in each OEM's collection jobs, written once per OEM (see ADR-013 for when it moves to write time). The job contract is identical for the TypeScript collectors and their Python ports, so the config list works across the migration.

ADR-003: Registries removed in favor of durable run state​

Status: Accepted (2026-08-05)

Context​

Three module-global dict[str, Any] registries (pipeline_registry, oem_payload_registry, target_inputs_registry) held run state as the only copy: untyped, importable from anywhere, and not fault tolerant. A worker restart orphaned the run, pinning desired_count=1. Retyping them into a generic RunStore[T] was considered and rejected: a general "stash anything by run_id" facility allows the same misuse in typed form.

Decision​

The registry concept is removed outright. Each state category gets a durable home: snapshots in BigQuery, reference data in the warehouse, and OEM payloads plus the live selection pipelines via Temporal External Storage.

Selection pipelines are passed between activities rather than cached:

  • initialize_pipeline returns them
  • the selection activities take them
  • confirm_pipeline_order returns them, so the supply state it advanced reaches the next iteration

The workflow owns them and shuttles them as an opaque RawValue. It never decodes one, because unpickling a pipeline imports polars, whose module body reads os.environ. The workflow sandbox forbids that. The activities on either end are unsandboxed and do the decoding.

Passing them was chosen over a PipelineCache that rebuilds from durable state on a miss. The cache avoids moving the object but needs a rebuild path and a confirmed-order replay whose cost grows with every order a run places. Passing needs neither. The payload it moves is bounded by how large a pipeline is — which ADR-004 is what actually governs.

Consequences​

Any worker can serve any activity, so nothing pins a run to the worker that built its pipelines. Per-iteration cost scales with pipeline size: a scope map spanning every scope upfront measured ~414 MB, so the pipelines have to shrink to one live scope (ADR-004) before this fits. Nothing worker-local is left: pull_data returns the frames construct_targets computes over. These frames are packaged as a TargetInputs inside its result and pickled behind one field, so external storage carries them the same way. That is a holdover: ADR-005's snapshots remove the hop rather than move it. But it is what unblocks desired_count > 1.

ADR-004: Selection pipelines initialize per scope, at scope entry​

Status: Accepted (2026-08-05); amended (2026-08-12) — initialization at scope entry, not first recommendation

Context​

initialize_pipeline built every scope's pipeline upfront — all models' frames resident simultaneously, with a transient Arrow double-buffer at load. The 2026-08-04 allocation run OOM'd an 8 GB worker on a nine-order dealer.

Decision​

A scope's pipeline is built at scope entry and evicted when that scope's targets are done. Pipeline state crosses activities as typed inputs and outputs (ADR-003), so deferring the build to the first recommendation buys nothing — the workflow already knows which scope it enters next. Stellantis targets are emitted contiguously per model, so the working set is one scope. Mercedes slot targets may interleave, handled by sorting targets by scope (the workflow controls target order).

Consequences​

Memory is bounded at ~one scope regardless of how many models a dealer orders. Eviction is safe because the state the loop needs next is whatever the last activity returned (ADR-003) — dropping a finished scope discards nothing a later step reads.

ADR-005: BigQuery snapshots keyed (dealer, table, date, collection_id); freshness-checks runs​

Status: Accepted (2026-08-05); amended (2026-08-14) — reads are resolve plus a generic typed read_rows; row structures are OEM-internal, and the ref is the only cross-OEM snapshot contract

Context​

Cadence collectors write snapshots to BigQuery; Dagster ingests them to Nessie/Iceberg. Ordering needs point-in-time-fresh OEM data at run start, and simulated and live runs must read the same source. Writing Iceberg directly from collectors (with Dagster external-asset reporting via report_asset_materialization) was considered and kept as the named later option — revisit when the lake becomes the primary consumer of these snapshots.

Decision​

BigQuery remains the shared snapshot store. An ordering run triggers collection as a child workflow, started with the run's user + dealer — the same captured session ordering uses. Snapshots are keyed (dealer, table, date, collection_id). collection_id names the completed session: today it is minted from its fetch_date at read; after ADR-013 ships, the child's workflow id is stamped at write instead. Either way, an intra-day re-pull never overwrites what an earlier run read, and simulation can replay exactly what a run saw. Reads go through one generic SnapshotStore (resolve plus read_rows(ref, row_model) — typed models, never dicts). Each OEM binds its own row models at the callsite, and rows never cross an activity boundary. Snapshot freshness is an orchestration prerequisite: a run that cannot collect must not order.

Consequences​

One collection code path serves cadence and run-triggered pulls. The run inherits collection's SLA and failure modes — accepted deliberately. No per-OEM read strategy above the interface: normalization is each OEM's row-model binding (at read until ADR-013 ships, at write after).

ADR-006: Orderable candidates — scored candidate in, OMS values out, auditable​

Status: Accepted (2026-08-05); amended (2026-08-12) — the mapping input is the scored candidate, which the orderable carries by composition; amended (2026-08-17) — only the base crosses an activity boundary, and each activity derives the OEM representation itself; amended (2026-08-23) — the OEM representation is service-internal values, not a subclass of the boundary type; amended (2026-08-24) — the boundary type is SelectedCandidate, and the layering is gated in CI; amended (2026-08-26) — the boundary type is NormalizedCandidate itself and the partition pin moved to RunContext

Context​

The candidate → order-format mapping was scattered through candidate_mapping.py and the wizard walk, and the workflow serialized full candidates + recommendations into every activity call. Everything internal should reference canonical ids; OEM codes belong only at the ordering boundary.

Decision​

The workflow hands the ordering service the scored candidate exactly as selection emitted it. Recommendation output already rides external storage, so nothing new crosses event history. (candidate.id, RunContext.partition_date) pins the mapping deterministically for replay. The partition is a run-level constant, not a per-candidate one: the warehouse probe takes a partition whole or not at all, so every candidate of a run was selected from the same date. The service resolves the private-API-facing values from it: OEM codes derived from the candidate's internal ids, with no repeated warehouse lookups for fields selection already resolved. These are plain values, consumed where they are computed, not carried as a type. Each brand does this inline, inside its own service under oem/: no per-brand record holds the mapping, and it is never lifted into types/, whose models are the cross-OEM contract. Brands share no portal fields — Stellantis takes model_code plus wire codes, Mercedes baumuster plus national_type — so a shared structure would only ever be a union of things no single brand wants. The mapping is auditable either way: the exact attempted representation is persisted per attempt. Whether orderable candidates are precomputed/materialized or generated at order time is the one open sub-question (Q2), decided within that work item.

No OEM-specific structure crosses an activity boundary. Activity signatures annotate NormalizedCandidate — the candidate exactly as selection emitted it. The data converter decodes to the annotation, so anything the annotation does not declare is dropped, silently, because the model ignores unknown keys. Each activity therefore resolves the OMS values it needs on entry, from the candidate it was given. Mapping is a function the service calls, not a step whose output is carried. NormalizedCandidate is not subclassed, so there is no per-OEM structure for a service to receive or assert on.

The layering that makes this hold is enforced rather than trusted: tests/test_domain_layer_boundary.py proves oem/, services/ and types/ import without temporalio, and that no module under oem/ imports ai_ordering.activities or ai_ordering.workflows.

Consequences​

Event history carries claim-check references, never inline candidates. Per-OEM mapping stays inside the service (shared contract models remain closed). Reporting and cluster work build on this contract.

Mapping runs once per activity that needs it rather than once per candidate, so it must stay cheap — for Stellantis it is one consolidated/models read and a sort. The factory hands one service instance to every run a worker takes, so anything a mapping holds between calls is keyed by what it was read for: Stellantis holds the trim catalog against its partition date, and Mercedes memoizes catalog metadata behind its own api module.

The OEM's own wire representation reaches reporting through the per-attempt record, not through CureResult, whose candidate is the cured build normalized so it can re-enter the selection pool (ADR-007).

ADR-007: Three-state order reporting; cured results feed the candidate pool​

Status: Accepted (2026-08-05)

Context​

OEM systems cure, mutate, or add defaults to submitted candidates:

  • BMW's BYO validation returns what is effectively a new candidate (new hash)
  • Mercedes cures mid-flow
  • Stellantis parks non-conforming orders in BG/BX status

A derived diff alone is unanalyzable when the diff logic is wrong, and per-option pricing makes diffs large.

Decision​

Persist three raw builds in one overlayable schema per attempt:

  • original — the build on the existing order, for modify flows
  • cured — the build as the OEM's configurator accepted it for submission
  • final — the build the OEM kept

Diffs are derived for analysis, never the only record. A cured result may be a net-new candidate: its id is recomputed from what came back (order ids are many-to-one to candidates) and it feeds back into the candidate pool via the next collection poll. Mutation-by-restriction must remain distinguishable from dealer modification.

Consequences​

Reporting reflects truth, not intent. Which discrepancies are actionable (alerts/workflows) versus analytics-only remains open (Q5), as is the persistence target: run reporting (attempts, volume decisions, report builds) writes to Mongo today, with a standardized BigQuery stream the proposed successor — the volume-decision write moves with that answer.

ADR-008: Internal model_id is product identity​

Status: Accepted (2026-08-05)

Context​

Volume optimization manufactured a composite product_id = model_code|trim_identifier from internal model_id, only for ordering to decompose it back for the private API order wizard. The format declaration (ProductIdFormat) hung off iceberg_tables — an identity contract coupled to a storage backend choice. group_cols was declared in two homes with nothing keeping them in agreement.

Decision​

SelectionScope.product_id becomes internal model_id (year-specific — you order a specific year's car). Loaders filter on it directly. ProductIdFormat, compose_product_id, and parse_model_code are deleted. OEM codes (model_code, trim, Baumuster) live on the orderable candidate and cross-references only. group_cols is deleted from both homes — base_model_id is the hardcoded statistical granularity, because the per-OEM judgment is already absorbed in its derivation at ingestion.

Consequences​

Sometimes the right fix for a drifting contract is deleting the contract. The grouping knob returns only with a named use case.

ADR-009: OrderingInput.extras becomes a typed RunConfig​

Status: Accepted (2026-08-05); amended (2026-08-13) — the field itself is typed, and the workflows register ValidationError as a failure exception type; allocation_source leaves the run-input channel

Context​

extras: dict[str, Any] was a stringly-typed side channel. Per-block Pydantic parsing validated known blocks but silently ignored unknown top-level keys. Full typing was deferred to avoid requiring a matching change in Job Freezer (TypeScript), which constructs the input.

Decision​

OrderingInput.extras is one frozen, fully typed RunConfig: every block a declared field, extra="forbid" at the top level so an unknown block is rejected instead of ignored. The field itself is typed, so Temporal's converter validates the config while decoding the input — a bad config fails the run at its first workflow task, before any activity runs and well before an order is placed.

It fails the run rather than the workflow task only because both ordering workflows declare failure_exception_types=[ValidationError]. Without that, _convert_payloads reports the rejection as Failed decoding arguments and Temporal retries the workflow task forever: the run never fails, never ends, and orders nothing. Do not remove that registration. What it costs is that any ValidationError raised inside these workflows now fails the run, rather than retrying the task until a fix is deployed.

RunConfig holds only settings a run may choose (ADR-011): run_summary, volume_optimization, and the optional volume_fill. allocation_source — DealerConnect's market letter and ship-to dealer — is a per-OEM fact rather than a run choice, so ADR-010's field-variance rule puts it inside the adapter, OEM-private. Job Freezer needs no coordinated change; it never set extras.

Consequences​

A run is reproducible from its input alone, and a typo in its config cannot quietly run at defaults. extras is now a closed schema on the wire, so removing or renaming a block breaks decoding of any history that carries it — a completed run being replayed and an in-flight run's continuation alike. Treat a block removal as a versioning change, not a refactor. Every block is declared for every OEM even when only one reads it, so a run can set a block that nothing reads. Adding per-OEM blocks means changing what OrderingInput.extras is annotated as — the converter validates against that annotation, so a subclass carrying extra blocks is rejected by the base's extra="forbid". A discriminated union keyed on the OEM is the pattern that works.

ADR-010: OEM profile is pure job config; shared contracts stay closed​

Status: Accepted (2026-08-05)

Context​

Per-OEM facts were scattered:

  • product_id_format inside candidate_selection's storage config (fetched by oem-string file lookup into package internals)
  • order_label as a module constant
  • grouping declared twice

Meanwhile per-OEM data structures push shared models toward optional fields (trim: str | None) or per-OEM subclassing — both rejected (Temporal's converter round-trips declared types, and isinstance(target, StellantisTarget) in shared code is the coupling being removed).

Decision​

OrderingOEMProfile carries only genuine per-OEM choices: oem, market, operating_mode, order_label, collection_jobs — read once at the composition root. Everything downstream receives typed values, never file paths or oem strings. The profile is deliberately interim: the long-term direction is hard ingestion contracts (~95% standardized schema + canonical internal entity IDs with cross-references), and profile fields are removed as those ship. Field variance resolves by one question: does shared code need to understand the field?

  • OEM-private → typed models in the service
  • carried-but-unread → opaque slots
  • shared-math dimensions only some OEMs have → semantic attribute rows
  • required-everywhere → canonical typed fields (a dimension present for every OEM becomes canonical)

Consequences​

"Does OEM X have trim?" is never asked in code. Adding an OEM adds a profile, not fields on shared models.

ADR-011: Config selects code paths only at the composition root​

Status: Accepted (2026-08-05)

Context​

"Should volume optimization be a config flag?" generalizes to: where may configuration change behavior? A runtime flag in shared code (e.g. volume_optimization_enabled) is a branch every run re-decides from mutable data — the isolation-weakening move the simulated-queue design exists to prevent.

Decision​

Three buckets, strictly:

  • deployment decides capability (which services a worker injects — live, simulated, volume-optimizing)
  • the OEM profile declares per-OEM facts
  • run input (RunConfig) scopes a run (brands to allocate, thresholds, recipients)

Config may drive code selection only at worker startup — the composition root reads operating_mode and binds the right construct_targets — never mid-run. Mid-run, config says what and how much, never which code runs.

Consequences​

A worker either has a capability or doesn't. Behavior cannot be flipped per-run by a typo'd flag. If operators ever genuinely need per-run volume-optimization toggling (open Q3), it becomes an explicit run-input selection among pre-wired strategies.

ADR-012: One active ordering job per supply cluster; supply ≠ demand clusters​

Status: Accepted (2026-08-05); per-iteration shared-supply refresh deferred (2026-08-06) — current supply formulas give nearby-dealer supply negligible weight. The shared-state refresh and (dealer, candidate) write-back activate when that weighting requires realtime granularity. The run loop's structure does not change when they do.

Context​

Per-dealer ordering is locally optimal and collectively wrong: whole cars can't be fractionally allocated, and every subdivision amplifies rounding error. Two dealers in one cluster starting concurrently would read the same blank supply state and place identical top-scored orders. The existing "smart clusters" are buyer-similarity demand clusters — not geographically aligned, and not reusable for supply coordination without validation.

Decision​

Coordination happens at the smallest impactful supply cluster (a new, empirically derived concept). Concurrency control is a job-level lock — one active ordering job per supply cluster — not distributed locking in the data layer. Shared cluster supply state records which dealer ordered which candidate (neighbor supply weighs differently than own-store) and is re-read every loop iteration, never only at job start. Its durable store lives with candidate_selection (the extension of the supply state confirm_order already mutates). Restriction evidence stays in ordering persistence. Ordering keeps a multi-dealer-capable dealer_ids input even while executing per dealer.

Consequences​

Order-sequencing bias is mitigated by daily dry runs estimating what the rest of the cluster would order. Cluster definition and cluster-level take rates are upstream work. Stellantis cross-dealer allocation additionally requires OEM authorization. The trigger/windowing mechanism is open (Q4).

ADR-013: All collectors run in the Python runner​

Status: Accepted (2026-08-05)

Context​

The same private API parsers existed twice — TypeScript cadence collectors in ppm-ordering-services and Python in ai_ordering. Giving transport clients + parsers a home of their own — one package per private API, packages/ai_dealer_connect and packages/ai_netstar — removed the root cause.

Decision​

Effective immediately, all new collectors are written in the Python runner on those packages. The existing TypeScript collectors are migrated per source (DealerConnect, NetSTAR), keep the same BigQuery destination, and are then decommissioned. From that point the collection jobs' normalization runs at write, retiring the read-side shims behind SnapshotStore.

Consequences​

One pull implementation per OEM serves cadence collection, run-triggered collection, and ordering. Until the migration completes, the TS collectors write raw structures and normalization runs at read.

ADR-014: One image; one ECS service per (OEM, run kind); trigger-driven scaling​

Status: Accepted (2026-08-05)

Context​

Ordering runs a few days per month per OEM. Collectors run daily. Today a single always-on worker process (fixed desired_count=1) hosts every queue. Temporal polling is not filterable by input, so scaling can never be finer than routing — per-OEM scaling requires per-OEM queues.

Decision​

One codebase, one image, one ECS service definition — instantiated per (OEM, run kind) with config selecting queue and OEM. Per-OEM queues stay (routing = the scaling and isolation granularity). Ordering services scale 0↔1 with the run trigger — the trigger already knows work is coming, no backlog polling. Collection workers scale on task-queue backlog metrics. ResourceBasedSlotSupplier caps per-worker concurrency by memory/CPU. Simulated runs get their own queue + deployment so "cannot order" stays structural.

Consequences​

OEMs stay at zero outside their windows. desired_count > 1 per OEM is conditioned on ADR-003. Scale-down must never terminate in-flight work (the risk KEDA scale-to-zero carries, avoided by trigger-driven rather than backlog-driven scaling for ordering).

ADR-015: A scope-level stop: SupplyExhausted ends the scope and lets the run continue​

Status: Accepted (2026-09-03)

Context​

A submit has two ways out. An OrderingError escapes the activity as a non-retryable failure and the run fails. Anything else comes back as a FAILURE attempt, and the loop asks for the next candidate. Neither fits a scope whose supply is gone mid-run — an allocation remainder that reached zero because another order consumed it. Failing the run discards every other scope; a FAILURE attempt retries the same candidate until max_retries_per_target is spent, then the scope's next slot repeats that, so the run spends max_retries_per_target × remaining slots OMS calls, each reading a remainder of zero.

Decision​

SupplyExhausted(OrderingError) in types/errors.py means the scope's supply is gone. A service raises it where its OMS reports that fact — Audi from submit, when validateRemainingQuantity answers less than the quantity requested. The workflow catches the failure in _fill_target, records the attempt as FAILURE with error.type="SupplyExhausted", makes no attempt on the scope's remaining targets, and continues with the next scope. Those targets count as skipped. OrderingExitReason is unchanged; the ended scope's SupplyExhausted attempt is the record of the stop.

Consequences​

The workflow keys on the canonical type name, never on a service's error string. Only submit reads the remainder, so only an execute run raises the stop; _cure turns every activity failure into a refused cure. The stop is per scope: a carline whose allocation is gone leaves the run's other carlines to fill.