Skip to main content

Ordering Platform Design

services/ai_ordering is a Python Temporal worker that runs candidate selection in a recommend, submit, confirm loop, one candidate per iteration, per OEM. This page describes the target architecture; Decisions records why, and services/ai_ordering/CLAUDE.md marks which rules are enforced in code today.

Terminology​

  • OEM — the manufacturer we order from (Stellantis, Mercedes, Audi, BMW), and the unit of ordering. Each (oem, market) has one profile and one service class. Each has at most one private API login. BMW has none, because BMW's Ordering Hub places its orders.
  • Brand (make) — a consumer marque an OEM owns (Stellantis: Jeep, RAM, …; BMW: BMW, MINI). Many brands per OEM; brands appear inside run scoping and product data, never as a dispatch key.
  • Private dealer API — the authenticated, undocumented HTTP API of a dealer application we automate (DealerConnect, NetSTAR); the transport the injected client (packages/ai_dealer_connect, packages/ai_netstar) speaks. Sessions, logins, and auth failures are private API concepts.
  • OMS — the order management system behind the private API: the logic that validates orders, adds defaults, refuses what it will not build, and accepts the rest. Submission outcomes are OMS concepts. Cure is our name for handling that response.

Architecture​

One OEM-agnostic workflow calls per-OEM services (Strategy pattern) through a single generic activity layer. An OEM is fully defined by an OrderingOEMProfile (job config) plus its OrderingService implementation, both bound at the worker's composition root.

The ordering service interface and the factories that resolve an OEM's service live in services/ordering.py; types/ holds everything else that crosses the boundary:

ModuleContents
services/ordering.pyThe OrderingService interface; OrderingServiceFactory with live and simulated implementations
types/collection.pyCollectionJob — the OEM-declared collection jobs an ordering run triggers
types/snapshots.pySnapshotRef — key of one stored snapshot table; ResolvedSnapshot pairs one with its session timestamp
types/curing.pyCureResult — validation/curing outcome: the cured build normalized, whether it is submittable, and the target-scoped signals the pipeline records
types/target_scopes.pyMode contracts: volume fill, order modification
types/run_datasets.pyRunDatasets — the bucket and keys holding the warehouse tables a run reads once for its dealer. Names and keys only, so no row structure crosses the boundary
types/errors.pyCanonical taxonomy: SessionConflict, AuthExpired, OrderingNotPermitted, ConfigurationError, SnapshotUnavailable

Activity names are fixed. The task queue determines the run kind: one live queue and one simulated queue, each serving every OEM. A run names its OEM in its input, and the activity layer dispatches on (oem, market). Whether a run may reach a private API is fixed at deployment: the simulated queue registers no activity that reaches one, and the services composed onto it hold a mock private API client, so a simulated run cannot reach a private API whatever its input says.

Operating modes​

Two modes cover every current OEM, and a new OEM declares which it uses. "Volume" and "modification" for short:

ModePatternOEMs
Order modificationPick modifiable orders from inventory → one target per slot → modifyMercedes
Volume fillVolume optimization over projected supply → one scope per product with a volume count → placeStellantis; BMW (dry-run only)

Run kinds​

Four kinds run the same logic almost end to end. Two workflows implement them, one per queue: OrderingRunWorkflow covers the three live kinds and SimulatedOrderingRunWorkflow the simulated one. They differ in the steps below:

StepSimulatedLive dry runLive stageLive submission
Collection jobsResolved as of the run's as_of_date; never triggeredTriggered when past refresh_afterTriggered when past refresh_afterTriggered when past refresh_after
Validate and cure against the private APISkipped — the recommendation stands as selectedExecutedExecutedExecuted
SubmitSkippedSkippedThe order is left staged in the OMS, where a person submits or discards itExecuted

A stage run is available only to an OMS that stages orders before submission; today that is AccessAudi.

A simulated run reads the snapshot that was current at its as_of_date, so a backtest is a replay with a different date.

A third workflow, StagedOrderCancellationWorkflow on the live queue, removes staged orders from the OMS. Its input names the dealer, the models in scope (none covers every model the dealer holds allocation for), and whether to cancel every staged order in scope or only the ones identified as our placements; an explicit list of staged order ids narrows either. A dry run lists the staged orders it would cancel and cancels none; an execute run cancels them and each removed order becomes an attempt with status CANCELLED, the removed build in original. Started alone, it records its attempts under its own workflow id; started by an ordering run, under that run's id.

The run's cancel_staged_orders setting decides which staged orders an ordering run cancels:

  • all cancels every staged order in scope.
  • ours cancels only our placements.
  • none starts no cancellation run.

A run that omits the setting uses its OEM profile's default, which is all for Audi and none for every other OEM (ADR-017).

Scopes and targets​

A scope is the product × dealer space one selection pipeline initializes over. A target is one thing ordered: an existing order to modify, or one new order to place. construct_targets emits one mode-tagged OrderingScope per pipeline space, with its production windows as inclusive date intervals. types/target_scopes.py is the contract.

initialize_scope_pipeline runs once per scope, and the workflow drops the pipeline when the scope ends, so a run's in-memory DataFrames belong to one scope however many products it orders (ADR-004). Targets within a scope share that pipeline. RUN_LEVEL_TABLES names the tables read once per run instead of once per scope. See Data flow and the context layer.

Run flow​

Every box in the phases below is one or more activities (or child workflows) invoked by OrderingWorkflow. The workflow owns the sequence, the per-target loop, signals, and continue-as-new, and holds run state only as opaque activity output:

Each step's logic has one code home. Where it executes is the worker whose queue the activity arrives on, so selection steps run on the ordering worker as generic activities delegating into the library:

StepLogic lives in
Run sequence, per-target loop, signals, continue-as-newai_ordering/workflows (Temporal sandbox)
Collect OEM dataOEM-declared collection jobs (profile config), executed as Dagster asset materializations
Freshness check, pull inventory candidates, emit/persistGeneric activities: ai_ordering/activities, services/, persistence/
Construct targets, simulate/validate/order, report dataOrderingService impls in ai_ordering/oem/
Volume optimization (inside construct targets, volume mode)packages/ai_ppm
Pipeline init, recommendations, supply-state confirmpackages/candidate_selection — ordering only triggers
Which tables a run reads once vs per scopeRUN_LEVEL_TABLES in packages/candidate_selection; ordering decides when (once per run) because it owns the scope loop
Private API transport + parsers (collection and submit)packages/ai_dealer_connect (DealerConnect), packages/ai_netstar (NetSTAR)
Report build & deliveryai_ordering/reports, services/ (SendGrid, Slack)

The full logic those phases run, mode branching included:

Loop mechanics:

  • Pipelines are bounded at one scope. A scope's SelectionPipeline exists only while its scope loop runs, and it crosses activities through external storage (ADR-003, ADR-004).
  • A rejection adds to the current target's restrictions, and the loop retries with the next candidate. A placement updates the pipeline's supply state, and the next scoring pass derives supply take rates from that state.
  • A modification target's own vehicle is left out of supply until the target ends. Before the workflow initializes the first scope's pipeline, resolve_target_inventory_ids maps each target's order_id to the inventory_id of the consolidated/inventory row whose order_number matches it. Each scope's pipeline gets the vehicles of its own targets in exclude_inventory_ids, in addition to the vehicles the dealer's collected inventory does not list. When a target ends with a confirmed order, the confirmed build replaces the original vehicle in supply. When it ends without one and the scope has another target to fill, restore_pipeline_inventory (activity id t<target_index>-restore) adds the original vehicle back to the scope's supply. No restore runs after a scope's last target, after SupplyExhausted, or after a cancellation, because the workflow then drops the scope's pipeline.
  • A scope can end before its targets do. A submit that finds the scope's supply gone fails with SupplyExhausted. The workflow records that attempt, makes no attempt on the scope's remaining targets, and continues with the next scope. Those targets count as skipped, and the exit reason is unchanged (ADR-015).
  • The event history alone describes the loop. Every activity in one try has the activity id t<target_index>-a<attempt>-<step>, so the slot a row belongs to and whether it is a retry are visible on the timeline without decoding a payload. A refused build fails validate_and_cure with a non-retryable error whose details hold the CureResult, which the workflow reads back. The candidates the OMS refused are therefore the failing rows, and a target that needed four tries is one target with four attempts.
  • Cluster-shared supply state is deferred (ADR-012). A run's loop reads only its own confirms.

How a run ends​

A run that reaches the end of its targets returns an OrderingResult whose exit_reason names the outcome: QUOTA_EXHAUSTED, NO_CANDIDATES, DRY_RUN_COMPLETE or CANCELLED. A simulated run that finds no usable collection session also returns, with SNAPSHOT_TOO_STALE and status SKIPPED. Like a failing run, it persists its state and closes its event stream, but it notifies no one.

Every other ending fails the workflow. Before failing, the run persists its state, notifies Slack, and closes its event stream. A run with a reused run_id only notifies Slack before failing (Run IDs). It sends no report, because a run that stopped early has no complete allocation to report on. Two cases:

EndingFailure the workflow raises
A reason the run itself decided: expired session, a live run's stale collection session, no completed partition, a pause nobody liftedNon-retryable ApplicationError typed by the exit_reason, with the OrderingResult in its details
Any other activity failureThe activity's own error, unchanged

A caller therefore reads a stopped run's outcome from the failure.

A run records its exit_reason on its ordering_runs document when it ends. A run with a reused run_id writes no document.

Externally triggered runs​

Some OEMs run their own order management and place their own orders. For those, a run produces validated order templates and never submits, and the OEM's system asks for them over HTTP rather than the run being scheduled here. The OEM system holds no warehouse credentials and reads no Iceberg or BigQuery table: the API is the whole boundary.

Three calls make up the contract:

CallEffect
Request templatesValidates the volumes, starts the batch workflow, returns the request's id and URL
Get the requestPer-model status and counts while it runs; its templates once they are ready
Report order statusWhat the OEM system placed, keyed on the template that recommended it

The contract is named for what the caller asked for rather than for the run that serves it, so the fan-out above can change without the contract changing.

One run per model, every dealer in it. Volume arbitrage needs a model's volumes across all dealers before it can split control from test, and national take-rate steering scores a candidate against counts accumulated across the whole model. Neither is answerable one dealer at a time, so the model is the unit the batch fans out on and the dealer is a scope inside one run. This makes a model's run larger than a single-dealer run, and the pipeline's per-scope bound (ADR-004) applies to the dealer scope within it, not to the model.

Results are asynchronous because a model's run takes minutes. The request call returns as soon as the batch starts; nothing in the contract waits on selection.

Placement outcomes return through the same API. The OEM system reports what it placed, and that report reaches the warehouse in one agreed format, so measurement reads placements the same way whether we placed them or the OEM did.

Data flow​

An ordering run never pulls a private API for collected data. It resolves one collection_id per table of each declared collection job (CollectionJob.tables) and reads that snapshot. When the snapshot is stale, a live run requests the job's partition from Dagster and waits for the materialization. The verdict table below gives the rule. Collection jobs are Dagster assets, and snapshots are written to Iceberg keyed (dealer, table, date, collection_id), every collection of a day appended to its partition (ADR-016). A source whose collector has not migrated still writes BigQuery, and the store selects the backend per source. A live run's target construction enumerates the dealer's configurable models from the private API (get_eligible_model_ids); a simulated run reads that set from the warehouse instead. The private API is used again at validate and submit.

The warehouse probe checks presence: it resolves the newest date, at or before the run's as_of_date and within partition_lookback_days, on which every table the run reads has completed. A brand's ingestion completes at some point within the UTC day it covers, so a run starting before that reads the day before instead of terminating. No date inside the window terminates the run, live or simulated. A date is taken whole. The resolved date replaces the run's as_of_date before any activity that reads data is dispatched, so one date covers the partition, the production window and the collection reference time of a simulated run. The run records the date it requested as requested_as_of_date and the resolved date as partition_date on its ordering_runs document. The dependency list is derived from the IcebergTableMap the selection config declares plus the service's volume_optimization_reads, so a table added to a read is covered without a second list.

Before it resolves a date, the probe checks that each table's current Iceberg schema contains every column that the run selects from the table. The selected columns are the select_columns and scope_filters of each IcebergTableRef, plus the columns that volume_optimization_reads lists. If a column is missing, the probe stops the run with WAREHOUSE_COLUMN_MISSING, and the error names each table and each missing column. The check reads only table metadata.

Warehouse reads happen at two levels. Most tables are read per scope, where the partition is worth scanning for one model's rows. The tables in candidate_selection.RUN_LEVEL_TABLES hold every dealer and model of the market in one partition, so a scope's predicate narrows the rows returned while the bytes scanned stay the same. load_run_dealer_datasets reads those once for the dealer at the start of the ordering loop, and the workflow passes the result to every scope, which narrows it in memory. The scan happens once per run instead of once per model. For the same reason, load_run_dealer_datasets computes the dealer's days-in-status once from the trailing partitions of inventory, and every scope reads that frame.

RunDatasets crosses activities as a bucket and keys. The tables stay in the cache bucket, and a scope fetches the whole-table payloads plus its own model's slices. It has no row model, so SnapshotRef remains the only row reference between activities.

CollectionJob.assess_freshness is a pure, sandbox-safe function the workflow computes inline against its reference time: its own clock for a live run, and for a simulated run the end of its as_of_date or the run's start, whichever is earlier. Resolution therefore stays one activity that returns data, and the trigger-or-stop branching is workflow control flow:

VerdictAgeLive runSimulated run
fresh≤ refresh_afterreusereuse
stale≤ max_staleness (live) or simulated_max_staleness (simulated)trigger the jobreuse
expired> that limittrigger the jobskip
missingno completed sessiontrigger the jobskip

A skipped simulated run completes with status SKIPPED and exit reason SNAPSHOT_TOO_STALE. It is not a failure, because the run only reads what was collected. A live run that still lacks a session after collecting fails.

A dealer with no completed session resolves to None, and absence is a freshness input. The store raises only for faults: ConfigurationError for a logical table the source does not configure, SnapshotUnavailable for a ref naming a session its table never recorded.

While the TypeScript collectors write the snapshots, a session's identity is its fetch_date: one timestamp written on every row a session writes to a checked table. SnapshotStore.resolve derives collection_id from that timestamp, and reads translate it back into a fetch_date filter. A migrated collector writes its Dagster run id as the collection_id on every row it appends, and reads filter on it. A table without a per-session timestamp cannot be checked for freshness. feature_catalog has one per combo and sales_* one per row. SnapshotRefs are the only snapshot reference in activity inputs and outputs, and rows are read inside activities.

Mercedes collects its inventory through a CollectionJob like Stellantis and Audi: MERCEDES_US declares the mercedesUsInventoryWorkflow TypeScript workflow, and a session's timestamp is insertedAt. resolve counts a session only when it has at least SnapshotSource.min_session_rows rows: 2 for Mercedes, 1 for every other source. The legacy ordering runner writes each order it modifies back to the Mercedes inventory table as a one-row session with its own insertedAt, and that row is not a fetch of the dealer's inventory. So a dealer whose whole inventory is one order has no session that counts.

Reads go through services/snapshots.py, a single generic SnapshotStore with resolve and a typed read_rows(ref, row_model). Row structures are OEM-internal: each OEM defines its own row models beside its service and binds them where it calls read_rows. The ref is the only cross-OEM snapshot contract.

An OEM service never calls resolve. In Step 0, the ordering workflow resolves one session per snapshot table through the resolve_snapshots activity. It sets the refs on RunContext.snapshots and records them on the run's ordering_runs document. An OEM service reads the sessions those refs name, so every activity of an ordering run reads the session the run recorded.

A staged-order cancellation that an ordering run starts receives the run's refs. A cancellation started on its own resolves its sessions up to the end of its partition date. It triggers no collection.

Run state​

StateWhere it lives
Pulled snapshots (inventory — orders included)BigQuery (dealer, table, date, collection_id); read via SnapshotStore
Warehouse reference data (models, dealers, stats)Iceberg/Nessie via services/warehouse.py, read at the run's partition
OEM payload between a service's own activitiesTemporal External Storage — typed activity args/returns; over-threshold payloads offload to S3
Live SelectionPipelinePassed between the selection activities, offloaded to Temporal External Storage like the OEM payload. The workflow passes it as an opaque RawValue and never decodes one, because deserializing it with pickle imports polars, which the workflow sandbox forbids
Cluster-shared supply state (future)candidate_selection library — the durable extension of the supply state confirm_order already mutates
Runs, attempts, events, restriction evidence, volume decisions, report buildsMongoDB; events sync to Iceberg, the volume decision included through its own event

Run IDs​

A run's run_id is its Temporal workflow ID. Each run must have its own run_id. A child workflow of a simulated run with several dealers takes the run_id of its parent, as A simulated run with several dealers describes.

ordering_runs holds one document per dealer of a run. Its _id is {run_id}:{dealer_id}, and its run_id and dealer_id fields name the run and the dealer. volume_optimizations uses the same value as its _id, and ordering_run_cache stores it in its run_id field. Readers select ordering_runs documents by the run_id and dealer_id fields and never parse _id.

Documents written before ordering_runs had a run_id field have no such field, and their _id is the run id.

Each ordering_runs document records the first_execution_run_id of the run that wrote it. A run fails with RUN_ID_REUSED when the document for its run_id and dealer has a different value. A reset, a workflow retry and continue-as-new keep the value, so they continue the same run.

Dagster starts runs with WorkflowIDReusePolicy.REJECT_DUPLICATE, so Temporal rejects a reused workflow ID before the run starts.

A simulated run with several dealers​

Dagster starts SimulatedOrderingRunWorkflow for a partition with every dealer that the Dagster op selects in dealer_ids. Simulated runs describes the workflow ID and how the op selects the dealers. Started with one dealer, the workflow runs that dealer itself. Started with several, it:

  1. checks ordering_run_groups for another execution of its run_id, and writes its document there;
  2. runs probe_warehouse once;
  3. starts one child workflow of its own type per dealer, with the ID {run_id}-{dealer_id}, one dealer in dealer_ids and the resolved partition as parent_partition_date;
  4. records each child's outcome on its ordering_run_groups document.

A child takes its run_id from its parent, so the children write every dealer's ordering_runs document, attempts and events under one run_id. A child reads the partition its parent resolved and does not probe the warehouse itself. Its as_of_date is the date the run was requested for, so its ordering_runs document records the requested date and the partition read, as a run with one dealer does. A failed child is recorded as FAILED and does not fail the run. If at least one child failed and none finished, the run ends FAILED.

RunConfig.child_runs sets these limits:

  • max_running is the most children that run at once. By default every child starts at once.
  • deadline_hours, 3 by default, is the number of hours after the run starts at which the run signals cancel_ordering to every child still running. The run then waits CHILD_CANCEL_GRACE for those children to end. It cancels each child still running after that through Temporal, and records the child as UNFINISHED.

The run forwards pause_ordering, resume_ordering and cancel_ordering to its running children. It starts no child while it is paused or cancelled. The live OrderingRunWorkflow accepts one dealer only.

Target layout​

services/ai_ordering/src/ai_ordering/
├── worker.py # composition root: profile → services → activities → queue
├── profiles.py # OrderingOEMProfile per OEM (job config only)
├── queues.py # the live and simulated ordering queues
├── types/ # the entire boundary: everything that crosses it
├── activities/ # generic Temporal layer — only temporalio importer
├── workflows/ # sandbox loop internals
├── oem/ # per-OEM OrderingService implementations + their private API clients
├── services/ # storage + integrations: warehouse, snapshots, sendgrid, …
├── persistence/ # Mongo: runs, attempts, events, oem_restrictions, volume, reports
└── state/ # what a run must hold in memory, and nothing else

Restrictions​

There are two kinds. Both treat a live restriction as evidence a later attempt can confirm or overturn:

  • Static: known before the run — oem_restrictions (transcribed by operators today and fed by ingestion later) and the dealer's dealer_rules. Both key a rule by base_model_id.
    • The construct_targets activity adds oem_restrictions and the dealer's OPTION_CODE rules to every OEM's scopes after the OEM builds them. Candidate selection prunes on these rules. For each scope, the activity:
      1. looks up each model's base_model_id in consolidated/models
      2. reads that base model's rules through services/restrictions.py
      3. appends them with OrderingScope.with_added_restrictions
    • Mercedes reads the dealer's ACCESSORY rules in validate_and_cure and applies them while it edits the order. It removes must-not-have accessories and adds must-have ones. These rules never reach candidate selection.
  • Discovered: found when a submit is rejected, and slot-scoped today. Mercedes restrictions depend on date and threshold, so moving them into shared scope needs a per-OEM policy for scope, TTL and confidence. Moving them too early suppresses the attempts that would prove or disprove the restriction.

Deployment and scaling​

One codebase, one image, one ECS service definition, instantiated per (OEM, run kind), with each instance's queue and OEM set by configuration. Ordering services scale between zero and one with the run trigger, which already knows work is coming. Collection workers scale on task-queue backlog. ResourceBasedSlotSupplier caps per-worker concurrency by memory and CPU. desired_count > 1 depends on durable run state (ADR-003). Scaling is per queue, so per-OEM scaling requires per-OEM queues.