Ordering Platform Design
services/ai_ordering is a Python Temporal worker that runs candidate selection in a recommend,
submit, confirm loop, one candidate per iteration, per OEM. This page
describes the target architecture; Decisions records why, and
services/ai_ordering/CLAUDE.md marks which rules are enforced in code today.
Terminology
- OEM — the manufacturer we order from (Stellantis, Mercedes, Audi, BMW), and the unit of
ordering. Each
(oem, market)has one profile and one service class. Each has at most one private API login. BMW has none, because BMW's Ordering Hub places its orders. - Brand (make) — a consumer marque an OEM owns (Stellantis: Jeep, RAM, …; BMW: BMW, MINI). Many brands per OEM; brands appear inside run scoping and product data, never as a dispatch key.
- Private dealer API — the authenticated, undocumented HTTP API of a dealer application we
automate (DealerConnect, NetSTAR); the transport the injected client
(
packages/ai_dealer_connect,packages/ai_netstar) speaks. Sessions, logins, and auth failures are private API concepts. - OMS — the order management system behind the private API: the logic that validates orders, adds defaults, refuses what it will not build, and accepts the rest. Submission outcomes are OMS concepts. Cure is our name for handling that response.
Architecture
One OEM-agnostic workflow calls per-OEM services (Strategy pattern) through a single generic
activity layer. An OEM is fully defined by an OrderingOEMProfile (job config) plus its
OrderingService implementation, both bound at the worker's composition root.
The ordering service interface and the factories that resolve an OEM's service live in
services/ordering.py;
types/ holds everything else that crosses the boundary:
| Module | Contents |
|---|---|
services/ordering.py | The OrderingService interface; OrderingServiceFactory with live and simulated implementations |
types/collection.py | CollectionJob — the OEM-declared collection jobs an ordering run triggers |
types/snapshots.py | SnapshotRef — key of one stored snapshot table; ResolvedSnapshot pairs one with its session timestamp |
types/curing.py | CureResult — validation/curing outcome: the cured build normalized, whether it is submittable, and the target-scoped signals the pipeline records |
types/target_scopes.py | Mode contracts: volume fill, order modification |
types/run_datasets.py | RunDatasets — the bucket and keys holding the warehouse tables a run reads once for its dealer. Names and keys only, so no row structure crosses the boundary |
types/errors.py | Canonical taxonomy: SessionConflict, AuthExpired, OrderingNotPermitted, ConfigurationError, SnapshotUnavailable |
Activity names are fixed. The task queue determines the run kind: one live queue and one
simulated queue, each serving every OEM. A run names its OEM in its input, and the activity layer
dispatches on (oem, market). Whether a run may reach a private API is fixed at deployment: the
simulated queue registers no activity that reaches one, and the services composed onto it hold a
mock private API client, so a simulated run cannot reach a private API whatever its input says.
Operating modes
Two modes cover every current OEM, and a new OEM declares which it uses. "Volume" and "modification" for short:
| Mode | Pattern | OEMs |
|---|---|---|
| Order modification | Pick modifiable orders from inventory → one target per slot → modify | Mercedes |
| Volume fill | Volume optimization over projected supply → one scope per product with a volume count → place | Stellantis; BMW (dry-run only) |
Run kinds
Four kinds run the same logic almost end to end. Two workflows implement them, one per queue:
OrderingRunWorkflow covers the three live kinds and SimulatedOrderingRunWorkflow the
simulated one. They differ in the steps below:
| Step | Simulated | Live dry run | Live stage | Live submission |
|---|---|---|---|---|
| Collection jobs | Resolved as of the run's as_of_date; never triggered | Triggered when past refresh_after | Triggered when past refresh_after | Triggered when past refresh_after |
| Validate and cure against the private API | Skipped — the recommendation stands as selected | Executed | Executed | Executed |
| Submit | Skipped | Skipped | The order is left staged in the OMS, where a person submits or discards it | Executed |
A stage run is available only to an OMS that stages orders before submission; today that is AccessAudi.
A simulated run reads the snapshot that was current at its as_of_date, so a backtest is a
replay with a different date.
A third workflow, StagedOrderCancellationWorkflow on the live queue, removes staged orders
from the OMS. Its input names the dealer, the models in scope (none covers every model the
dealer holds allocation for), and whether to cancel every
staged order in scope or only the ones identified as our placements; an explicit list of
staged order ids narrows either. A dry run lists the staged orders it would cancel and cancels
none; an execute run cancels them and each removed order becomes an attempt with status
CANCELLED, the removed build in original. Started alone, it records its attempts under its
own workflow id; started by an ordering run, under that run's id.
The run's cancel_staged_orders setting decides which staged orders an ordering run
cancels:
allcancels every staged order in scope.ourscancels only our placements.nonestarts no cancellation run.
A run that omits the setting uses its OEM profile's default, which is all for Audi and
none for every other OEM (ADR-017).
Scopes and targets
A scope is the product × dealer space one selection pipeline initializes over. A
target is one thing ordered: an existing order to modify, or one new order to place.
construct_targets emits one mode-tagged OrderingScope per pipeline space, with its production
windows as inclusive date intervals. types/target_scopes.py is the contract.
initialize_scope_pipeline runs once per scope, and the workflow drops the pipeline when the
scope ends, so a run's in-memory DataFrames belong to one scope however many products it orders
(ADR-004). Targets within a scope share that pipeline. RUN_LEVEL_TABLES names the tables read
once per run instead of once per scope. See Data flow and
the context layer.
Run flow
Every box in the phases below is one or more activities (or child workflows) invoked
by OrderingWorkflow. The workflow owns the sequence, the per-target loop, signals, and
continue-as-new, and holds run state only as opaque activity output:
Each step's logic has one code home. Where it executes is the worker whose queue the activity arrives on, so selection steps run on the ordering worker as generic activities delegating into the library:
| Step | Logic lives in |
|---|---|
| Run sequence, per-target loop, signals, continue-as-new | ai_ordering/workflows (Temporal sandbox) |
| Collect OEM data | OEM-declared collection jobs (profile config), executed as Dagster asset materializations |
| Freshness check, pull inventory candidates, emit/persist | Generic activities: ai_ordering/activities, services/, persistence/ |
| Construct targets, simulate/validate/order, report data | OrderingService impls in ai_ordering/oem/ |
| Volume optimization (inside construct targets, volume mode) | packages/ai_ppm |
| Pipeline init, recommendations, supply-state confirm | packages/candidate_selection — ordering only triggers |
| Which tables a run reads once vs per scope | RUN_LEVEL_TABLES in packages/candidate_selection; ordering decides when (once per run) because it owns the scope loop |
| Private API transport + parsers (collection and submit) | packages/ai_dealer_connect (DealerConnect), packages/ai_netstar (NetSTAR) |
| Report build & delivery | ai_ordering/reports, services/ (SendGrid, Slack) |
The full logic those phases run, mode branching included:
Loop mechanics:
- Pipelines are bounded at one scope. A scope's
SelectionPipelineexists only while its scope loop runs, and it crosses activities through external storage (ADR-003, ADR-004). - A rejection adds to the current target's restrictions, and the loop retries with the next candidate. A placement updates the pipeline's supply state, and the next scoring pass derives supply take rates from that state.
- A modification target's own vehicle is left out of supply until the target ends. Before
the workflow initializes the first scope's pipeline,
resolve_target_inventory_idsmaps each target'sorder_idto theinventory_idof theconsolidated/inventoryrow whoseorder_numbermatches it. Each scope's pipeline gets the vehicles of its own targets inexclude_inventory_ids, in addition to the vehicles the dealer's collected inventory does not list. When a target ends with a confirmed order, the confirmed build replaces the original vehicle in supply. When it ends without one and the scope has another target to fill,restore_pipeline_inventory(activity idt<target_index>-restore) adds the original vehicle back to the scope's supply. No restore runs after a scope's last target, afterSupplyExhausted, or after a cancellation, because the workflow then drops the scope's pipeline. - A scope can end before its targets do. A submit that finds the scope's supply gone fails
with
SupplyExhausted. The workflow records that attempt, makes no attempt on the scope's remaining targets, and continues with the next scope. Those targets count as skipped, and the exit reason is unchanged (ADR-015). - The event history alone describes the loop. Every activity in one try has the activity id
t<target_index>-a<attempt>-<step>, so the slot a row belongs to and whether it is a retry are visible on the timeline without decoding a payload. A refused build failsvalidate_and_curewith a non-retryable error whose details hold theCureResult, which the workflow reads back. The candidates the OMS refused are therefore the failing rows, and a target that needed four tries is one target with four attempts. - Cluster-shared supply state is deferred (ADR-012). A run's loop reads only its own confirms.
How a run ends
A run that reaches the end of its targets returns an OrderingResult whose exit_reason names
the outcome: QUOTA_EXHAUSTED, NO_CANDIDATES, DRY_RUN_COMPLETE or CANCELLED. A simulated
run that finds no usable collection session also returns, with SNAPSHOT_TOO_STALE and status
SKIPPED. Like a failing run, it persists its state and closes its event stream, but it
notifies no one.
Every other ending fails the workflow. Before failing, the run persists its state, notifies Slack, and closes its event stream. A run with a reused run_id only notifies Slack before failing (Run IDs). It sends no report, because a run that stopped early has no complete allocation to report on. Two cases:
| Ending | Failure the workflow raises |
|---|---|
| A reason the run itself decided: expired session, a live run's stale collection session, no completed partition, a pause nobody lifted | Non-retryable ApplicationError typed by the exit_reason, with the OrderingResult in its details |
| Any other activity failure | The activity's own error, unchanged |
A caller therefore reads a stopped run's outcome from the failure.
A run records its exit_reason on its ordering_runs document when it ends. A run with a
reused run_id writes no document.
Externally triggered runs
Some OEMs run their own order management and place their own orders. For those, a run produces validated order templates and never submits, and the OEM's system asks for them over HTTP rather than the run being scheduled here. The OEM system holds no warehouse credentials and reads no Iceberg or BigQuery table: the API is the whole boundary.
Three calls make up the contract:
| Call | Effect |
|---|---|
| Request templates | Validates the volumes, starts the batch workflow, returns the request's id and URL |
| Get the request | Per-model status and counts while it runs; its templates once they are ready |
| Report order status | What the OEM system placed, keyed on the template that recommended it |
The contract is named for what the caller asked for rather than for the run that serves it, so the fan-out above can change without the contract changing.
One run per model, every dealer in it. Volume arbitrage needs a model's volumes across all dealers before it can split control from test, and national take-rate steering scores a candidate against counts accumulated across the whole model. Neither is answerable one dealer at a time, so the model is the unit the batch fans out on and the dealer is a scope inside one run. This makes a model's run larger than a single-dealer run, and the pipeline's per-scope bound (ADR-004) applies to the dealer scope within it, not to the model.
Results are asynchronous because a model's run takes minutes. The request call returns as soon as the batch starts; nothing in the contract waits on selection.
Placement outcomes return through the same API. The OEM system reports what it placed, and that report reaches the warehouse in one agreed format, so measurement reads placements the same way whether we placed them or the OEM did.
Data flow
An ordering run never pulls a private API for collected data. It resolves one collection_id per table
of each declared collection job (CollectionJob.tables) and reads that snapshot. When the
snapshot is stale, a live run requests the job's partition from Dagster and waits for the
materialization. The verdict table below gives the rule. Collection jobs are Dagster assets, and
snapshots are written to Iceberg keyed (dealer, table, date, collection_id), every
collection of a day appended to its partition (ADR-016). A source whose
collector has not migrated still writes BigQuery, and the store selects the backend per source.
A live run's target construction enumerates the dealer's configurable models from the private API
(get_eligible_model_ids); a simulated run reads that set from the warehouse instead. The private
API is used again at validate and submit.
The warehouse probe checks presence: it resolves the newest date, at or before the run's
as_of_date and within partition_lookback_days, on which every table the run reads has
completed. A brand's ingestion completes at some point within the UTC day it covers, so a run
starting before that reads the day before instead of terminating. No date inside the window
terminates
the run, live or simulated. A date is taken whole. The resolved date replaces the run's
as_of_date before any activity that reads data is dispatched, so one date covers the
partition, the production window and the collection reference time of a simulated run. The
run records the date it requested as requested_as_of_date and the resolved date as
partition_date on its ordering_runs document. The dependency list is derived from the IcebergTableMap the selection config declares plus the
service's volume_optimization_reads, so a table added to a read is covered without a
second list.
Before it resolves a date, the probe checks that each table's current Iceberg schema contains
every column that the run selects from the table. The selected columns are the select_columns
and scope_filters of each IcebergTableRef, plus the columns that volume_optimization_reads
lists. If a column is missing, the probe stops the run with WAREHOUSE_COLUMN_MISSING, and the
error names each table and each missing column. The check reads only table metadata.
Warehouse reads happen at two levels. Most tables are read per scope, where the partition is
worth scanning for one model's rows. The tables in candidate_selection.RUN_LEVEL_TABLES hold
every dealer and model of the market in one partition, so a scope's predicate narrows the rows
returned while the bytes scanned stay the same. load_run_dealer_datasets reads those once for
the dealer at the start of the ordering loop, and the workflow passes the result to every scope,
which narrows it in memory. The scan happens once per run instead of once per model. For the
same reason, load_run_dealer_datasets computes the dealer's days-in-status once from the
trailing partitions of inventory, and every scope reads that frame.
RunDatasets crosses activities as a bucket and keys. The tables stay in the cache bucket, and a
scope fetches the whole-table payloads plus its own model's slices. It has no row model, so
SnapshotRef remains the only row reference between activities.
CollectionJob.assess_freshness is a pure, sandbox-safe function the workflow computes inline
against its reference time: its own clock for a live run, and for a simulated run the end of its
as_of_date or the run's start, whichever is earlier. Resolution therefore stays one activity
that returns data, and the trigger-or-stop branching is workflow control flow:
| Verdict | Age | Live run | Simulated run |
|---|---|---|---|
fresh | ≤ refresh_after | reuse | reuse |
stale | ≤ max_staleness (live) or simulated_max_staleness (simulated) | trigger the job | reuse |
expired | > that limit | trigger the job | skip |
missing | no completed session | trigger the job | skip |
A skipped simulated run completes with status SKIPPED and exit reason SNAPSHOT_TOO_STALE. It
is not a failure, because the run only reads what was collected. A live run that still lacks a
session after collecting fails.
A dealer with no completed session resolves to None, and absence is a freshness input. The
store raises only for faults: ConfigurationError for a logical table the source does not
configure, SnapshotUnavailable for a ref naming a session its table never recorded.
While the TypeScript collectors write the snapshots, a session's identity is its fetch_date:
one timestamp written on every row a session writes to a checked table. SnapshotStore.resolve
derives collection_id from that timestamp, and reads translate it back into a fetch_date
filter. A migrated collector writes its Dagster run id as the collection_id on every row it
appends, and reads filter on it. A table without a per-session timestamp cannot be checked for
freshness. feature_catalog has one per combo and sales_* one per row. SnapshotRefs are the
only snapshot reference in activity inputs and outputs, and rows are read inside activities.
Mercedes collects its inventory through a CollectionJob like Stellantis and Audi: MERCEDES_US
declares the mercedesUsInventoryWorkflow TypeScript workflow, and a session's timestamp is
insertedAt. resolve counts a session only when it has at least SnapshotSource.min_session_rows
rows: 2 for Mercedes, 1 for every other source. The legacy ordering runner writes each order it
modifies back to the Mercedes inventory table as a one-row session with its own insertedAt, and
that row is not a fetch of the dealer's inventory. So a dealer whose whole inventory is one order
has no session that counts.
Reads go through services/snapshots.py, a single generic SnapshotStore with resolve and a
typed read_rows(ref, row_model). Row structures are OEM-internal: each OEM defines its own row
models beside its service and binds them where it calls read_rows. The ref is the only
cross-OEM snapshot contract.
An OEM service never calls resolve. In Step 0, the ordering workflow resolves one session per
snapshot table through the resolve_snapshots activity. It sets the refs on RunContext.snapshots
and records them on the run's ordering_runs document. An OEM service reads the sessions those refs
name, so every activity of an ordering run reads the session the run recorded.
A staged-order cancellation that an ordering run starts receives the run's refs. A cancellation started on its own resolves its sessions up to the end of its partition date. It triggers no collection.
Run state
| State | Where it lives |
|---|---|
| Pulled snapshots (inventory — orders included) | BigQuery (dealer, table, date, collection_id); read via SnapshotStore |
| Warehouse reference data (models, dealers, stats) | Iceberg/Nessie via services/warehouse.py, read at the run's partition |
| OEM payload between a service's own activities | Temporal External Storage — typed activity args/returns; over-threshold payloads offload to S3 |
Live SelectionPipeline | Passed between the selection activities, offloaded to Temporal External Storage like the OEM payload. The workflow passes it as an opaque RawValue and never decodes one, because deserializing it with pickle imports polars, which the workflow sandbox forbids |
| Cluster-shared supply state (future) | candidate_selection library — the durable extension of the supply state confirm_order already mutates |
| Runs, attempts, events, restriction evidence, volume decisions, report builds | MongoDB; events sync to Iceberg, the volume decision included through its own event |
Run IDs
A run's run_id is its Temporal workflow ID. Each run must have its own run_id. A child workflow of a simulated run with several dealers takes the run_id of its parent, as A simulated run with several dealers describes.
ordering_runs holds one document per dealer of a run. Its _id is {run_id}:{dealer_id}, and
its run_id and dealer_id fields name the run and the dealer. volume_optimizations uses the
same value as its _id, and ordering_run_cache stores it in its run_id field. Readers select
ordering_runs documents by the run_id and dealer_id fields and never parse _id.
Documents written before ordering_runs had a run_id field have no such field, and their _id
is the run id.
Each ordering_runs document records the first_execution_run_id of the run that wrote it. A run
fails with RUN_ID_REUSED when the document for its run_id and dealer has a different value. A
reset, a workflow retry and continue-as-new keep the value, so they continue the same run.
Dagster starts runs with WorkflowIDReusePolicy.REJECT_DUPLICATE, so Temporal rejects a reused
workflow ID before the run starts.
A simulated run with several dealers
Dagster starts SimulatedOrderingRunWorkflow for a partition with every dealer that the Dagster
op selects in dealer_ids. Simulated runs describes the workflow ID and how
the op selects the dealers. Started with one dealer, the workflow runs that dealer itself.
Started with several, it:
- checks
ordering_run_groupsfor another execution of its run_id, and writes its document there; - runs
probe_warehouseonce; - starts one child workflow of its own type per dealer, with the ID
{run_id}-{dealer_id}, one dealer indealer_idsand the resolved partition asparent_partition_date; - records each child's outcome on its
ordering_run_groupsdocument.
A child takes its run_id from its parent, so the children write every dealer's ordering_runs
document, attempts and events under one run_id. A child reads the partition its parent resolved
and does not probe the warehouse itself. Its as_of_date is the date the run was requested for,
so its ordering_runs document records the requested date and the partition read, as a run with
one dealer does. A failed child is recorded as FAILED and does not fail the run. If at least one
child failed and none finished, the run ends FAILED.
RunConfig.child_runs sets these limits:
max_runningis the most children that run at once. By default every child starts at once.deadline_hours, 3 by default, is the number of hours after the run starts at which the run signalscancel_orderingto every child still running. The run then waitsCHILD_CANCEL_GRACEfor those children to end. It cancels each child still running after that through Temporal, and records the child asUNFINISHED.
The run forwards pause_ordering, resume_ordering and cancel_ordering to its running
children. It starts no child while it is paused or cancelled. The live OrderingRunWorkflow
accepts one dealer only.
Target layout
services/ai_ordering/src/ai_ordering/
├── worker.py # composition root: profile → services → activities → queue
├── profiles.py # OrderingOEMProfile per OEM (job config only)
├── queues.py # the live and simulated ordering queues
├── types/ # the entire boundary: everything that crosses it
├── activities/ # generic Temporal layer — only temporalio importer
├── workflows/ # sandbox loop internals
├── oem/ # per-OEM OrderingService implementations + their private API clients
├── services/ # storage + integrations: warehouse, snapshots, sendgrid, …
├── persistence/ # Mongo: runs, attempts, events, oem_restrictions, volume, reports
└── state/ # what a run must hold in memory, and nothing else
Restrictions
There are two kinds. Both treat a live restriction as evidence a later attempt can confirm or overturn:
- Static: known before the run —
oem_restrictions(transcribed by operators today and fed by ingestion later) and the dealer'sdealer_rules. Both key a rule bybase_model_id.- The
construct_targetsactivity addsoem_restrictionsand the dealer'sOPTION_CODErules to every OEM's scopes after the OEM builds them. Candidate selection prunes on these rules. For each scope, the activity:- looks up each model's
base_model_idinconsolidated/models - reads that base model's rules through
services/restrictions.py - appends them with
OrderingScope.with_added_restrictions
- looks up each model's
- Mercedes reads the dealer's
ACCESSORYrules invalidate_and_cureand applies them while it edits the order. It removes must-not-have accessories and adds must-have ones. These rules never reach candidate selection.
- The
- Discovered: found when a submit is rejected, and slot-scoped today. Mercedes restrictions depend on date and threshold, so moving them into shared scope needs a per-OEM policy for scope, TTL and confidence. Moving them too early suppresses the attempts that would prove or disprove the restriction.
Deployment and scaling
One codebase, one image, one ECS service definition, instantiated per (OEM, run kind), with each
instance's queue and OEM set by configuration. Ordering services scale between zero and one with
the run trigger, which already knows work is coming. Collection workers scale on task-queue
backlog. ResourceBasedSlotSupplier caps per-worker concurrency by memory and CPU.
desired_count > 1 depends on durable run state (ADR-003). Scaling is per queue, so per-OEM
scaling requires per-OEM queues.