PPM Data Platform
Multi-OEM vehicle data platform built with Dagster. Shared logic lives in packages/ai_core; each OEM gets its own Dagster project under projects/.
- Getting Started — Installation, running Dagster locally, materializing assets, monitoring runs, running tests
- Dagster Execution Model — How the ECS executor works, run task vs step tasks, resource sizing,
container_context.yamlconfiguration - Monitoring — Sentry alerting for asset check failures and run crashes
- Debugging ECS Containers — Using ECS Exec / SSM to shell into Fargate containers (Nessie, code locations)
- Branch Deployments — When a pull request gets a branch deployment, and which workflows delete it
- Deploys from Main — Which components a CI run on main deploys, and how to redeploy them
- CI Credentials — Which GitHub environments hold CI's credentials, which jobs use them, and what a branch deploy can reach
- GCP Access — How user code reaches BigQuery and Cloud Storage through Workload Identity Federation
- Releases — Automated versioning and publishing of
packages/libraries to Nexus via release-please + Conventional Commits
Ingestion
- Design — Three-tier pipeline, asset key conventions, checks, partitioning
- Code Structure — Package layout, component classes, YAML config, adding a new OEM
- Component Reference — Advanced component attributes, raw tier schema, utilities
- ConsolidatedComponent — Source groups, gap fill, merge spec, foreign keys
- CompositeInventoryComponent — Daily snapshot of a market's public inventory sources, merged per vehicle
- Inferred Sales — Sales inferred from each composite inventory: the vehicles that left it, one row per sale
- Operations — Running Dagster, materializing assets, monitoring runs
- Enhancement Ideas — Cross-cutting ingestion enhancements we want to pursue
Derived
Downstream of consolidated ingestion, the platform produces a graph of OEM-specific assets that turn raw entity records into KPIs, predictions, and smoothed dealer-level take rates. Each asset is partitioned by day and emitted by a reusable component class.
Asset keys always start with <oem>/ and use the two-segment <oem>/<name> pattern. Dealer classifications are written under <oem>/dealers/<name> (e.g. mb/dealers/neighborhoods). Every asset uses DailyPartitionsDefinition with end_offset=1, matching the consolidated tier.
- Enrichment — Spatial dealer clustering; computed sold date, days on lot, and sold-window flags are configured directly on ConsolidatedComponent's inventory/models specs
- ACS Demographic Features — the Census ACS features in
ai_core.acs: scalars and ratios of one geography, how each is computed over several geographies, the bracket median interpolation, and the ACS vintages whose variable labels match the bracket bounds - Featurization Schemes — When a binary classification scheme states criteria, which clause a false criterion ends with, and how to check a scheme change with the
feature_hierarchyeval - Featurization Checks — The asset checks that measure whether vehicles have a value of each required attribute type
- Featurization Groups — Attribute encoding, per-vehicle sets over the feature hierarchy's top-level categories, the
all_features_pricemeasure and crosses - Inventory Statistics — Current inventory count, rolling sales counts, days-supply, average days-on-lot per grouping
- Days on Lot — Trains a regression model and scores every active vehicle with predicted days on lot
- Take Rates — Per-attribute take rates and metric KPIs, blended across geographic layers to dealer granularity
- Attribute Weights — How the weight of each attribute type is fit, and the configurable weight bounds
- Dealer Preferences — Per-dealer must-have / must-not-have inference from take rates, and the MongoDB sync the ordering app reads
- Volume Optimization — Unit-agnostic MIP that allocates integer volumes across decision units (build codes, models, brands, auction candidates, …) to drive per-attribute supply toward target supply, weighted L1 objective
Warehouse
- Nessie — Iceberg catalog: deployment, VPC vs public connectivity, Quarkus OIDC + CEL RBAC, cached service token, device-flow login, utilities
- DuckDB Browser Visualizer (Design) — proposal: DuckDB-WASM in-browser SQL over Nessie-cataloged Iceberg tables, per-branch, via credential vending
AWS
- AI Agent AWS Access — temporary IAM Identity Center credentials for local agents
- Cost Attribution — the
autointel:*tag scheme, where each tag is set, payer-account activation, and what spend tags cannot split
Ordering
Design of the services/ai_ordering Temporal worker — the brand-pluggable order-placement platform.
- Ordering Platform Design — target architecture: the OrderingService interface, run flow, state model, data flow, deployment
- Decisions (ADRs) — the settled architecture decisions and their rationale
- Template API — the HTTP contract an OEM ordering system uses to request templates and report placements
- Codec Server — reading offloaded payloads in the Temporal UI and CLI: local setup, configuration, deployment
- Serverless Workers — running the ordering worker on AWS Lambda: what it can serve today, deployment, limits
- Simulated Runs — running selection and volume optimization on a warehouse partition without calling an OEM OMS: isolation, the DAG trigger, the as-of date
auto_swe
The workflows/auto_swe Temporal application, which runs coding-agent sessions on work items from GitHub.
- Architecture — what is deployed and how a work item moves through it
- The pull-request agent — reviews and answers on pull requests: commands, turning it on, tuning, limits and feedback
- The triage agent — files the root causes of failed Dagster runs and checks in Linear: how failures reach it, grading, where a group goes, correcting it
- The implementation agent — turns a planned Linear issue into a draft pull request: plan check, implementation, review and fix rounds, the reading guide
- How a session runs — work items, subjects, routing, and a step run
- Configuration — the agent YAML, repository context,
SessionConfigand the environment - Security model — writes, branch protection, token scopes, credentials in the sandbox
- Observability — lifecycle events, history, search attributes, tokens and the LangSmith thread
- Deploying auto_swe — the worker and webhook functions, what deploys what, secrets, and the first deploy
- MCP tools — MCP servers as a tool source, and the AWS MCP Server's credentials
- Internals — the registries, the sandbox activities, commands with credentials, and a work item by module
Reporting
Measurement of what the ordering platform recommended and what became of the orders it posted — shared components replacing the per-OEM Dataform projects.
- Reporting Layer Design — target architecture: the two inputs, run modes, modification detection, order-to-inventory matching, grouping levels
- Decisions (ADRs) — the architecture decisions and their rationale
- Build Deltas — the component that computes build deltas from the ordering events and the private inventory
- Placed Orders — the component that resolves each placed order to its inventory record
- Dealer Modifications — the component that measures how often a dealer changes the build we ordered
- Order Set Evaluation — the components that compare a run's recommended order sets with reference order sets from consolidated inventory or with demand take rates
- Session Activity — the component that reports when Cookie Monster last recorded OMS session activity for each dealer and marks dealers that need a login reminder
- Allocation Windows — the component that posts to Slack when an OEM's allocation window opens
- Run Metrics — the component that measures what each ordering run recommended
- Simulated Runs — the component that posts a daily Slack report of an OEM's simulated ordering runs
Public Sources
Sources in the projects/ai_public code location — public data that is not tied to a single OEM's own APIs.
- TVM Source (Team Velocity) — public dealer-website inventory API: dealers read from
website_platform, where the AccountId/CampaignId identifiers come from, and what the API does and does not enforce - Dealer Inspire Source — the Cars Commerce search service behind Dealer Inspire sites: the shared API key, where the ccid comes from, and why one ccid can cover several rooftops
- Dealer.com Source — public dealer-website inventory endpoint: the dealers that
website_platformclassifies as Dealer.com, the request state a search page holds, and why a request names the site's own rooftop - Motive Source — the shared Algolia index behind Motive dealer websites: the search-only key in every site's page source, where the Motive dealer id comes from, and why a search filters on the site's own rooftop
- DealerOn Source — public dealer-website inventory API: dealers read from
website_platform, where the DealerId/PageId identifiers come from, the fixed page-size set, and the fields a card carries - Dealer Website Platform Classification — which vendor runs each dealer's website and the parameters its inventory API needs: the
evidencevalues, homepage fetching and refusals, the Zyte fallback, and the configured OEMs - Composite Inventory — the markets whose OEM feed is merged with dealer-website sources: source priority, which fields take the latest value, and column alignment
- Inferred Sales — the markets with sales inferred from their composite inventory, and each market's windows and thresholds
- Census Tract Boundaries —
census/tiger/cb_tract: the national cartographic boundary GeoPackage, the NAD83 to EPSG:4326 transformation, the checks, and adding a vintage - Census ZCTA Boundaries —
census/tiger/cb_zcta520: the 2020 cartographic boundary ZCTA shapefile, its columns, and the checks - Census Blocks —
census/tiger/tabblock20: each 2020 block's housing units, population and internal point, read from the TIGER/LineTABBLOCK20attribute tables with HTTP range requests, and the checks - Census States —
census/tiger/state: each state's FIPS code, abbreviation, census region and division, read from the TIGER/Line state attribute table with HTTP range requests, and the checks - Valhalla Routing Tiles — the US routing graph for
pyvalhalla: the dated Geofabrik extract each version is built from, where the archive is written, and the task size - Dealer Isochrones — drive-time isochrones of each franchise and the other franchises inside them: the census tracts each isochrone covers, the request settings, reusing the previous day's polygons, and the coverage checks
Candidate Generation
The candidate set a dealer can be recommended from: real inventory configurations plus, for an OEM that declares rules, configurations mapped forward from the prior model year.
- Synthetic Candidates — Maps prior-model-year inventory into the next year's configuration space via YAML rules; three-layer design confining OEM fields to rule authoring — the
model_id-keyed core and output carry none - Candidate Coverage Checks — Asset checks on
candidates_output: the models in scope, each model's configuration space, the per-model and per-feature baselines, and the variety checks, which measure candidate variety prioritized by demand - Model-Year Transitions — What changed between each base model's newest model year and the year before, per option code and per attribute
Candidate Selection
Architecture, configuration, and scoring for packages/candidate_selection.
-
System Overview — component responsibilities and production-date weighting
-
Scoring and Selection — scorers and selection strategies
-
MLCDFSupplyModel Design — per-VIN ML-derived supply matrix
-
BMW ML CDF Batch Evaluation Plan — running the
bmw_ml_cdfpipeline over a sample to validate output