Skip to main content

PPM Data Platform

Multi-OEM vehicle data platform built with Dagster. Shared logic lives in packages/ai_core; each OEM gets its own Dagster project under projects/.

  • Getting Started — Installation, running Dagster locally, materializing assets, monitoring runs, running tests
  • Dagster Execution Model — How the ECS executor works, run task vs step tasks, resource sizing, container_context.yaml configuration
  • Monitoring — Sentry alerting for asset check failures and run crashes
  • Debugging ECS Containers — Using ECS Exec / SSM to shell into Fargate containers (Nessie, code locations)
  • Releases — Automated versioning and publishing of packages/ libraries to Nexus via release-please + Conventional Commits

Ingestion​

  • Design — Three-tier pipeline, asset key conventions, checks, partitioning
  • Code Structure — Package layout, component classes, YAML config, adding a new OEM
  • Component Reference — Advanced component attributes, raw tier schema, utilities
  • ConsolidatedComponent — Source groups, gap fill, merge spec, foreign keys
  • Operations — Running Dagster, materializing assets, monitoring runs
  • Enhancement Ideas — Cross-cutting ingestion enhancements we want to pursue

Derived​

Downstream of consolidated ingestion, the platform produces a graph of OEM-specific assets that turn raw entity records into KPIs, predictions, and smoothed dealer-level take rates. Each asset is partitioned by day and emitted by a reusable component class.

Asset keys always start with <oem>/ and use the two-segment <oem>/<name> pattern. Dealer classifications are written under <oem>/dealers/<name> (e.g. mb/dealers/neighborhoods). Every asset uses DailyPartitionsDefinition with end_offset=1, matching the consolidated tier.

  • Enrichment — Spatial dealer clustering; computed sold date, days on lot, and sold-window flags are configured directly on ConsolidatedComponent's inventory/models specs
  • Vehicle Attributes — Option classification into structured attribute types and per-vehicle feature assembly
  • Attribute Tagging — LLM-powered mapping of OEM features to canonical functional attributes
  • Inventory Statistics — Current inventory count, rolling sales counts, days-supply, average days-on-lot per grouping
  • Days on Lot — Trains a regression model and scores every active vehicle with predicted days on lot
  • Take Rates — Per-attribute take rates and metric KPIs, blended across geographic layers to dealer granularity
  • Dealer Rules — Per-dealer must-have / must-not-have inference from take rates, and the MongoDB sync the ordering app reads
  • Volume Optimization — Unit-agnostic MIP that allocates integer volumes across decision units (build codes, models, brands, auction candidates, …) to drive per-attribute supply toward target supply, weighted L1 objective

Warehouse​

  • Nessie — Iceberg catalog: deployment, VPC vs public connectivity, Quarkus OIDC + CEL RBAC, cached service token, device-flow login, utilities
  • DuckDB Browser Visualizer (Design) — proposal: DuckDB-WASM in-browser SQL over Nessie-cataloged Iceberg tables, per-branch, via credential vending

AWS​

  • AI Agent AWS Access — temporary IAM Identity Center credentials for local agents
  • Cost Attribution — the autointel:* tag scheme, where each tag is set, payer-account activation, and what spend tags cannot split

Ordering​

Design of the services/ai_ordering Temporal worker — the brand-pluggable order-placement platform.

  • Ordering Platform Design — target architecture: service ports, run flow, state model, data flow, deployment
  • Decisions (ADRs) — the settled architecture decisions and their rationale
  • Codec Server — reading offloaded payloads in the Temporal UI and CLI: local setup, configuration, deployment
  • Serverless Workers — running the ordering worker on AWS Lambda: what it can serve today, deployment, limits

Public Sources​

Sources in the projects/ai_public code location — public data that is not tied to a single OEM's own APIs.

  • TVM Source (Team Velocity) — public dealer-website inventory API: per-OEM dealer JSON, where the AccountId/CampaignId identifiers come from, and what the API does and does not enforce
  • Dealer Inspire Source — the Cars Commerce search service behind Dealer Inspire sites: the shared API key, where the ccid comes from, and why one ccid can cover several rooftops
  • DealerOn Source — public dealer-website inventory API: per-OEM dealer JSON, where the DealerId/PageId identifiers come from, the fixed page-size set, and the fields a card carries

Candidate Generation​

The candidate set a dealer can be recommended from: real inventory configurations plus, for an OEM that declares rules, configurations mapped forward from the prior model year.

  • Synthetic Candidates — Maps prior-model-year inventory into the next year's configuration space via YAML rules; three-layer design confining OEM fields to rule authoring — the model_id-keyed core and output carry none
  • Candidate Coverage Checks — Asset checks on candidates_output: the orderable-model population, the per-model floor read off each model's option grid, and the shortfall reasons

Candidate Selection​

Design notes for the packages/candidate_selection ML-CDF supply model.