Skip to main content

PPM Data Platform

Multi-OEM vehicle data platform built with Dagster. Shared logic lives in packages/ai_core; each OEM gets its own Dagster project under projects/.

  • Getting Started — Installation, running Dagster locally, materializing assets, monitoring runs, running tests
  • Dagster Execution Model — How the ECS executor works, run task vs step tasks, resource sizing, container_context.yaml configuration
  • Monitoring — Sentry alerting for asset check failures and run crashes
  • Debugging ECS Containers — Using ECS Exec / SSM to shell into Fargate containers (Nessie, code locations)
  • Releases — Automated versioning and publishing of packages/ libraries to Nexus via release-please + Conventional Commits

Ingestion​

  • Design — Three-tier pipeline, asset key conventions, checks, partitioning
  • Code Structure — Package layout, component classes, YAML config, adding a new OEM
  • Component Reference — Advanced component attributes, raw tier schema, utilities
  • ConsolidatedComponent — Source groups, gap fill, merge spec, foreign keys
  • Operations — Running Dagster, materializing assets, monitoring runs
  • Enhancement Ideas — Cross-cutting ingestion enhancements we want to pursue

Derived​

Downstream of consolidated ingestion, the platform produces a graph of OEM-specific assets that turn raw entity records into KPIs, predictions, and smoothed dealer-level take rates. Each asset is partitioned by day and emitted by a reusable component class.

Asset keys always start with <oem>/ and use the two-segment <oem>/<name> pattern. Dealer classifications land under <oem>/dealers/<name> (e.g. mb/dealers/neighborhoods). Every asset uses DailyPartitionsDefinition with end_offset=1, matching the consolidated tier.

  • Enrichment — Spatial dealer clustering and computed sold-date, days-on-lot, half-life weights on consolidated inventory
  • Vehicle Attributes — Option classification into structured attribute types and per-vehicle feature assembly
  • Attribute Tagging — LLM-powered mapping of OEM features to canonical functional attributes
  • Inventory Statistics — Current inventory count, rolling sales counts, days-supply, average days-on-lot per grouping
  • Days on Lot — Trains a regression model and scores every active vehicle with predicted days on lot
  • Take Rates — Per-attribute take rates and metric KPIs, blended across geographic layers to dealer granularity
  • Volume Optimization — Unit-agnostic MIP that allocates integer volumes across decision units (build codes, models, brands, auction candidates, …) to drive per-attribute supply toward target supply, weighted L1 objective

Warehouse​

  • Nessie — Iceberg catalog: deployment, VPC vs public connectivity, Quarkus OIDC + CEL RBAC, cached service token, device-flow login, utilities
  • DuckDB Browser Visualizer (Design) — proposal: DuckDB-WASM in-browser SQL over Nessie-cataloged Iceberg tables, per-branch, via credential vending

Candidate Selection​

Design notes for the packages/candidate_selection ML-CDF supply model.