Skip to main content

PPM Data Platform

Multi-OEM vehicle data platform built with Dagster. Shared logic lives in packages/ai_core; each OEM gets its own Dagster project under projects/.

  • Getting Started — Installation, running Dagster locally, materializing assets, monitoring runs, running tests
  • Dagster Execution Model — How the ECS executor works, run task vs step tasks, resource sizing, container_context.yaml configuration
  • Monitoring — Sentry alerting for asset check failures and run crashes
  • Debugging ECS Containers — Using ECS Exec / SSM to shell into Fargate containers (Nessie, code locations)
  • Branch Deployments — When a pull request gets a branch deployment, and which workflows delete it
  • Deploys from Main — Which components a CI run on main deploys, and how to redeploy them
  • CI Credentials — Which GitHub environments hold CI's credentials, which jobs use them, and what a branch deploy can reach
  • GCP Access — How user code reaches BigQuery and Cloud Storage through Workload Identity Federation
  • Releases — Automated versioning and publishing of packages/ libraries to Nexus via release-please + Conventional Commits

Ingestion​

  • Design — Three-tier pipeline, asset key conventions, checks, partitioning
  • Code Structure — Package layout, component classes, YAML config, adding a new OEM
  • Component Reference — Advanced component attributes, raw tier schema, utilities
  • ConsolidatedComponent — Source groups, gap fill, merge spec, foreign keys
  • CompositeInventoryComponent — Daily snapshot of a market's public inventory sources, merged per vehicle
  • Inferred Sales — Sales inferred from each composite inventory: the vehicles that left it, one row per sale
  • Operations — Running Dagster, materializing assets, monitoring runs
  • Enhancement Ideas — Cross-cutting ingestion enhancements we want to pursue

Derived​

Downstream of consolidated ingestion, the platform produces a graph of OEM-specific assets that turn raw entity records into KPIs, predictions, and smoothed dealer-level take rates. Each asset is partitioned by day and emitted by a reusable component class.

Asset keys always start with <oem>/ and use the two-segment <oem>/<name> pattern. Dealer classifications are written under <oem>/dealers/<name> (e.g. mb/dealers/neighborhoods). Every asset uses DailyPartitionsDefinition with end_offset=1, matching the consolidated tier.

  • Enrichment — Spatial dealer clustering; computed sold date, days on lot, and sold-window flags are configured directly on ConsolidatedComponent's inventory/models specs
  • ACS Demographic Features — the Census ACS features in ai_core.acs: scalars and ratios of one geography, how each is computed over several geographies, the bracket median interpolation, and the ACS vintages whose variable labels match the bracket bounds
  • Featurization Schemes — When a binary classification scheme states criteria, which clause a false criterion ends with, and how to check a scheme change with the feature_hierarchy eval
  • Featurization Checks — The asset checks that measure whether vehicles have a value of each required attribute type
  • Featurization Groups — Attribute encoding, per-vehicle sets over the feature hierarchy's top-level categories, the all_features_price measure and crosses
  • Inventory Statistics — Current inventory count, rolling sales counts, days-supply, average days-on-lot per grouping
  • Days on Lot — Trains a regression model and scores every active vehicle with predicted days on lot
  • Take Rates — Per-attribute take rates and metric KPIs, blended across geographic layers to dealer granularity
  • Attribute Weights — How the weight of each attribute type is fit, and the configurable weight bounds
  • Dealer Preferences — Per-dealer must-have / must-not-have inference from take rates, and the MongoDB sync the ordering app reads
  • Volume Optimization — Unit-agnostic MIP that allocates integer volumes across decision units (build codes, models, brands, auction candidates, …) to drive per-attribute supply toward target supply, weighted L1 objective

Warehouse​

  • Nessie — Iceberg catalog: deployment, VPC vs public connectivity, Quarkus OIDC + CEL RBAC, cached service token, device-flow login, utilities
  • DuckDB Browser Visualizer (Design) — proposal: DuckDB-WASM in-browser SQL over Nessie-cataloged Iceberg tables, per-branch, via credential vending

AWS​

  • AI Agent AWS Access — temporary IAM Identity Center credentials for local agents
  • Cost Attribution — the autointel:* tag scheme, where each tag is set, payer-account activation, and what spend tags cannot split

Ordering​

Design of the services/ai_ordering Temporal worker — the brand-pluggable order-placement platform.

  • Ordering Platform Design — target architecture: the OrderingService interface, run flow, state model, data flow, deployment
  • Decisions (ADRs) — the settled architecture decisions and their rationale
  • Template API — the HTTP contract an OEM ordering system uses to request templates and report placements
  • Codec Server — reading offloaded payloads in the Temporal UI and CLI: local setup, configuration, deployment
  • Serverless Workers — running the ordering worker on AWS Lambda: what it can serve today, deployment, limits
  • Simulated Runs — running selection and volume optimization on a warehouse partition without calling an OEM OMS: isolation, the DAG trigger, the as-of date

auto_swe​

The workflows/auto_swe Temporal application, which runs coding-agent sessions on work items from GitHub.

  • Architecture — what is deployed and how a work item moves through it
  • The pull-request agent — reviews and answers on pull requests: commands, turning it on, tuning, limits and feedback
  • The triage agent — files the root causes of failed Dagster runs and checks in Linear: how failures reach it, grading, where a group goes, correcting it
  • The implementation agent — turns a planned Linear issue into a draft pull request: plan check, implementation, review and fix rounds, the reading guide
  • How a session runs — work items, subjects, routing, and a step run
  • Configuration — the agent YAML, repository context, SessionConfig and the environment
  • Security model — writes, branch protection, token scopes, credentials in the sandbox
  • Observability — lifecycle events, history, search attributes, tokens and the LangSmith thread
  • Deploying auto_swe — the worker and webhook functions, what deploys what, secrets, and the first deploy
  • MCP tools — MCP servers as a tool source, and the AWS MCP Server's credentials
  • Internals — the registries, the sandbox activities, commands with credentials, and a work item by module

Reporting​

Measurement of what the ordering platform recommended and what became of the orders it posted — shared components replacing the per-OEM Dataform projects.

  • Reporting Layer Design — target architecture: the two inputs, run modes, modification detection, order-to-inventory matching, grouping levels
  • Decisions (ADRs) — the architecture decisions and their rationale
  • Build Deltas — the component that computes build deltas from the ordering events and the private inventory
  • Placed Orders — the component that resolves each placed order to its inventory record
  • Dealer Modifications — the component that measures how often a dealer changes the build we ordered
  • Order Set Evaluation — the components that compare a run's recommended order sets with reference order sets from consolidated inventory or with demand take rates
  • Session Activity — the component that reports when Cookie Monster last recorded OMS session activity for each dealer and marks dealers that need a login reminder
  • Allocation Windows — the component that posts to Slack when an OEM's allocation window opens
  • Run Metrics — the component that measures what each ordering run recommended
  • Simulated Runs — the component that posts a daily Slack report of an OEM's simulated ordering runs

Public Sources​

Sources in the projects/ai_public code location — public data that is not tied to a single OEM's own APIs.

  • TVM Source (Team Velocity) — public dealer-website inventory API: dealers read from website_platform, where the AccountId/CampaignId identifiers come from, and what the API does and does not enforce
  • Dealer Inspire Source — the Cars Commerce search service behind Dealer Inspire sites: the shared API key, where the ccid comes from, and why one ccid can cover several rooftops
  • Dealer.com Source — public dealer-website inventory endpoint: the dealers that website_platform classifies as Dealer.com, the request state a search page holds, and why a request names the site's own rooftop
  • Motive Source — the shared Algolia index behind Motive dealer websites: the search-only key in every site's page source, where the Motive dealer id comes from, and why a search filters on the site's own rooftop
  • DealerOn Source — public dealer-website inventory API: dealers read from website_platform, where the DealerId/PageId identifiers come from, the fixed page-size set, and the fields a card carries
  • Dealer Website Platform Classification — which vendor runs each dealer's website and the parameters its inventory API needs: the evidence values, homepage fetching and refusals, the Zyte fallback, and the configured OEMs
  • Composite Inventory — the markets whose OEM feed is merged with dealer-website sources: source priority, which fields take the latest value, and column alignment
  • Inferred Sales — the markets with sales inferred from their composite inventory, and each market's windows and thresholds
  • Census Tract Boundaries — census/tiger/cb_tract: the national cartographic boundary GeoPackage, the NAD83 to EPSG:4326 transformation, the checks, and adding a vintage
  • Census ZCTA Boundaries — census/tiger/cb_zcta520: the 2020 cartographic boundary ZCTA shapefile, its columns, and the checks
  • Census Blocks — census/tiger/tabblock20: each 2020 block's housing units, population and internal point, read from the TIGER/Line TABBLOCK20 attribute tables with HTTP range requests, and the checks
  • Census States — census/tiger/state: each state's FIPS code, abbreviation, census region and division, read from the TIGER/Line state attribute table with HTTP range requests, and the checks
  • Valhalla Routing Tiles — the US routing graph for pyvalhalla: the dated Geofabrik extract each version is built from, where the archive is written, and the task size
  • Dealer Isochrones — drive-time isochrones of each franchise and the other franchises inside them: the census tracts each isochrone covers, the request settings, reusing the previous day's polygons, and the coverage checks

Candidate Generation​

The candidate set a dealer can be recommended from: real inventory configurations plus, for an OEM that declares rules, configurations mapped forward from the prior model year.

  • Synthetic Candidates — Maps prior-model-year inventory into the next year's configuration space via YAML rules; three-layer design confining OEM fields to rule authoring — the model_id-keyed core and output carry none
  • Candidate Coverage Checks — Asset checks on candidates_output: the models in scope, each model's configuration space, the per-model and per-feature baselines, and the variety checks, which measure candidate variety prioritized by demand
  • Model-Year Transitions — What changed between each base model's newest model year and the year before, per option code and per attribute

Candidate Selection​

Architecture, configuration, and scoring for packages/candidate_selection.