Scaffolding a New OEM

This guide covers everything required to add a new OEM to the platform. The platform runs on a Dagster hybrid ECS deployment — code locations run as Fargate tasks in our AWS VPC, not on Dagster's managed serverless infrastructure. Adding a new OEM therefore requires both Python scaffolding and AWS infrastructure changes, all of which must land in the same PR.

Overview

Step	What it does
1. Copier scaffold	Generates the Python project skeleton + Pants `BUILD` files under `projects/ai_<oem>/`
2. Dagster workspace	Registers the project in `dg.toml` and `dagster_cloud.yaml`
3. Pants resolve	Registers the resolve in `pants.toml` and generates `3rdparty/python/ai_<oem>.lock`
4. Terraform	Registers the OEM for S3, ECR, derived Nessie config, and CloudFormation IAM
5. Dockerfile	PEX-based container image built from the repo root after `pants package`
6. Verify CI	No workflow edits — CI and deploy auto-discover the new project from `pants.toml`

The affected-component matrix (scripts/pants_affected_components.py) and pants.toml's [python.resolves] are the source of truth for the OEM list; no separate CI test matrix needs editing.

The /scaffold-oem Claude skill automates steps 1–3 and walks through 4–6 interactively. To scaffold a new OEM, ask Claude:

"Scaffold a new OEM project for BMW"

Step 1 — Copier scaffold

Run from the repo root:

uvx copier copy \
  .claude/skills/scaffold-oem/template \
  projects/ai_<oem> \
  --data oem_name=<oem> \
  --data oem_display_name="<Display Name>" \
  --data start_date=<YYYY-MM-DD> \
  --defaults

The template generates:

pyproject.toml — dependencies and uv.sources path reference to ai_core
definitions.py — calls load_from_defs_folder to auto-discover YAML components
components/raw.py — skeleton for RawSourceComponent subclasses
sources/ — empty package for pure-Python API client modules
defs/raw/defs.yaml, defs/transformed/defs.yaml, defs/consolidated/defs.yaml — placeholder pipeline components
tests/test_<oem>.py — YAML validation test that calls load_from_defs_folder
BUILD, src/BUILD, tests/BUILD — Pants targets (requirements, sources, tests)
build.yaml — ECR registry pointer (999655274916.dkr.ecr.us-east-1.amazonaws.com/ai-<oem>)
Dockerfile — PEX-based image; see Step 5 if the project uses cosy-encryption

Step 2 — Register in `dg.toml` and `dagster_cloud.yaml`

dg.toml — append:

[[workspace.projects]]
path = "projects/ai_<oem>"

dagster_cloud.yaml — append to the locations list:

- location_name: ai_<oem>
  code_source:
    package_name: ai_<oem>
  build:
    directory: ./projects/ai_<oem>

Step 3 — Register the Pants resolve

pants.toml — under [python.resolves], append:

ai_<oem> = "3rdparty/python/ai_<oem>.lock"

The shared_source_targets Pants backend auto-generates the per-resolve reqs_<pkg>_<resolve> and src_for_<pkg>_<resolve> targets for every shared package the project declares as a dependency, as soon as the resolve appears in [python.resolves] — no additional BUILD files are needed.

Then generate the lockfile and sync the uv venv:

pants generate-lockfiles --resolve=ai_<oem>
uv sync --directory projects/ai_<oem>

Verify the scaffold loads:

pants check 'projects/ai_<oem>/src::'
pants test 'projects/ai_<oem>/tests::'

scripts/pants_affected_components.py derives its OEM list from pants.toml automatically — no edit needed there.

Step 4 — Terraform

All AWS infrastructure for a new OEM is controlled by files in deployments/aws/terraform/ and deployments/aws/. Two changes are required.

4a. S3 bucket + ECR repository

Add the new OEM name to local.oems in deployments/aws/terraform/solutions/dagster-agent/locals.tf:

locals {
  oems = toset(["audi", "mercedes", "stellantis", "nissan", "gm", "ford", "<oem>"])
}

This single change covers both the S3 Iceberg warehouse (bucket name derived automatically as ai-app-<oem>-iceberg-prod) and the ECR repository (ai-<oem>). All associated resources — S3 versioning, encryption, public-access-block, ECR lifecycle policy, and the Nessie warehouse env var — expand automatically. The ecr_repository_urls output map is updated automatically.

4b. CloudFormation IAM

Add the new bucket to the UserCodeExecutionRole inline policy in deployments/aws/cloudformation/ecs-agent-vpc-private.yaml. This role is assumed by Dagster user-code ECS tasks; without it, asset materializations will fail with an S3 AccessDenied error. Add both the object-level and bucket-level ARNs:

              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:DeleteObject
                Resource:
                  - "arn:aws:s3:::ai-app-<oem>-iceberg-prod/*"
                  # ... existing entries ...
              - Effect: Allow
                Action:
                  - s3:ListBucket
                  - s3:GetBucketLocation
                Resource:
                  - "arn:aws:s3:::ai-app-<oem>-iceberg-prod"
                  # ... existing entries ...

Deploy the dagster-agent Terraform solution after these changes to create the S3 bucket and ECR repository and update the Nessie task definition before the first CI build tries to push to it.

Step 5 — Dockerfile

The Copier template generates the PEX-based Dockerfile used by CI. The build context in CI is the repo root because pants package writes the PEX under dist/ before the image build:

FROM python:3.13-slim

WORKDIR /app

COPY dist/projects.ai_<oem>.src/pex.pex /app/pex.pex

RUN chmod +x /app/pex.pex && /app/pex.pex venv /app/venv

ENV PATH="/app/venv/bin:$PATH"

EXPOSE 4000

Never build from inside the project directory.

Projects that use `cosy-encryption` (Nexus private index)

Add the private package to the project dependencies and point uv at Nexus:

dependencies = [
    "ai_core",
    "cosy-encryption==<version>",
    "dagster-cloud",
]

[[tool.uv.index]]
name = "nexus"
url = "https://nexus.sulzer-us.com/repository/pypi-private/simple/"
priority = "explicit"

[tool.uv.sources]
ai_core = { path = "../../packages/ai_core", editable = true }
cosy-encryption = { index = "nexus" }

If the package exposes modules that do not match the distribution name, add a module_mapping entry to the project's python_requirements target in BUILD.

After changing pyproject.toml, regenerate both lockfiles before verifying or deploying:

pants generate-lockfiles --resolve=ai_<oem>
uv lock --directory projects/ai_<oem>

Do not change the Dockerfile for Nexus credentials. CI writes Nexus credentials to ~/.netrc before running pants package, so private wheels are resolved into the PEX before docker build starts. No Docker BuildKit secret mounts are needed.

Step 6 — Verify CI

No workflow edits are needed. CI (.github/workflows/ci.yml) — including its build and deploy jobs — derives the full list of OEM projects from pants.toml via scripts/pants_affected_components.py. Once Step 3 is complete, the new project is automatically included in lint, check, test, lockfile validation, and deploy targeting.

After the PR merges

Run the Deploy Dagster Infrastructure workflow (dagster-agent workspace) to apply the Terraform changes — this creates the S3 bucket, ECR repository, and updates the Nessie task definition.
The next hybrid deploy CI run will build and push the new OEM image and register it with Dagster Cloud.

Implementing data sources

Once the scaffold is merged, implement the OEM's data sources:

Add source modules in sources/ — pure Python, no Dagster imports
Implement RawSourceComponent subclasses in components/raw.py (see ai_audi for reference)
Replace the placeholder YAML with real datasets pointing at the new subclasses
Write source tests under tests/sources/

See Code Structure for the full implementation guide.

Overview​

Step 1 — Copier scaffold​

Step 2 — Register in dg.toml and dagster_cloud.yaml​

Step 3 — Register the Pants resolve​

Step 4 — Terraform​

4a. S3 bucket + ECR repository​

4b. CloudFormation IAM​

Step 5 — Dockerfile​

Projects that use cosy-encryption (Nexus private index)​

Step 6 — Verify CI​

After the PR merges​

Implementing data sources​