Skip to main content

Getting Started

Prerequisites — direnv​

The repo uses direnv to inject shell environment variables automatically when you cd into the project. Install it once:

# macOS
brew install direnv

# Ubuntu / Debian
sudo apt install direnv

Then add the hook to your shell profile and restart your shell:

# ~/.zshrc  (or ~/.bashrc for bash)
eval "$(direnv hook zsh)" # replace zsh with bash if needed

Finally, allow the repo's .envrc from the project root:

direnv allow

direnv will load the variables automatically every time you enter the directory. The .envrc file is gitignored — never commit it.

note

direnv allow must be re-run whenever .envrc changes.

Installation via uv sync​

Each package manages its own env. Run uv sync --directory <path> once per package:

uv sync --directory packages/ai_core
uv sync --directory projects/ai_<oem> # e.g. projects/ai_audi
uv sync --directory deployments/local # Dagster UI tooling

Environment Setup​

Each OEM project needs a .env file in its project root (e.g. projects/ai_audi/.env). This file is gitignored — never commit it.

Create the file with the following variables, replacing <REPO> with the absolute path to your local clone of the repo:

VariablePurposeExample value
ICEBERG_CATALOG_URISQLite catalog for local Iceberg tablessqlite:////<REPO>/temp/iceberg/catalog.db (macOS/Linux) or sqlite:///<REPO>/temp/iceberg/catalog.db (Windows)
ICEBERG_WAREHOUSEDirectory for Iceberg data filesfile:///<REPO>/temp/iceberg/warehouse (macOS/Linux) or file://<REPO>/temp/iceberg/warehouse (Windows)
RAW_STORAGE_URIDirectory for raw JSON responsestemp/raw
DUCKDB_PATHPath for OEM DuckDB filetemp/audi.duckdb
PPM_SOURCE_OVERRIDESPath to source override YAML (optional)temp/source_overrides.yaml
note

The Iceberg URI formats differ by platform. macOS/Linux uses four slashes after sqlite: (sqlite:////abs/path), while Windows uses three (sqlite:///C:/path). The same applies to ICEBERG_WAREHOUSE — macOS/Linux uses file:///, Windows uses file://.

The Iceberg directories are created automatically on first run if they don't exist, so you don't need to create them manually.

Connecting to the production Nessie catalog​

Set NESSIE_URI=https://nessie.app.autointel.ai/ and ICEBERG_CATALOG_TYPE=nessie (the default) to connect directly to the production Nessie catalog.

Nessie enforces authentication and role-based access control. There are three roles: reader (read-only), service (read/write, used by pipelines), and admin. For local inspection, set NESSIE_AUTH=device and run uv run --directory packages/ai_nessie nessie login once to authenticate through your browser. The token is cached to disk and refreshed silently, so re-login is rare. load_nessie_catalog raises NessieLoginRequired if no valid token is cached. RBAC enforces your role regardless of what you attempt to write. See Nessie for the device-flow env vars and full setup.

The production pipeline and CI do not mint tokens. They read a shared service token from AWS Secrets Manager (NESSIE_AUTH=cached), refreshed automatically every 23h by a Lambda — developers never manage it. This is configured through each project's projects/ai_<oem>/container_context.yaml and the CI workflow, not anything you set locally.

Migrating from the old M2M auth

NESSIE_AUTH=auth0 raises an error. Remove it along with NESSIE_AUTH0_CLIENT_ID and NESSIE_AUTH0_CLIENT_SECRET from your .env, and add NESSIE_AUTH=device plus NESSIE_DEVICE_CLIENT_ID (the nessie-cli app). Run uv run --directory packages/ai_nessie nessie login once to complete the browser login before starting dg dev.

Individual OEM projects may define additional environment variables — check the project's README for details.

Prerequisites — Postman MCP​

The public-api-discovery workflow creates and maintains Postman collections as a required artifact of API discovery — the Postman workspace is the team's authoritative record of every endpoint an OEM exposes, ingested or not. So a postman:<entity> gate must be successful before the public-api-implement workflow will run. The Postman MCP server is configured in .mcp.json at the repo root and uses OAuth — no API key or manual installation is needed.

First-time setup:

On first use, Claude Code will prompt you to authenticate with Postman through your browser. Follow the OAuth flow and accept the requested permissions. The session is cached — re-authentication is rare.

If the OAuth prompt does not appear or the tools are unavailable, run:

claude mcp add --transport http postman https://mcp.postman.com/mcp

Without a valid Postman session the document skill does not silently skip the collection. It writes a Postman Collection v2.1 JSON file under docs/content/oems/<oem>/postman/ and flags each postman:<entity> gate as manual_import (pointing at that file). This is non-blocking — discovery still passes the gate — but the file must be imported into the workspace by hand so the team's Postman record stays complete. Re-running the document skill once the MCP session is restored upserts the same requests and flips the gates to pass.

Running Dagster Locally​

Start the Dagster UI from the repo root:

deployments/local/.venv/bin/dg dev

The UI is available at http://localhost:3000.

note

The deployments/local venv is only for the dg dev UI server. Do not use it to materialize assets — it doesn't have OEM project modules installed.

Rematerializing After Code Changes​

After changing entity classes, component logic, or YAML config, you only need to rematerialize the affected tiers — not re-collect raw data. Raw assets represent point-in-time API fetches that Dagster reads from Iceberg. They do not need to be re-run unless you specifically need fresh source data.

# Rematerialize only transformed + downstream for a specific partition
cd projects/ai_<oem>
uv run dg launch \
--assets 'key:"stellantis/transformed/...",key:"stellantis/consolidated/..."' \
--partition YYYY-MM-DD

Dagster resolves upstream dependencies from whatever is already in Iceberg.

Clean Slate (Nuclear Reset)​

warning

reset_iceberg_local.sh deletes all local Iceberg data including raw. Only use this when a breaking schema change requires dropping all tables. For normal development, use dg launch --assets to rematerialize only what changed.

To surgically drop a single table without losing everything:

sqlite3 temp/iceberg/catalog.db \
"DELETE FROM iceberg_tables WHERE table_name='<table>';"
rm -rf temp/iceberg/warehouse/stellantis/<table>/

1. Stop Dagster if it's running (Ctrl+C in the terminal).

2. Reset Iceberg warehouse and catalog:

# macOS / Linux
./scripts/reset_iceberg_local.sh

# Windows
scripts\reset_iceberg_local.bat

3. Remove OEM-specific DuckDB files (if any):

rm -f temp/<oem>.duckdb temp/iceberg_views.duckdb     # macOS / Linux
del temp\<oem>.duckdb temp\iceberg_views.duckdb 2>nul # Windows

4. Rebuild the OEM venv (picks up any ai_core / ai_http / ai_dagster changes):

uv sync --directory projects/ai_<oem> \
--reinstall-package ai-core \
--reinstall-package ai-http \
--reinstall-package ai-dagster

5. Start Dagster:

deployments/local/.venv/bin/dg dev      # macOS / Linux
deployments\local\.venv\Scripts\dg.exe dev # Windows

Materializing Assets​

There are two ways to materialize assets depending on whether a Dagster webserver is running.

Direct launch (no webserver needed)​

Use the OEM project's own venv to invoke dg launch directly. The CLI creates a local instance without requiring a running webserver:

macOS / Linux

cd projects/ai_audi
source .venv/bin/activate
dg launch --assets 'key:"audi/*"' --partition 2026-01-27

Windows (PowerShell)

cd projects/ai_audi
.venv\Scripts\dg.exe launch --assets 'key:"audi/*"' --partition 2026-01-27

Replace the partition date with today's date in YYYY-MM-DD format.

Webserver submission (requires running dg dev)​

If dg dev is already running, use the /dagster-run skill to submit and monitor a run:

/dagster-run

Describe what you want to materialize and the skill will submit the run, track step progress, and surface any errors — all from Claude Code.

Monitoring a Run​

Use the /dagster-run skill to monitor progress or investigate a failure on any run, whether it was launched via the UI, dg launch, or the skill itself:

/dagster-run

The skill can poll a run until it reaches a terminal state and collect structured error output and tracebacks if it fails. The run ID appears in the Dagster UI URL (/runs/<run-id>) or in dg launch output.

Running Tests​

Tests use pytest under Pants for Dagster/OEM code; Pants is the source of truth for those CI checks. From the repo root:

# Test the slice of code affected by your branch (matches PR CI)
pants --changed-since=origin/main --changed-dependents=transitive check test

# Or, scope explicitly to a package / project
pants test 'packages/ai_core/tests::'
pants test 'projects/ai_<oem>/tests::'

# Type-check
pants check 'packages/ai_core::'

# Full sweep
pants check test ::

Add -- -v to forward verbose flags to pytest:

pants test 'packages/ai_core/tests::' -- -v

uv run --directory <pkg> pytest and uv run --directory <pkg> mypy work for fast individual-package iteration, but Pants is the build system CI uses and the one that catches cross-resolve dependency drift.

Pants prerequisites​

Run the repo setup script once; it checks for uv and Pants, installs the scie-pants launcher if needed, warns about ~/.local/bin, and installs hooks:

scripts/setup-dev.sh

The first pants <command> run in the repo bootstraps Pants 2.33.0 automatically (a few minutes; cached after that).

Lockfile hygiene​

When you modify pyproject.toml, regenerate both lockfiles to keep Docker/dev (uv) and CI/Pants in sync:

uv lock --directory <package_or_project>
pants generate-lockfiles --resolve=<resolve_name>

CI fails if either is stale.