Nessie

Project Nessie is the Iceberg REST catalog for the platform. All three pipeline tiers — raw, transformed, and consolidated — write to Iceberg tables registered in Nessie. S3 holds the data files; Nessie holds the catalog metadata and provides atomic branching and merging of table state so branch deployments can run without touching production data.

Deployment

Nessie runs as a Fargate service in the AWS VPC (account 999655274916, region us-east-1), registered via Cloud Map for service discovery within the VPC. The public endpoint is fronted by an ALB for TLS termination; Nessie itself enforces OIDC authentication and CEL-based authorization (RBAC) on every request.

Connectivity

How you connect depends on where the code runs:

Context	NESSIE_URI	Authentication
ECS code-location tasks (inside VPC)	http://nessie.ppm.internal:19120	Cached service token (NESSIE_AUTH=cached)
Local development	https://nessie.app.autointel.ai/	Per-user device flow (NESSIE_AUTH=device)
CI / branch deployments	https://nessie.app.autointel.ai/	Cached service token (read from Secrets Manager)

Nessie validates the bearer token on every request — including the VPC-internal path — and applies role-based authorization, so all clients authenticate.

Roles

Tokens carry a namespaced roles claim (https://nessie-catalog/roles):

• reader — read-only across the catalog
• service — read/write, including commits and merges to main; used by the pipeline and CI
• admin — full administrative access, including main

Authorization is enforced by Nessie's CEL rules (defined in config/nessie/prod.json.tftpl): reads require any of reader / writer / service / admin; writes require service or admin; an optional writer role may write to non-main branches only. Tokens with no Nessie role are rejected even on reads — assign at least reader in Auth0 for any human who needs to inspect the catalog.

Local development — device flow

Human console access uses the OAuth2 Device Authorization Flow via the nessie CLI. Set these env vars (add to your .envrc or shell profile):

NESSIE_AUTH=device
NESSIE_AUTH0_DOMAIN=autointel-admin.us.auth0.com
NESSIE_DEVICE_CLIENT_ID=<nessie-cli Native app client ID>
NESSIE_AUTH0_API_AUDIENCE=https://nessie-catalog/

Then authenticate explicitly before using the catalog:

uv run --directory packages/ai_core nessie login

This prints a URL + code, opens your browser, and caches the access and refresh tokens to ~/.config/ppm-data-platform/nessie-token.json. Subsequent calls to load_nessie_catalog use the cached token silently; the refresh token keeps it valid so re-login is rare.

If no valid token is cached, load_nessie_catalog raises NessieLoginRequired — run nessie login and retry. RBAC enforces your role regardless of what you attempt to write.

Pipeline and CI — cached service token

The pipeline and CI do not mint a token per run. A Lambda mints one service token every 23h and stores it in AWS Secrets Manager (dagster/nessie/m2m-token); clients set NESSIE_AUTH=cached and read it. This is wired in each project's container_context.yaml and the CI workflow — not something a local user sets.

How the platform uses it

All Iceberg IO routes through build_iceberg_io_manager in ai_core.io_managers.iceberg. The catalog backend is selected by ICEBERG_CATALOG_TYPE (default nessie). When nessie, NESSIE_URI must be set and the IO manager connects to the Nessie REST catalog. The active catalog branch is derived from the Dagster deployment name in branch deployments and falls back to main otherwise. For local development, set ICEBERG_CATALOG_TYPE=sql in the OEM project's .env.

Utilities

packages/ai_core/src/ai_core/nessie.py provides lightweight helpers for working with Nessie outside of the Dagster IO manager:

• nessie_catalog_properties — builds the pyiceberg catalog property dict for a given URI, warehouse, and branch; includes the auth block when auth is supplied
• load_nessie_catalog — constructs and returns a pyiceberg Catalog from env vars; used when code needs to interact with Iceberg tables directly
• nessie_client / NessieClient — thin wrapper around the Nessie REST API v2 for branch management (create, delete, merge); auth is resolved from the NESSIE_AUTH env vars automatically

Environment variables

Variable	Purpose
NESSIE_URI	Nessie base URI. Required when ICEBERG_CATALOG_TYPE=nessie (the default).
NESSIE_AUTH	Auth strategy: device (per-user console login) or cached (shared service token).
NESSIE_AUTH0_DOMAIN	Auth0 tenant domain (autointel-admin.us.auth0.com). Required for device.
NESSIE_DEVICE_CLIENT_ID	Auth0 nessie-cli Native app client ID (non-secret). Required for device.
NESSIE_AUTH0_API_AUDIENCE	Auth0 API audience (https://nessie-catalog/). Required for device.
NESSIE_TOKEN_SECRET_ID	Secrets Manager secret holding the cached service token. Used by cached (default dagster/nessie/m2m-token).

Troubleshooting

Symptom: every Nessie call returns 401 across all OEMs

Every materialization fails with 401 Unauthorized (visible in Dagster step logs), and the client side raises Bearer … token rejection. CI's Read cached Nessie token step may also fail with an empty-token guard.

Diagnostic order:

1. Is the cached token populated?

   aws secretsmanager get-secret-value \
     --secret-id dagster/nessie/m2m-token \
     --query SecretString --output text | jq .

   • Empty / null / missing access_token → see "Token secret empty" below.
   • JSON with expires_at in the past → the Lambda hasn't run successfully today.
2. Has the refresher Lambda failed? Check the nessie-token-refresher-errors CloudWatch alarm, the nessie-token-refresher-dlq-messages alarm, and the Lambda's CloudWatch log group /aws/lambda/nessie-token-refresher.
3. Is Auth0 healthy? The most recent log line in the Lambda will include the Auth0 error / error_description if the M2M grant was rejected (e.g. unauthorized_client, invalid_client). An Auth0 outage shows up as URLError with retry exhaustion.
4. Has the token claim changed? If QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH was edited or quoting was dropped, Quarkus will fail to resolve roles and every write silently 403s. Decode a recent service token at jwt.io and confirm it carries "https://nessie-catalog/roles": ["service"].

Recovery — force a fresh refresh:

aws lambda invoke \
  --function-name nessie-token-refresher \
  --payload '{}' --cli-binary-format raw-in-base64-out \
  /dev/stdout

A successful response is {"ok": true, "expires_at": <epoch>}. The Lambda re-fills dagster/nessie/m2m-token; clients pick up the new value on their next auth_header() call (within the reread_throttle window, default 30s).

Symptom: terraform apply failed on aws_lambda_invocation.nessie_token_refresher_bootstrap

This is the fail-fast bootstrap firing on first apply. The Lambda's response carries the redacted Auth0 error.

Common causes:

• dagster/nessie/m2m-service-client-secret was not filled before applying the dagster-agent solution. Fill it (see the "Auth0 setup" section in deployments/aws/terraform/README.md) and re-run terraform apply.
• The nessie-service M2M app is not authorized for the Nessie API in Auth0. Add it to the API's "Machine to Machine Applications" tab and re-apply.
• The Client-Credentials Action doesn't add the service role to the token. Without it, the token is valid but role-less and every write is denied. Confirm the Action source matches the snippet in terraform/README.md.

The Lambda's CloudWatch log group has the raw Auth0 response — useful when the Terraform output isn't specific enough.

Symptom: device-flow login succeeds but writes are denied

A human user has authenticated through nessie-cli (Auth0 device flow) but every commit / branch operation returns 403.

Cause: the user is missing a Nessie role assignment in Auth0. The Login Action injects only the roles the user is actually assigned; an unassigned user gets an empty roles claim and matches no CEL rule.

Recovery: assign the user reader (read-only) or admin (full access) in the Auth0 dashboard → Users → Roles. They'll need to re-authenticate — delete ~/.config/ppm-data-platform/nessie-token.json and run nessie login again.