Skip to main content

DuckDB-WASM In-Browser Iceberg Visualizer — Design

Status: phase 0 spike PASSED (2026-07-05) — design validated end-to-end against prod; implementation not started.

Query and visualize any Iceberg table in the warehouse — on any Nessie branch — from a browser tab, with no local Python query engine, no pasted credentials, and no hosted service. Based on the approach in DuckDB: Iceberg in the browser: DuckDB-WASM plus the iceberg extension attaches to an Iceberg REST catalog and reads Parquet from object storage, entirely client-side.

Goals​

  • Browse namespaces/tables per OEM warehouse, run SQL, view results in a table grid.
  • Work against any Nessie branch (branch deployments included), selectable in the UI.
  • Reuse the existing Auth0 device-flow identity: a user's Nessie roles govern what they can see — no new credential system, no token copy-pasting (the token is read from the existing ~/.config/ppm-data-platform/nessie-token.json cache).
  • No new long-running services; no hosted app at all — the page is served by a local CLI.

Non-goals​

  • Writes from the browser (read-only visualizer).
  • Replacing Dagster/pyiceberg for pipelines; nothing in the pipeline path changes.

Phase 0 spike results (2026-07-05)​

Run against prod Nessie (0.107.6, now upgraded to 0.108.2) with a device-flow token, using duckdb-wasm 1.33.1-dev57.0 (iceberg ext build f9d5902) in a CORS-disabled Chrome, cross-checked with native DuckDB 1.5.4 (iceberg ext 75726455):

DoubtVerdict
wasm iceberg extension speaks to Nessie✅ ATTACH + full reads work. Note: no stable 1.33.0 exists on npm — use the latest dist-tag (1.33.1-dev*).
Nessie {ref}|{warehouse} prefix handling✅ with a bare /iceberg endpoint. ❌ with branch-in-endpoint: DuckDB ignores Nessie's uri config override and appends the prefix to the attach ENDPOINT, building doubled paths (/iceberg/main/v1/main%7Cmercedes/… → 404 with an empty Invalid Configuration Error). ENCODE_ENTIRE_PREFIX not needed. Worth filing upstream.
3-level namespace listing✅ SUPPORT_NESTED_NAMESPACES true lists all 185 schemas — but takes ~26 s in-browser (one REST call per namespace) and SHOW ALL TABLES returned 0 rows in ~16 s. The visualizer must build its tree from Nessie's own API, not DuckDB catalog listing.
Bearer token auth from browser✅ CREATE SECRET (TYPE ICEBERG, TOKEN …) against prod OIDC.
Branch reads✅ via a local path-rewriting proxy (see below): branch ATTACH 452 ms, count(*) on a diverged branch returned a different (correct) count than main.
S3 fallback reads (SSO creds)✅ 23.3 M-row count(*) on mercedes.mbusa.transformed.nafta_inventory (~57 s cold in-browser, 1.6 s warm). Requires INSTALL httpfs; LOAD httpfs; in wasm before CREATE SECRET (TYPE S3 …), and ACCESS_DELEGATION_MODE 'none' at ATTACH while the catalog doesn't vend credentials (the default vended_credentials mode fails when no creds come back).

Two design consequences, folded in below:

  1. Branch support requires the local server to proxy the catalog — and since that makes all catalog traffic same-origin, no Nessie CORS configuration is needed at all. Only the S3 buckets need CORS.
  2. The table tree comes from Nessie's REST API (fast, one call per level), not from DuckDB SHOW ALL TABLES.

Architecture​

The user runs nessie viz (new nessie_cli.py subcommand): a tiny HTTP server bound to 127.0.0.1 on a fixed port (19121 — fixed so bucket-CORS origins can be pinned) that serves the bundled SPA assets, a same-origin GET /token endpoint backed by the existing DeviceCodeAuthManager (cache read → silent refresh → device flow if needed), and a catalog proxy that fixes DuckDB's branch handling:

/nessie-proxy/<branch>/v1/config?…  ->  {nessie}/iceberg/<branch>/v1/config?…   (branch-scoped prefix)
/nessie-proxy/<branch>/v1/<rest>    ->  {nessie}/iceberg/v1/<rest>              (prefix carries the branch)

The rewrite is needed because Nessie's /v1/config returns a {ref}|{warehouse} prefix meant to be applied to the Iceberg base URI (advertised via the uri override), but DuckDB appends it to the attach ENDPOINT verbatim. The proxy injects the server-side bearer token on every upstream call — the browser attaches none, and the SPA's own /nessie-api calls (branch list, table tree) would otherwise be unauthenticated.

The page fetches the token on load and re-fetches on any 401, so token expiry is invisible to the user (device tokens live ~24 h; the SPA re-issues CREATE OR REPLACE SECRET + re-ATTACH). Only one credential is in play: the Auth0 bearer token (authenticates to Nessie, governs RBAC and credential vending). S3 credentials are vended by Nessie per-table in the loadTable response. Nothing is ever copy-pasted.

Local-trust note: /token and the proxy are same-origin and the server binds loopback only. Any local process could call them — but any local process can already read the cache file itself, so this adds no new exposure. The browser vector is not benign, though: a site the user visits can rebind its hostname to 127.0.0.1 and reach the loopback server through the browser to lift the token. The server therefore rejects any request whose Host isn't 127.0.0.1:19121 / localhost:19121 or whose Origin, when present, isn't the viz origin (same-origin SPA GETs omit Origin). The proxy also injects the server-side bearer token on every upstream call — the SPA attaches none, so its /nessie-api catalog/branch calls would otherwise 401.

Spike-validated session setup (issued by the SPA):

INSTALL httpfs; LOAD httpfs;                     -- wasm: required before TYPE S3 secrets
CREATE SECRET nessie (TYPE ICEBERG, TOKEN '<from /token>');
ATTACH 'mercedes' AS cat (
    TYPE ICEBERG,
    SECRET nessie,
    -- main: bare base URI; other branches: the local proxy path
    ENDPOINT 'https://nessie.app.autointel.ai/iceberg',
    -- ENDPOINT 'http://localhost:19121/nessie-proxy/<branch>',
    SUPPORT_NESTED_NAMESPACES true
);
SELECT count(*) FROM cat."mercedes.mbusa.transformed".nafta_inventory;

Namespaces are three levels (oem.market.tier); duckdb-iceberg exposes them as a single quoted, dot-flattened schema.

Branch support​

The SPA lists branches via GET {nessie}/api/v2/trees (same bearer token, via the local proxy) and re-ATTACHes through /nessie-proxy/<branch> on selection. Branch deployments therefore work with no extra machinery. Validated in the spike against a real diverged branch.

S3 credentials (Nessie credential vending)​

Nessie vends short-lived, per-table-scoped S3 credentials in the loadTable REST response. The ATTACH uses ACCESS_DELEGATION_MODE 'vended_credentials' so DuckDB requests and uses vended credentials automatically — no separate AWS secret needed. RBAC determines the scope: reader tokens get read-only creds, service/admin get read-write. Enabled in Nessie 0.108.2 (fix for projectnessie/nessie#12703).

Required changes​

1. Warehouse bucket CORS (terraform — nessie_s3.tf)​

The only infra change needed for the fallback-credentials mode. aws_s3_bucket_cors_configuration on each ai-app-<oem>-iceberg-prod bucket: GET, HEAD from http://localhost:19121 / http://127.0.0.1:19121, AllowedHeaders: ["*"] (covers Authorization and Range — SigV4 fetches always trigger preflight), and ExposeHeaders: ["ETag", "Content-Range", "Accept-Ranges", "Content-Length"] — DuckDB reads Parquet via HTTP Range requests and browser JS cannot read non-safelisted response headers. Bucket policy and public-access blocks are untouched — CORS only permits the browser to make signed requests; the credentials still authorize them.

(No Nessie/Quarkus CORS config is needed: all catalog traffic is same-origin through the nessie viz proxy.)

2. Credential vending (terraform — nessie_iam.tf + nessie_ecs.tf)​

Per-OEM vending roles in nessie_iam.tf; client-IAM env vars in nessie_ecs.tf. Enabled with Nessie 0.108.2 (fix for projectnessie/nessie#12703).

3. nessie viz CLI subcommand (implemented — ai_core.nessie_cli + ai_core.nessie_viz)​

Local server, no hosting, no new Auth0 configuration:

  • Binds 127.0.0.1:19121, serves the SPA HTML (bundled as package data at ai_core/viz/index.html). The duckdb-wasm engine itself loads from the jsDelivr CDN — the one external dependency; bundling the multi-MB wasm into the repo isn't worth it for an internal dev tool (and the large-file pre-commit hook would reject it).
  • GET /token via the existing DeviceCodeAuthManager (cache read + silent refresh; runs the device flow on first use if no cache exists).
  • GET /aws-credentials from the ambient boto3 chain (legacy fallback; unused now that credential vending is enabled).
  • Catalog proxy (/nessie-proxy/<branch>, /nessie-api/) with the branch rewrite above; injects the server-side bearer token and forwards the X-Iceberg-Access-Delegation header on every upstream call.
  • Opens the browser tab on start.
  • UX: uv run --directory packages/ai_core nessie viz → tab opens, already authenticated.

The command reads its Nessie config from the environment (NESSIE_URI, NESSIE_AUTH0_DOMAIN, NESSIE_DEVICE_CLIENT_ID, NESSIE_AUTH0_API_AUDIENCE). The repo loads these via direnv (the gitignored root .envrc), so once that's set up nessie viz picks them up from the shell with no extra flag.

The code itself does not auto-load a .env file. If you keep the Nessie config in a standalone .env instead of .envrc, pass it through uv:

uv run --env-file .env --directory packages/ai_core nessie viz

4. Visualizer SPA​

  • duckdb-wasm latest dist-tag (1.33.1-dev*; no stable 1.33.0 exists) + a minimal UI: warehouse (OEM) dropdown, branch dropdown, table tree, SQL editor, result grid.
  • Table tree and branch list from Nessie's REST API via the proxy — not DuckDB catalog listing (~26 s for 185 namespaces, and SHOW ALL TABLES returned nothing).
  • On load: fetch /token, LOAD httpfs, create Iceberg secret, ATTACH with ACCESS_DELEGATION_MODE 'vended_credentials'; re-fetch + re-attach on 401 or branch/warehouse switch. S3 creds vended per-table by Nessie.
  • All query compute and data stay in the browser; tokens held in memory only.
  • Result grid and table tree render via textContent / DOM APIs, never innerHTML: cell values and catalog names ultimately originate from third-party OEM data and must not be parseable as markup (an XSS payload in a row could otherwise read /token).

5. Hosted variant (ECS sidecar)​

Hosted at https://nessie.app.autointel.ai/viz/ via a sidecar container in the Nessie ECS task. Path-based ALB routing (/viz* → sidecar:19121). Auth0 PKCE replaces the device-flow token injection; S3 credentials are vended by Nessie via RBAC-scoped credential vending. See hosted deploy.

Rollout​

Delivered as one PR, but enabled in stages — phase 2's Nessie config must not reach prod until the branch-deployment vending test is green.

PhaseContentsRisk
0✅ Compatibility spike — passed 2026-07-05, see results above—
1✅ Bucket CORS + nessie viz CLI + SPANone — additive headers, local-only code.
2✅ Credential vending — IAM roles + client-IAM enabled with Nessie 0.108.2 (nessie#12703 fixed)Low — roles already provisioned, just env var + version bump
3✅ Hosted variant — ECS sidecar on Nessie task, Auth0 PKCE, Nessie credential vendingLow — additive infra, same ALB, no new domain. See hosted deploy.

Verification​

  1. Phase 1: browser page loads a table read on main and on a branch without --disable-web-security (proves bucket CORS + proxy).

Open questions​

  • Result-set size guardrails in the SPA (DuckDB-WASM memory is browser-bounded; the 23.3 M-row count took ~57 s cold over residential broadband).
  • Upstream duckdb-iceberg issue for the ignored uri config override / prefix double-append: filed as duckdb-iceberg#1145 — once fixed, the proxy can be simplified away.