Skip to main content

DuckDB-WASM In-Browser Iceberg Visualizer — Design

Query and visualize any Iceberg table in the warehouse — on any Nessie branch — from a browser tab, with no local Python query engine, no pasted credentials, and no hosted service. Based on the approach in DuckDB: Iceberg in the browser: DuckDB-WASM plus the iceberg extension attaches to an Iceberg REST catalog and reads Parquet from object storage, entirely client-side.

Goals​

  • Browse namespaces/tables per OEM warehouse, run SQL, view results in a table grid.
  • Work against any Nessie branch (branch deployments included), selectable in the UI.
  • Reuse the existing Auth0 device-flow identity: a user's Nessie roles govern what they can see — no new credential system, no token copy-pasting (the token is read from the existing ~/.config/ppm-data-platform/nessie-token.json cache).
  • No new long-running services; no hosted app at all — the page is served by a local CLI.

Non-goals​

  • Writes from the browser (read-only visualizer).
  • Replacing Dagster/pyiceberg for pipelines; nothing in the pipeline path changes.

Phase 0 spike results (2026-07-05)​

Run against prod Nessie (0.107.6, now upgraded to 0.108.2) with a device-flow token, using duckdb-wasm 1.33.1-dev57.0 (iceberg ext build f9d5902) in a CORS-disabled Chrome, cross-checked with native DuckDB 1.5.4 (iceberg ext 75726455):

QuestionResult
wasm iceberg extension speaks to Nessie✅ ATTACH + full reads work. Note: no stable 1.33.0 exists on npm — use the latest dist-tag (1.33.1-dev*).
Nessie {ref}|{warehouse} prefix handling✅ with a bare /iceberg endpoint. ❌ with branch-in-endpoint: DuckDB ignores Nessie's uri config override and appends the prefix to the attach ENDPOINT, building doubled paths (/iceberg/main/v1/main%7Cmercedes/… → 404 with an empty Invalid Configuration Error). ENCODE_ENTIRE_PREFIX not needed. Worth filing upstream.
3-level namespace listing✅ SUPPORT_NESTED_NAMESPACES true lists all 185 schemas — but takes ~26 s in-browser (one REST call per namespace) and SHOW ALL TABLES returned 0 rows in ~16 s. The visualizer must build its tree from Nessie's own API, not DuckDB catalog listing.
Bearer token auth from browser✅ CREATE SECRET (TYPE ICEBERG, TOKEN …) against prod OIDC.
Branch reads✅ via a local path-rewriting proxy (see below): branch ATTACH 452 ms, count(*) on a diverged branch returned a different (correct) count than main.
S3 fallback reads (SSO creds)✅ 23.3 M-row count(*) on mercedes.mbusa.transformed.nafta_inventory (~57 s cold in-browser, 1.6 s warm). Requires INSTALL httpfs; LOAD httpfs; in wasm before CREATE SECRET (TYPE S3 …), and ACCESS_DELEGATION_MODE 'none' at ATTACH while the catalog doesn't vend credentials (the default vended_credentials mode fails when no creds come back).

Two design consequences, folded in below:

  1. Branch support requires the local server to proxy the catalog — and since that makes all catalog traffic same-origin, no Nessie CORS configuration is needed at all. Only the S3 buckets need CORS.
  2. The table tree comes from Nessie's REST API (fast, one call per level), not from DuckDB SHOW ALL TABLES.

Architecture​

The user runs nessie viz (an ai_nessie/cli.py subcommand): a tiny HTTP server bound to 127.0.0.1 on a fixed port (19121 — fixed so bucket-CORS origins can be pinned). It:

  • serves the bundled SPA assets
  • exposes a same-origin GET /token endpoint backed by the existing DeviceCodeAuthManager (cache read → silent refresh → device flow if needed)
  • provides a catalog proxy that fixes DuckDB's branch handling:
/nessie-proxy/<branch>/v1/config?…  ->  {nessie}/iceberg/<branch>/v1/config?…   (branch-scoped prefix)
/nessie-proxy/<branch>/v1/<rest>    ->  {nessie}/iceberg/v1/<rest>              (prefix carries the branch)

The rewrite is needed because Nessie's /v1/config returns a {ref}|{warehouse} prefix meant to be applied to the Iceberg base URI (advertised via the uri override), but DuckDB appends it to the attach ENDPOINT verbatim. The proxy injects the server-side bearer token on every upstream call — the browser attaches none, and the SPA's own /nessie-api calls (branch list, table tree) would otherwise be unauthenticated.

The page fetches the token on load and re-fetches on any 401, so token expiry is invisible to the user (device tokens live ~24 h; the SPA re-issues CREATE OR REPLACE SECRET + re-ATTACH). Only one credential is in play: the Auth0 bearer token (authenticates to Nessie, governs RBAC and credential vending). S3 credentials are vended by Nessie per-table in the loadTable response. Nothing is ever copy-pasted.

Local-trust note: /token and the proxy are same-origin and the server binds loopback only. Any local process could call them — but any local process can already read the cache file itself, so this adds no new exposure. The browser vector is not benign, though: a site the user visits can rebind its hostname to 127.0.0.1 and reach the loopback server through the browser to lift the token. The server therefore rejects any request whose Host isn't 127.0.0.1:19121 / localhost:19121 or whose Origin, when present, isn't the viz origin (same-origin SPA GETs omit Origin). The proxy also injects the server-side bearer token on every upstream call — the SPA attaches none, so its /nessie-api catalog/branch calls would otherwise 401.

Spike-validated session setup (issued by the SPA):

INSTALL httpfs; LOAD httpfs;                     -- wasm: required before TYPE S3 secrets
CREATE SECRET nessie (TYPE ICEBERG, TOKEN '<from /token>');
ATTACH 'mercedes' AS cat (
    TYPE ICEBERG,
    SECRET nessie,
    -- main: bare base URI; other branches: the local proxy path
    ENDPOINT 'https://nessie.app.autointel.ai/iceberg',
    -- ENDPOINT 'http://localhost:19121/nessie-proxy/<branch>',
    SUPPORT_NESTED_NAMESPACES true
);
SELECT count(*) FROM cat."mercedes.mbusa.transformed".nafta_inventory;

Namespaces are three levels (oem.market.tier). duckdb-iceberg exposes them as a single quoted, dot-flattened schema.

The ATTACH identifier is the Nessie warehouse, which usually — but not always — equals the namespace's first segment. The SPA resolves it per-entry from the entries?content=true response's content.metadataLocation (s3://ai-app-<warehouse>-iceberg-prod/..., per the Terraform-managed bucket-per-warehouse convention in nessie_s3.tf), falling back to the first namespace segment when that field is absent or unparseable. This matters for ai_public: it calls build_core_defs("public"), so every table it writes lives in warehouse "public" regardless of its asset-key prefix (e.g. census.cbp.cbp_county still attaches as "public", not "census").

Branch support​

The SPA lists branches via GET {nessie}/api/v2/trees (same bearer token, via the local proxy) and re-ATTACHes through /nessie-proxy/<branch> on selection. Branch deployments therefore work with no extra machinery. Validated in the spike against a real diverged branch.

S3 credentials (Nessie credential vending)​

Nessie vends short-lived, per-table-scoped S3 credentials in the loadTable REST response. The ATTACH uses ACCESS_DELEGATION_MODE 'vended_credentials' so DuckDB requests and uses vended credentials automatically — no separate AWS secret needed. RBAC determines the scope: reader tokens get read-only creds, service/admin get read-write. Enabled in Nessie 0.108.2 (fix for projectnessie/nessie#12703).

Required changes​

1. Warehouse bucket CORS (terraform — nessie_s3.tf)​

The only infra change needed for the fallback-credentials mode. aws_s3_bucket_cors_configuration on each ai-app-<oem>-iceberg-prod bucket:

  • GET, HEAD from http://localhost:19121 / http://127.0.0.1:19121
  • AllowedHeaders: ["*"] (covers Authorization and Range — SigV4 fetches always trigger preflight)
  • ExposeHeaders: ["ETag", "Content-Range", "Accept-Ranges", "Content-Length"] — DuckDB reads Parquet via HTTP Range requests and browser JS cannot read non-safelisted response headers

Bucket policy and public-access blocks are untouched — CORS only permits the browser to make signed requests. The credentials still authorize them.

(No Nessie/Quarkus CORS config is needed: all catalog traffic is same-origin through the nessie viz proxy.)

2. Credential vending (terraform — nessie_iam.tf + nessie_ecs.tf)​

Per-OEM vending roles in nessie_iam.tf; client-IAM env vars in nessie_ecs.tf. Enabled with Nessie 0.108.2 (fix for projectnessie/nessie#12703).

3. nessie viz CLI subcommand (implemented — the ai_nessie package: cli.py + server.py)​

Local server, no hosting, no new Auth0 configuration:

  • Binds 127.0.0.1:19121, serves the SPA HTML (bundled as package data at ai_nessie/viz/index.html). The duckdb-wasm engine itself loads from the jsDelivr CDN — the one external dependency. Bundling the multi-MB wasm into the repo isn't worth it for an internal dev tool (and the large-file pre-commit hook would reject it).
  • GET /token via the existing DeviceCodeAuthManager (cache read + silent refresh; runs the device flow on first use if no cache exists).
  • GET /aws-credentials from the ambient boto3 chain.
  • Catalog proxy (/nessie-proxy/<branch>, /nessie-api/) with the branch rewrite above; injects the server-side bearer token and forwards the X-Iceberg-Access-Delegation header on every upstream call.
  • Opens the browser tab on start.
  • UX: uv run --directory packages/ai_nessie nessie viz → tab opens, already authenticated.

The command reads its Nessie config from the environment (NESSIE_URI, NESSIE_AUTH0_DOMAIN, NESSIE_DEVICE_CLIENT_ID, NESSIE_AUTH0_API_AUDIENCE). The repo loads these via direnv (the gitignored root .envrc), so once that's set up nessie viz picks them up from the shell with no extra flag.

The code itself does not auto-load a .env file. If you keep the Nessie config in a standalone .env instead of .envrc, pass it through uv — with an absolute path, since --directory changes the working directory and a relative --env-file would resolve against packages/ai_nessie rather than the repo root:

# from the repo root
uv run --env-file "$PWD/.env" --directory packages/ai_nessie nessie viz

A relative path fails with error: No environment file found at: .env. This is also the form to use from a git worktree, which has no .envrc of its own.

4. Visualizer SPA​

  • duckdb-wasm latest dist-tag (1.33.1-dev*; no stable 1.33.0 exists) + a minimal UI: warehouse (OEM) dropdown, branch dropdown, table tree, SQL editor, result grid.
  • Table tree and branch list from Nessie's REST API via the proxy — not DuckDB catalog listing (~26 s for 185 namespaces, and SHOW ALL TABLES returned nothing).
  • On load: fetch /token, LOAD httpfs, create Iceberg secret, ATTACH with ACCESS_DELEGATION_MODE 'vended_credentials'; re-fetch + re-attach on 401 or branch/warehouse switch. S3 creds vended per-table by Nessie.
  • All query compute and data stay in the browser; tokens held in memory only.
  • Result grid and table tree render via textContent / DOM APIs, never innerHTML: cell values and catalog names ultimately originate from third-party OEM data and must not be parseable as markup (an XSS payload in a row could otherwise read /token). The cell-expand modal is the one exception — it builds highlighted JSON as a string, so syntaxHighlight HTML-escapes its input first (JSON.stringify does not escape <).

Paging the result grid​

The grid draws one page of MAX_PAGE_ROWS (1000) rows at a time, with prev/next and a jump-to-page input that appear only when there is more than one page.

  • Every row stays reachable — paging slices the retained Arrow table, so there is no truncation and no re-query when you move between pages.
  • Only one page is ever in the DOM. A 2 M-row result at 29 columns would otherwise mean ~58 M cell nodes; the page cap is what keeps the tab alive.
  • Out-of-range or non-numeric page jumps clamp to a valid page rather than blanking the grid.

Exporting results​

Export CSV / Export JSON buttons next to Run, enabled once a query succeeds.

  • Exports the full result set, not just the page on screen. The Arrow table already holds every row, so this costs nothing extra.
  • Pure client-side serialization from that retained Arrow table — no re-query and no server round trip. That is deliberate: re-running the SQL through COPY (…) TO would double the S3 scan and could return different rows than the grid shows (a CURRENT_DATE rollover, an unordered scan, a branch commit arriving in between).
  • Files are written as nessie-<branch>-<YYYYMMDD-HHMMSS>.<ext>, branch captured at run time.
  • CSV is RFC4180. NULL is a bare empty field and an empty string is "", matching DuckDB's own COPY output. JSON is an array with one object per line — loadable in one json.load() and still greppable line-by-line.
  • List(Struct(...)) columns serialize as real nested JSON (Arrow's Vector.toJSON() / StructRow.toJSON()). In CSV that JSON document is quoted into a single field. Int64 values beyond 2^53 become JSON strings rather than silently rounding.
  • Known limitation: timestamps nested inside a struct stay raw epoch integers — the temporal formatter is keyed off top-level schema.fields only. The grid has always rendered them the same way.
  • Above 500k rows the SPA asks for confirmation first. Serialization is chunked and yields to the event loop, but the honest ceiling is ~2 GB of text, where Blob construction throws and the SPA reports export failed: …. Past that, use Trino/Athena.
  • Parquet is deliberately not offered: it only earns its keep at sizes where the WASM path runs out of memory anyway.

5. Hosted variant (ECS sidecar)​

Hosted at https://nessie.app.autointel.ai/viz/ via a sidecar container in the Nessie ECS task. Path-based ALB routing (/viz* → sidecar:19121). Auth0 PKCE replaces the device-flow token injection; S3 credentials are vended by Nessie via RBAC-scoped credential vending. See hosted deploy.

Verification​

  1. Phase 1: browser page loads a table read on main and on a branch without --disable-web-security (proves bucket CORS + proxy).
  2. Export: run a query with LIMIT 1500, confirm the grid shows page 1 of 2 and the downloaded file has 1500 rows — this is what proves export reads the Arrow table and not the rendered page. Open the CSV in a spreadsheet (the real RFC4180 test) and round-trip the JSON through json.load().
  3. Paging: on that same result, step to page 2 and confirm it holds the remaining 500 rows, then type an out-of-range page number and confirm it clamps instead of blanking.

Open questions​

  • Result-set size guardrails in the SPA (DuckDB-WASM memory is browser-bounded; the 23.3 M-row count took ~57 s cold over residential broadband). Export has a 500k-row confirmation prompt; query execution itself is still unguarded.
  • Local development mode for the repo's SQLite Iceberg catalog + local warehouse, so temp/iceberg tables can be browsed without first exporting temp/iceberg_views.duckdb for a separate DuckDB client.
  • Upstream duckdb-iceberg issue for the ignored uri config override / prefix double-append: filed as duckdb-iceberg#1145 — once fixed, the proxy can be simplified away.