Codec Server

Ordering payloads are offloaded to external storage: the data converter writes the value to S3 and puts a small claim reference in Temporal event history (see Design, Run state handoff). That keeps history small, but it also means the Temporal UI has nothing to show. Every offloaded argument and result renders as:

{ "claimData": { "bucket": "ai-ordering-payloads-prod", "key": "…" } }

The UI cannot resolve that — only something holding this service's data converter and credentials for the bucket can. A codec server is the extension point Temporal provides for exactly this: the UI and CLI POST payloads to an HTTP endpoint you run, and it hands back payloads they can display.

ai_ordering.codec_server is that endpoint. It builds the same converter the worker uses (data_converter_from_env), so it reads whatever store the worker wrote to.

Quick start (local)

local_temporal.sh brings up the Temporal dev server and the codec server together, and points the UI at it. The codec server reads the same .env as the worker, so it picks up EXTERNAL_STORAGE_DIR or EXTERNAL_STORAGE_BUCKET with no extra configuration.

services/ai_ordering/scripts/local_temporal.sh

uv run --directory services/ai_ordering python -m ai_ordering.worker

The UI at http://localhost:8233 then resolves payloads with no further setup. --no-codec does not start the codec server. Run it by hand with:

uv run --directory services/ai_ordering python -m ai_ordering.codec_server

Override the port with CODEC_SERVER_PORT and TEMPORAL_CODEC_ENDPOINT if 8888 is taken.

Confirm it is up:

curl -s localhost:8888/health

{"status": "ok", "externalStorage": true} means it found a store. false means DISABLE_EXTERNAL_STORAGE is set and there is nothing to resolve.

Pointing Temporal Cloud at it

The Cloud UI calls the codec server from your browser, not from Temporal's servers. So the endpoint has to be reachable from wherever the operator is sitting — which is why the deployed one is internet-facing rather than in the VPC.

The deployed endpoint (set once, works for everyone)

Namespaces → your namespace → Edit → Codec Server:

Field	Value
Endpoint	https://codec.tmprl.api.autointel.ai
Pass the user access token	on
Include cross-origin credentials	off

Pass the user access token is required, not optional: the deployed server rejects every request without a valid Temporal token. Leave it off and the UI still shows claim references, with a 401 in the browser console.

Set at the Cluster level it applies to everyone viewing that namespace, with no per-person setup. Anyone can override it for themselves with Use my browser setting in the same dialog.

Other workflows in the namespace are unaffected: a payload that is not a claim reference is returned untouched, so anything that does not use external storage renders exactly as before.

A local server instead

Useful when you want payloads never to leave your machine, or when the deployed one is down:

EXTERNAL_STORAGE_BUCKET=ai-ordering-payloads-prod \
TEMPORAL_NAMESPACE=ppm-prod.obsii \
AWS_REGION=us-east-1 \
uv run --directory services/ai_ordering python -m ai_ordering.codec_server

You need credentials that can s3:GetObject on that bucket — see AI Agent AWS Access. Then set the endpoint to http://localhost:8888, using Use my browser setting so you do not point the whole namespace at a process on your laptop. A loopback server serves every caller, so the access token is optional there. Browsers treat http://localhost as a trustworthy origin, so an HTTPS page calling it is not blocked as mixed content.

https://cloud.temporal.io is in the default CORS allowlist. A self-hosted UI on another origin needs CODEC_SERVER_ALLOWED_ORIGINS.

Using it from the CLI

temporal workflow show --workflow-id <id> --codec-endpoint http://localhost:8888

Or set it once for an environment:

temporal env set --env prod --codec-endpoint http://localhost:8888

What you get back

Payload	Rendered as
Inline (under the offload threshold)	unchanged
Offloaded, under the inline limit	the real value
Offloaded, over the inline limit	a stub naming its size

Two things never come back verbatim.

Anything over the limit. /decode renders a whole event list at once, so it fetches only what is worth showing inline (1 MiB by default). A selection pipeline runs to hundreds of megabytes and no browser should be handed one. The stub tells you the size:

{
  "_codecServer": "not-fetched",
  "sizeBytes": 434110464,
  "limitBytes": 1048576,
  "detail": "… exceeds the /decode limit of 1048576 bytes …"
}

Clicking through to a single payload calls /download, which has its own, larger ceiling (64 MiB). Above that, read the object directly — aws s3 cp does it without holding it in the codec server's memory.

Pickled selection state. A SelectionPipeline is a live Python object graph, not a document (see storage/pickle_converter.py). There is nothing to render, so the stub names the encoding and size instead.

In practice a real pipeline trips the size guard first and you see not-fetched, never not-renderable. The size is checked on the reference, before the fetch, while the encoding is only known once the value is in hand. A reference carries the claim and a size and nothing else, so the server cannot tell a large document from selection state without retrieving it.

Raising the limit is therefore the wrong move for selection state — it pulls hundreds of megabytes into the codec server and then stubs anyway. scripts/show_history.py materializes payloads in-process and is what both stubs point at.

Configuration

Variable	Default	Meaning
CODEC_SERVER_HOST	127.0.0.1	Bind address. Loopback by default — anyone who can reach this endpoint can read decoded payloads.
CODEC_SERVER_PORT	8888
CODEC_SERVER_ALLOWED_ORIGINS	Temporal Cloud + localhost:8233	Comma-separated CORS allowlist.
CODEC_SERVER_INLINE_LIMIT_BYTES	1 MiB	Largest payload /decode will fetch.
CODEC_SERVER_MAX_DOWNLOAD_BYTES	64 MiB	Largest payload /download will fetch.
TEMPORAL_NAMESPACE	unset	When set, a request carrying a different X-Namespace is refused. A request with no such header is served — this catches a UI pointed at the wrong codec server, it is not an access control.
CODEC_SERVER_REQUIRE_AUTH	derived from the bind address	Whether a valid Temporal access token is demanded.
CODEC_SERVER_JWKS_URL	https://login.tmprl.cloud/.well-known/jwks.json	Key set tokens are verified against.
CODEC_SERVER_TOKEN_AUDIENCE	https://saas-api.tmprl.cloud	Required aud claim.

Plus the external-storage variables the worker uses — one of EXTERNAL_STORAGE_BUCKET, EXTERNAL_STORAGE_DIR, DISABLE_EXTERNAL_STORAGE.

EXTERNAL_STORAGE_DIR is a local directory, so a codec server configured that way can only resolve payloads written by a worker on the same machine.

Authentication

A codec server resolves offloaded payloads, so whatever can reach it can read a run's business data. What bounds that depends on where it is bound, and the default follows the binding. Loopback serves every caller — nothing but a process on that machine can reach it — while any other bind address demands a token.

With auth on, a request must carry Authorization: Bearer <token>. The token is the caller's own Temporal login JWT, forwarded by the UI when the namespace has Pass the user access token enabled. Without that setting the UI sends nothing and every request is refused. It is verified against Temporal's JWKS (signature, aud, expiry), which is what proves the caller signed in — this server holds no credential of its own and never sees a password.

CODEC_SERVER_REQUIRE_AUTH overrides the default either way. Turning it off on a public binding serves a run's payloads to anyone who finds the URL.

/health is always open, so a load balancer can poll it without a token.

A valid signature is not enough on its own. Temporal Cloud mints tokens for every customer against one shared audience (https://saas-api.tmprl.cloud), so signature validation alone would authorize any Temporal user anywhere.

The second check is the account. Temporal's token carries the account its holder signed in to, and CODEC_SERVER_TEMPORAL_ACCOUNT_ID is the one this server serves:

"https://saas-api.tmprl.cloud/context/accountID": "obsii"

A token from another Temporal account names a different id and is refused. The issuer is pinned alongside the audience. Nothing else in the token is checked: Temporal does not put a consistent codec marker on the tokens the UI forwards, so any login token for the account is accepted.

That authorizes at the account level: anyone in the account can read. Per-user granularity is not attempted, and the namespace check in _serves_namespace remains a misconfiguration guard rather than access control.

A good token from another account gets 403, not 401: another Temporal token would not help.

None of this applies on loopback, where no verifier is built at all.

Still not done: payloads are stored unencrypted, and this endpoint returns them in the clear to an authorized caller.

Endpoints

Method	Path
POST	/decode	Resolve claim references. ?preserveStorageRefs=true returns them untouched.
POST	/encode	Offload payloads over the threshold — the inverse, used when the UI sends input.
POST	/download	Resolve one reference the caller asked for explicitly.
GET	/health	Liveness, and whether a store is configured.

Request and response bodies are a temporalio.api.common.v1.Payloads message in proto JSON. See Temporal's codec server docs.

Deployment

The task definition is in deployments/aws/terraform/solutions/ai-ordering/codec_server.tf. It reuses the worker's image with a different command, so there is no second build and no second ECR repository — the codec server ships with whatever the worker is running.

It sits behind an internet-facing ALB at https://codec.tmprl.api.autointel.ai, because the Cloud UI calls a codec server from the operator's browser rather than from Temporal's servers — an in-VPC address would mean every operator running their own local instance. HTTPS is required: the Cloud UI is served over HTTPS, so a plain-HTTP endpoint is blocked as mixed content.

The ALB terminates TLS and forwards. It performs no authentication, because an ALB authenticate-oidc action drives a browser redirect flow that an XHR from the Cloud UI cannot follow. The codec server authenticates each request itself — see Authentication. The task sets CODEC_SERVER_REQUIRE_AUTH=true explicitly: the server would infer it from the bind address anyway, but a public endpoint's access control should be something the deployment states rather than something a host value implies. The task's security group accepts traffic from the ALB only, so nothing in the VPC can reach it directly and skip TLS.

The certificate ARN and hostname are codec_server_acm_certificate_arn and codec_server_domain_name in the environment's tfvars. The DNS record is created outside Terraform — terraform output codec_server_alb_dns_name gives the target.

The task role gets s3:GetObject and s3:PutObject on the payload bucket, the same grant the worker holds.

Troubleshooting

UI shows the claim reference and no error. The endpoint is not configured for that namespace. Cloud stores it per-namespace, the local UI takes it from --ui-codec-endpoint.

UI shows a CORS or network error. Check the server is running (/health) and that the UI's origin is in CODEC_SERVER_ALLOWED_ORIGINS. The browser console names the origin it sent.

403 with a namespace in the message. The codec server is bound to one namespace via TEMPORAL_NAMESPACE and the request asked for another — its payloads are in a different store. Start a second codec server on another port.

No external storage driver named 's3'. The payload was written by a worker using a different driver than this codec server has configured — usually a codec server on EXTERNAL_STORAGE_DIR pointed at history from a deployed run. Set EXTERNAL_STORAGE_BUCKET instead.

NoSuchKey / Access Denied from S3. Either the credentials cannot read the bucket, or the object aged out. Offloaded payloads expire after payload_expiration_days (45), which is deliberately longer than namespace retention, so an unreadable payload on a live run means credentials.