Skip to main content

Serverless Workers

Temporal invokes a Lambda when a task arrives on a queue nothing is polling. The worker starts, polls, runs what it picks up, and drains before the invocation deadline — so a queue costs nothing while idle and needs no reconciler to scale it.

Every queue is served this way. One function per queue, and a rollback is pointing the deployment at the build id it had before.

What runs​

python -m ai_ordering.lambda_worker is not the entrypoint — Lambda talks to the container over its Runtime API, so Dockerfile.lambda runs awslambdaric against ai_ordering.lambda_worker.lambda_handler.

One function serves one task queue, so there is a function per queue. The SDK builds a single Worker from a single task_queue, and the invocation carries no queue identity — nothing tells a function which queue woke it. TEMPORAL_TASK_QUEUE decides, before it starts.

Every function runs the same image; only that variable differs. Workflows and activities come from QUEUE_REGISTRATIONS, so queue isolation holds here as it does anywhere. Naming a queue with no registration fails the invocation rather than polling a queue nobody is watching.

Each function gets its own Worker Deployment (ai-ordering-<name>), because Temporal makes one version Current per deployment and these release independently.

Versioning​

run_worker always enables Worker Deployment Versioning, and a versioned worker will not register a workflow with no versioning behavior. The workflows declare none, so the Lambda supplies one for all of them: pinned.

A run therefore finishes on the build it started on, and new runs pick up whatever version is current. Auto-upgrade would move an in-flight run to a new build at its next task, and these runs hold selection state and place real orders — a build that scores differently, or whose workflow code has moved, would change what a half-finished run does.

The cost is that a pinned run keeps its build alive: draining a version means waiting for its runs to finish rather than cutting over.

Nothing worker-local survives​

Each invocation creates and destroys its Worker, and consecutive tasks in one run arrive in different processes. An activity cannot hold state in worker memory for a later activity to read.

Nothing does this any more. Selection pipelines, OEM payloads, and the frames pull_data hands construct_targets all cross through external storage, so any invocation can serve any activity of a run.

A warm container keeps imported modules, so a module-level client does survive — onto a loop that does not. Each invocation runs under its own asyncio.run, and an AsyncMongoClient or an aioboto3 client cannot leave the loop that built it. Reusing one raises Cannot use AsyncMongoClient in different event loop on the next invocation. Every event-loop-bound resource is therefore built and closed inside configure, around its yield — the pattern the Temporal SDK prescribes for exactly this.

lambda_worker_queues maps a short name to a queue, one entry per function. The name suffixes both the function and its deployment, so changing it replaces both.

The run-kind queues (ordering-live, ordering-simulated) are the whole map.

Secrets​

Terraform passes each secret as a reference and the function resolves <NAME> through ai_core.secrets at cold start. An already-set variable wins, so a local run supplies values directly. Putting secret values in the function's variables would store them in its configuration as plaintext.

The function drops AWS_REGION and the other credential variables from the shared environment: the runtime sets them itself and CreateFunction rejects a function that declares them. The SDKs read the runtime's values.

GCP credentials​

BigQuery is reached by workload identity federation, so no service-account key exists. The trust path:

  1. GCP_CREDENTIAL_CONFIG carries an external_account config — the pool provider that trusts ai-ordering-lambda-worker-role, and the service account ppm-ordering-worker@ai-app-386318 that role may impersonate. It holds no secret, which is why it is passed as a plain environment variable.
  2. write_gcp_credential_config writes it to /tmp at cold start and sets GOOGLE_APPLICATION_CREDENTIALS. Application Default Credentials resolution then finds it, so no BigQuery caller takes a credentials argument.
  3. google-auth signs a GetCallerIdentity request with the function's AWS credentials, GCP's STS verifies the caller is the trusted account and role, and the resulting token is exchanged for an impersonated access token.

Step 3 depends on the Lambda runtime placing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_REGION in the environment. That is the one thing here that does not carry to other compute. google-auth's default AWS credential source reads those variables, and otherwise falls back to the EC2 metadata server — which an ECS task does not have, its credentials coming from the container credentials endpoint instead. Running this worker on ECS means supplying a custom AwsSecurityCredentialsSupplier, which is a constructor argument and cannot be expressed in the config file at all.

Terraform owns both ends: the GCP iam solution defines the pool, provider, service account and its BigQuery grants, and the ai-ordering solution renders the config from its ai_ordering_workload_identity_audience and ordering_service_account_email outputs. Widening which projects the worker can read is a grant on that service account, not a change here.

Known gaps​

Terraform Deploy and CI are separate workflows with no ordering between them, so Terraform can create the function pointing at an image CI has not pushed yet, and on a first deploy CI pushes to an ECR repository Terraform has not created yet.

Deploying​

Terraform creates the function, its role, and the role Temporal assumes to invoke it. It does not bind a queue to the function — that is a Temporal-side step.

Terraform owns the function's structure — role, VPC, memory, timeout, environment, the ECR repository, the invoke role. CI owns its code. Neither waits on the other.

  1. CI builds and pushes the image. The service image job also builds Dockerfile.lambda into the ai-ordering-lambda repository — the same PEX as the ECS image behind a different entrypoint. It is tagged with the commit SHA. On a pull request that is the PR head, not github.sha, which on a pull request is a synthetic merge commit belonging to no branch. That repository is immutable, so re-running a workflow on a commit already pushed skips the push rather than failing. To build one by hand:

    docker build . -f services/ai_ordering/Dockerfile.lambda --platform linux/amd64 -t <repo>:<sha>

  2. The same job publishes the version and registers it. It reads /ai-ordering/serverless-worker from SSM, runs update-function-code --publish, and registers the version that call returns:

    aws lambda update-function-code --function-name ai-ordering-worker-<name> \
      --image-uri <repo>:<sha> --publish --query Version --output text

    Once per function, from the workers list in the parameter. Both halves belong to the job that pushed the image: Lambda cannot run a tag ECR does not hold, and Temporal cannot invoke a version that was never published. Publishing an unchanged image returns the version already carrying it, so a re-run finds that version registered and registers nothing.

    A merge to main deploys; so does a PR carrying the branch-deploy label, which repoints the same functions, because there is one account.

  3. Terraform applies only when the infrastructure changes. image_uri is in ignore_changes, so an apply never rolls the code and never needs an image that a given commit may not have built. Creating a function does need some image, so Terraform reads the repository's most recent one — CI replaces it on the next deploy.

    Check the five Temporal invoker account ids in lambda_worker.tf against Temporal's own trust policy before the first apply — they are transcribed from that page, and a wrong digit either fails to assume or trusts an account that is not Temporal's.

The SSM handoff​

/ai-ordering/serverless-worker is a SecureString holding a workers list — function name, deployment name and queue for each — plus the invoke role ARN, its external id, and how to reach Temporal. Terraform writes it; CI reads it. The deploy therefore never runs Terraform, and the only ordering left between them is that Terraform has run at least once.

The external id is why the parameter is encrypted. A role ARN is not a secret, so without it another Temporal customer who learned this one could name it in their own deployment version and have Temporal assume it for them — the confused deputy problem. AWS's guidance is that the third party mints the value. Temporal lets you choose one, which puts uniqueness on us, hence 32 random characters. Nothing regenerates it — a new one would break every invocation until the deployment version was re-registered.

The build id is the Lambda version​

Temporal binds a build id to a qualified ARN — …:function:ai-ordering-worker-<name>:7 — not to the function itself. So the build id is that version number, read at startup from AWS_LAMBDA_FUNCTION_VERSION, which the runtime supplies and nothing can contradict. There is no build id in the function's environment to drift out of step with the code, and no value the deploy has to decide before it knows what it published.

An unqualified ARN follows $LATEST, which moves with every code update, so deployment_version() refuses it rather than pinning runs to something that changes.

Tracing a build id back to a commit goes through the image: the published version's ImageUri carries the SHA the code was built from.

aws lambda get-function --function-name ai-ordering-worker-<name>:7 \
  --query Configuration.ImageUri

Rollback is a re-registration, not a rebuild: point the deployment back at an earlier build id, whose Lambda version still exists.

Creating the Worker Deployment​

A Worker Deployment comes into existence when a worker polls under its name. A serverless version never polls, so nothing creates it and create-version fails with no Worker Deployment found. The deploy creates it when absent:

temporal worker deployment create --name ai-ordering-<name>

Cutting over on every deploy is safe because the workflows are pinned: set-current-version routes new runs only, and anything in flight finishes on the version it started on.

The version budget is namespace-wide​

Temporal caps the registered Versions that hold a compute config, counted across the whole namespace rather than per Deployment. Only a Version registered with create-version --aws-lambda-function-arn holds one. A worker that registers by polling does not, so the ECS workers are outside this budget. The cap on ppm-prod.obsii is 150, which VERSION_BUDGET in the deploy step mirrors — Temporal owns the number, and raising it is a support request. docs.temporal.io/cloud/limits describes a per-Deployment limit of 100 instead; the enforced limit is the namespace-wide one.

So the budget is split: each deployment is held to VERSION_BUDGET / <deployments> - 1 Versions, the spare slot being the registration it is about to receive, and the oldest drained or inactive Versions are deleted to get there. Pruning is a pass of its own that finishes before any deployment registers — a deployment still above its share would otherwise hold the namespace at the cap for the ones registering ahead of it. Current, Ramping and draining Versions are never deleted, so a deployment whose Versions are mostly those cannot be brought under its share.

Running it locally​

Deploying is not how you test a change to the worker. The handler ignores its event and reads only a deadline off its context, so an invocation needs nothing from AWS:

services/ai_ordering/scripts/local_temporal.sh
uv run --directory services/ai_ordering python scripts/local_lambda_worker.py \
  --task-queue ordering-simulated

That runs the real composition — same configure, same converter, same registrations — against the dev server, until Ctrl-C.

Bound the invocation to reproduce Lambda's actual pattern:

... --invocation-seconds 60 --invocations 3

Each invocation starts a worker, polls, drains and exits, and the next one starts from nothing. That is what exposes state a worker held in memory for a later invocation to read. Below about 15s there is no working time left after the shutdown buffer, and the SDK says so.

The script sets AWS_LAMBDA_FUNCTION_VERSION to --build-id (default local), since locally there is no runtime to supply it. It then waits for the worker to poll and makes that version current — a Worker Deployment and its versions come into existence when a worker polls under them, and a versioned worker is sent nothing until its version is the queue's current one.

What local cannot tell you, and therefore what a deploy is still for:

  • whether Temporal can assume the invoke role and reach the function
  • whether the function reaches Nessie and Mongo from the private subnets
  • whether an activity fits in 900s and 8 GB against a real scope
  • whether the image is packaged for linux/amd64

Limits worth knowing​

Invocation ceiling900s, Lambda's maximum — and therefore the ceiling on one activity
Memory8192 MB, well above Temporal's 256 MB baseline: a live selection pipeline is resident, and Lambda scales CPU with memory
NetworkingThe function joins the worker's private subnets and security group, because Nessie and Mongo are VPC-internal

An activity that cannot finish in 900s cannot run here at all. construct_targets is the one to watch — it is blocked on the registry above regardless, but the time limit is the constraint that outlives that fix.

The worker also stops polling a drain window before that ceiling, so what an activity can actually have is less than 900s:

Invocation ceiling900s — lambda_worker_timeout_seconds, mirrored as WORKER_INVOCATION_CEILING
Drain, plus the hooks after it185s — GRACEFUL_SHUTDOWN + SHUTDOWN_HOOK_ALLOWANCE, time the worker spends not polling
Left for one activity~715s

An activity timeout above what that leaves bounds nothing, because the invocation ends first. PIPELINE_ACTIVITY_TIMEOUT and DATA_ACTIVITY_TIMEOUT are 10 minutes for that reason, and PIPELINE_HEARTBEAT_TIMEOUT is 5 so a stall is detected from the heartbeat rather than from waiting out the whole window. test_lambda_worker holds them to that arithmetic.

The drain does not cover a whole selection activity — nothing sized for a national scope finishes in three minutes. What it covers is an activity that has finished computing and is storing its result: heartbeating stops when the activity function returns, while the SDK is still writing. The server measures that silence against heartbeat_timeout. Storing the largest payload a run moves has taken up to a minute.

Payload storage needs s3:ListBucket​

The storage driver HEADs a content-addressed key before writing it, so a store is idempotent. S3 answers a HEAD of an absent key with 404 only when the caller holds s3:ListBucket. Without it the answer is 403, which is not a not-found code, so the first store of any payload fails the activity. The function's role carries the grant on the bucket. 403 stays out of the driver's not-found set deliberately — reading it as "not stored yet" would make a misconfigured role look like a cache miss.

A worker killed with its invocation — the 900s ceiling, or an out-of-memory kill — reports nothing to Temporal, so the run detects it only from a timeout. Every long activity therefore carries a heartbeat_timeout, and every activity that has one heartbeats across its whole body (activities/_heartbeat.py), not only inside its blocking sections. Without that the failure costs the full start_to_close_timeout: for a ten-minute data activity, ten minutes of silence before the retry.

Troubleshooting​

Nothing runs and the queue shows no pollers: the deployment version has to be current (set-current-version). Without that step Temporal never invokes the function.

AccessDenied assuming the invoke role: the external id passed to create-version must match the one in /ai-ordering/serverless-worker.

create-version returns code 13 INTERNAL: something went wrong, please retry with a support id: the namespace is at its Version cap. The error names nothing about Versions, and the count in the step's own log line is for one deployment, so it can read as low while the namespace is full. temporal worker deployment list plus a describe per deployment gives the real total.

The function starts and exits immediately: check the log group for a RuntimeError from configure — an unset TEMPORAL_TASK_QUEUE, or a queue with no registration, both fail there by design.