Skip to main content

Serverless Workers

Temporal invokes a Lambda when a task arrives on a queue nothing is polling. The worker starts, polls, runs what it picks up, and drains before the invocation deadline — so a queue costs nothing while idle and needs no reconciler to scale it.

Every queue is served this way. One function per queue, and a rollback is pointing the deployment at the build id it had before.

What runs​

python -m ai_ordering.lambda_worker is not the entrypoint — Lambda talks to the container over its Runtime API, so Dockerfile.lambda runs awslambdaric against ai_ordering.lambda_worker.lambda_handler.

One function serves one task queue, so there is a function per queue. The SDK builds a single Worker from a single task_queue, and the invocation carries no queue identity — nothing tells a function which queue woke it. TEMPORAL_TASK_QUEUE decides, before it starts.

Every function runs the same image; only that variable differs. Workflows and activities come from ORDERING_REGISTRATIONS, so queue isolation holds here as it does anywhere. Naming a queue with no registration fails the invocation rather than polling a queue nobody is watching.

The cold-start bootstrap — secrets, GCP credentials, data converter, deployment version, the drain budget — is ai_temporal.lambda_worker.LambdaWorkerHarness, shared with every other Temporal application in the repo. ai_ordering.lambda_worker supplies the parts only this service has: its registrations, its payload converter, and a Mongo lifecycle entered and exited inside the invocation that owns the event loop.

Each function gets its own Worker Deployment (ai-ordering-<name>), because Temporal makes one version Current per deployment and these release independently.

Versioning​

run_worker always enables Worker Deployment Versioning, and a versioned worker will not register a workflow with no versioning behavior. The workflows declare none, so the Lambda supplies one for all of them: pinned.

A run therefore finishes on the build it started on, and new runs pick up whatever version is current. Auto-upgrade would move an in-flight run to a new build at its next task, and these runs hold selection state and place real orders — a build that scores differently, or whose workflow code has moved, would change what a half-finished run does.

Because a run on the Lambda workers never leaves its build, a change to the workflow's sequence of commands, such as which activities it schedules, needs no workflow.patched for runs on those workers.

The long-running worker, python -m ai_ordering.worker, is not versioned. A run in flight on it replays on whatever code the worker restarts with. If that code schedules different commands, the replay fails with a nondeterminism error.

The cost is that a pinned run keeps its build alive: draining a version means waiting for its runs to finish rather than cutting over.

Nothing worker-local survives​

Each invocation creates and destroys its Worker, and consecutive tasks in one run arrive in different processes. An activity cannot hold state in worker memory for a later activity to read.

Selection pipelines, OEM payloads, and the frames pull_data passes to construct_targets are stored in external storage between activities. Any invocation can therefore serve any activity in a run.

A warm container keeps imported modules, so a module-level client does survive — onto a loop that does not. Each invocation runs under its own asyncio.run, and an AsyncMongoClient or an aioboto3 client cannot leave the loop that built it. Reusing one raises Cannot use AsyncMongoClient in different event loop on the next invocation. Every event-loop-bound resource is therefore built and closed inside configure, around its yield — the pattern the Temporal SDK prescribes for exactly this.

A cached workflow outlives its worker unless the worker evicts it. At the deadline, run_worker cancels Worker.run, and the SDK shuts the worker down without evicting the workflows it caches. A workflow that was waiting on an activity keeps its coroutines suspended. Python's garbage collector later closes those coroutines on the thread where the collection runs. In a warm container, that thread can be running another workflow's task. A finally block in the closed coroutine then issues commands, and Temporal records them in that other workflow's history. That workflow's own code did not produce those commands, so every later replay of it fails with a nondeterminism error.

LambdaWorkerHarness therefore adds ai_temporal.eviction.EvictCachedWorkflows to every worker. Once the worker stops, the plugin activates each workflow still cached with a RemoveFromCache job, the same job the SDK processes when it evicts a workflow from a full cache. The workflow's coroutines unwind inside the workflow itself, and the SDK drops every command they issue. A workflow whose activation is deadlocked still holds its thread, so the plugin leaves it cached.

The SDK has no public eviction call, so evict_cached_workflows reads the private Worker._workflow_worker attribute and its _running_workflows dictionary. packages/ai_temporal/tests/test_eviction.py fails when a Temporal release renames either attribute.

lambda_worker_queues maps a short name to a queue, one entry per function. The name suffixes both the function and its deployment, so changing it replaces both.

The run-kind queues (ordering-live, ordering-simulated) are the whole map.

Secrets​

Terraform passes each secret as a reference and the function resolves <NAME> through ai_core.secrets at cold start. An already-set variable wins, so a local run supplies values directly. Putting secret values in the function's variables would store them in its configuration as plaintext.

The function drops AWS_REGION and the other credential variables from the shared environment: the runtime sets them itself and CreateFunction rejects a function that declares them. The SDKs read the runtime's values.

GCP credentials​

BigQuery is reached by workload identity federation, so no service-account key exists. The trust path:

  1. GCP_CREDENTIAL_CONFIG carries an external_account config — the pool provider that trusts ai-ordering-lambda-worker-role, and the service account ppm-ordering-worker@ai-app-386318 that role may impersonate. It holds no secret, which is why it is passed as a plain environment variable.
  2. write_gcp_credential_config writes it to /tmp at cold start and sets GOOGLE_APPLICATION_CREDENTIALS. Application Default Credentials resolution then finds it, so no BigQuery caller takes a credentials argument.
  3. google-auth signs a GetCallerIdentity request with the function's AWS credentials, GCP's STS verifies the caller is the trusted account and role, and the resulting token is exchanged for an impersonated access token.

Step 3 depends on the Lambda runtime placing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_REGION in the environment. That is the one thing here that does not carry to other compute. google-auth's default AWS credential source reads those variables, and otherwise falls back to the EC2 metadata server — which an ECS task does not have, its credentials coming from the container credentials endpoint instead. Running this worker on ECS means supplying a custom AwsSecurityCredentialsSupplier, which is a constructor argument and cannot be expressed in the config file at all.

Terraform owns both ends: the GCP iam solution defines the pool, provider, service account and its BigQuery grants, and the ai-ordering solution renders the config from its ai_ordering_workload_identity_audience and ordering_service_account_email outputs. Widening which projects the worker can read is a grant on that service account, not a change here.

Known gaps​

Terraform Deploy and CI are separate workflows with no ordering between them, so Terraform can create the function pointing at an image CI has not pushed yet, and on a first deploy CI pushes to an ECR repository Terraform has not created yet.

Deploying​

Terraform creates the function, its role, and the role Temporal assumes to invoke it. It does not bind a queue to the function — that is a Temporal-side step.

Terraform owns the function's structure — role, VPC, memory, timeout, environment, the ECR repository, the invoke role. CI owns its code. Neither waits on the other.

  1. CI builds and pushes the image. The service image job also builds Dockerfile.lambda into the ai-ordering-lambda repository — the same PEX as the ECS image behind a different entrypoint. It is tagged with the commit SHA. On a pull request that is the PR head, not github.sha, which on a pull request is a synthetic merge commit belonging to no branch. That repository is immutable, so re-running a workflow on a commit already pushed skips the push rather than failing. To build one by hand:

    docker build . -f services/ai_ordering/Dockerfile.lambda --platform linux/amd64 -t <repo>:<sha>
  2. The same job publishes the version and registers it. It reads /ai-ordering/serverless-worker from SSM, runs update-function-code --publish, and registers the version that call returns:

    aws lambda update-function-code --function-name ai-ordering-worker-<name> \
    --image-uri <repo>:<sha> --publish --query Version --output text

    Once per function, from the workers list in the parameter. Both halves belong to the job that pushed the image: Lambda cannot run a tag ECR does not hold, and Temporal cannot invoke a version that was never published. Publishing an unchanged image returns the version already carrying it, so a re-run finds that version registered and registers nothing.

    A run on main deploys the service when Deploys from main selects it. A pull request builds the image but neither pushes nor deploys it. To run a branch on Lambda, repoint a function by hand.

  3. Terraform applies only when the infrastructure changes. image_uri is in ignore_changes, so an apply never rolls the code and never needs an image that a given commit may not have built. Creating a function does need some image, so Terraform reads the repository's most recent one — CI replaces it on the next deploy.

    Check the five Temporal invoker account ids in lambda_worker.tf against Temporal's own trust policy before the first apply — they are transcribed from that page, and a wrong digit either fails to assume or trusts an account that is not Temporal's.

The SSM handoff​

/ai-ordering/serverless-worker is a SecureString holding a workers list — function name, deployment name and queue for each — plus the invoke role ARN, its external id, and how to reach Temporal. Terraform writes it; CI reads it. The deploy therefore never runs Terraform, and the only ordering left between them is that Terraform has run at least once.

The external id is why the parameter is encrypted. A role ARN is not a secret, so without it another Temporal customer who learned this one could name it in their own deployment version and have Temporal assume it for them — the confused deputy problem. AWS's guidance is that the third party mints the value. Temporal lets you choose one, which puts uniqueness on us, hence 32 random characters. Nothing regenerates it. The random_password resource has no keepers, and its prevent_destroy setting makes Terraform reject any plan that would replace it. Changing the external id describes what a new one requires.

Changing the external id​

Temporal stores the invoke role ARN and the external id with each Version, in the Version's compute config. It assumes the role with that pair each time it invokes the Version. Once the role's trust policy has a new external id, Temporal cannot assume the role for a Version that holds the old one, so it never invokes that Version. A run pinned to the Version waits with a workflow task scheduled, and its history records no failure. Making a later Version current does not move a pinned run.

Before it registers the new Version, scripts/deploy_serverless_worker.sh runs temporal worker deployment update-version-compute-config on every Version of each worker's deployment that Temporal does not report as drained. The call sets the role ARN and the external id from the SSM parameter. It sets the function ARN to the ARN that Lambda returns for the function version whose number is the Version's build id. If the function has no such version, the script does not update the Version and treats the update as failed. describe-version shows neither the role ARN nor the external id, so the script cannot tell which Versions hold the old pair, and it updates all of them. When an update fails, the script writes a GitHub Actions warning annotation that names the deployment and the build id, and continues. A failed update does not fail the step.

scripts/deploy_serverless_worker.sh --resync-invoke-role runs only these updates and exits. It needs only the PARAMETER environment variable. It exits with status 1 if it did not update every Version that is not drained. It runs with any AWS credentials that can:

  • read the SSM parameter;
  • read the Temporal API key secret that the parameter names;
  • call lambda:GetFunctionConfiguration on the worker functions.

It also needs the AWS CLI, jq and a Temporal CLI of version 1.10.0 or later on PATH.

The script does not update a drained Version, so a drained Version keeps the pair it holds. A rollback makes a drained Version current again, so update the Version with the following command before the rollback:

CONFIG=$(aws ssm get-parameter --name /ai-ordering/serverless-worker \
--with-decryption --query Parameter.Value --output text)
TEMPORAL_API_KEY=$(aws secretsmanager get-secret-value \
--secret-id "$(jq -r .temporal_api_key_secret_arn <<<"$CONFIG")" \
--query SecretString --output text)
export TEMPORAL_API_KEY
temporal worker deployment update-version-compute-config \
--address "$(jq -r .temporal_address <<<"$CONFIG")" \
--namespace "$(jq -r .temporal_namespace <<<"$CONFIG")" --tls \
--deployment-name ai-ordering-<name> --build-id <build id> \
--aws-lambda-function-arn arn:aws:lambda:us-east-1:999655274916:function:ai-ordering-worker-<name>:<build id> \
--aws-lambda-assume-role-arn "$(jq -r .invoke_role_arn <<<"$CONFIG")" \
--aws-lambda-assume-role-external-id "$(jq -r .invoke_external_id <<<"$CONFIG")"

A Terraform apply does not start a deploy. Between the apply that replaces the external id and the update of a Version, Temporal does not invoke that Version, even when it is the current Version.

A deploy reads the external id from the SSM parameter when it starts. A deploy that started before the apply registers its new Version with the old external id, makes that Version current, and can apply the old external id to the other Versions again. Rotate the external id while no deploy of the application runs:

  1. In a local checkout of main that matches the commit Terraform last applied, remove prevent_destroy from random_password.temporal_invoke_external_id in solutions/ai-ordering.

  2. Initialize the solution as in deployments/aws/terraform/CLAUDE.md, then run terraform apply -replace=random_password.temporal_invoke_external_id with the solution's -var-file. Before you confirm, check that the plan replaces the random_password and changes only the invoke role's trust policy and the SSM parameter. The apply writes the new id to the role's trust policy and to the SSM parameter.

  3. Right after the apply, update every Version that is not drained:

    PARAMETER=/ai-ordering/serverless-worker \
    scripts/deploy_serverless_worker.sh --resync-invoke-role

    If it exits with status 1, run it again, or run the command above for each build id that its warnings name. If a deploy of the application ran during the apply, run the command again after that deploy finishes.

  4. Restore prevent_destroy in the checkout.

The build id is the Lambda version​

Temporal binds a build id to a qualified ARN — …:function:ai-ordering-worker-<name>:7 — not to the function itself. So the build id is that version number, read at startup from AWS_LAMBDA_FUNCTION_VERSION, which the runtime supplies and nothing can contradict. There is no build id in the function's environment to drift out of step with the code, and no value the deploy has to decide before it knows what it published.

An unqualified ARN follows $LATEST, which moves with every code update, so deployment_version() refuses it rather than pinning runs to something that changes.

Tracing a build id back to a commit goes through the image: the published version's ImageUri carries the SHA the code was built from.

aws lambda get-function --function-name ai-ordering-worker-<name>:7 \
--query Configuration.ImageUri

Rollback is a re-registration, not a rebuild: point the deployment back at an earlier build id, whose Lambda version still exists.

Creating the Worker Deployment​

A Worker Deployment comes into existence when a worker polls under its name. A serverless version never polls, so nothing creates it and create-version fails with no Worker Deployment found. The deploy creates it when absent:

temporal worker deployment create --name ai-ordering-<name>

Cutting over on every deploy is safe because the workflows are pinned: set-current-version routes new runs only, and anything in flight finishes on the version it started on.

Creating the Mongo indexes​

A worker invocation creates no Mongo indexes. The ordering deploy step in .github/workflows/ci.yml sets SETUP_EVENT to the value of ai_ordering.lambda_worker.CREATE_INDEXES_EVENT. scripts/deploy_serverless_worker.sh invokes the first worker version it publishes with that event, before it registers any version with Temporal. lambda_handler then runs mongo_client.init_indexes() instead of starting a Temporal worker. The function already has the Mongo secret and connects to Mongo from the private subnets, so the deploy needs no Mongo access of its own.

The indexes therefore exist before any run reaches the new image. When the invocation fails, the script does not register that worker's version. It invokes the next worker's version with the event instead. The script registers a version only after one of these invocations succeeds. When every invocation fails, every worker stays on its previous version and the step fails. For each failed invocation, the step prints only the error type; the function's CloudWatch log group has the traceback. init_indexes() is idempotent, so re-running the deploy repeats it harmlessly.

The script invokes the qualified worker ARN, so CI's AWS identity needs lambda:InvokeFunction on it. The CI user's policy, which is not in this repository, has to grant it.

Worker invocations do not create indexes because init_indexes() calls create_index once per index. A batch that starts dozens of runs at once causes dozens of cold starts, and each one would repeat those calls. Mongo already refuses TLS handshakes under the load of those cold starts.

To create them by hand:

aws lambda invoke --function-name ai-ordering-worker-<name>:<version> \
--cli-binary-format raw-in-base64-out \
--payload '{"ai_ordering": "create_mongo_indexes"}' /dev/stdout

The version budget is namespace-wide​

Temporal caps the registered Versions that hold a compute config, counted across the whole namespace rather than per Deployment. Only a Version registered with create-version --aws-lambda-function-arn holds one. A worker that registers by polling does not, so the ECS workers are outside this budget. The cap on ppm-prod.obsii is 150. Temporal sets the number, and raising it takes a support request. scripts/deploy_serverless_worker.sh deploys every application's serverless workers, and each application has its own VERSION_BUDGET. Ordering's budget is 105. A workflow application's budget is the version_budget in its build.yaml. The budgets of all applications that deploy to the same namespace must together stay under that namespace's cap. auto_swe deploys to a namespace of its own (Deploying auto_swe). docs.temporal.io/cloud/limits describes a per-Deployment limit of 100 instead; the enforced limit is the namespace-wide one.

So the budget is split: each deployment is held to VERSION_BUDGET / <deployments> - 1 Versions, the spare slot being the registration it is about to receive, and the oldest drained or inactive Versions are deleted to get there. Pruning is a pass of its own that finishes before any deployment registers — a deployment still above its share would otherwise hold the namespace at the cap for the ones registering ahead of it. Current, Ramping and draining Versions are never deleted, so a deployment whose Versions are mostly those cannot be brought under its share.

An inactive Version is one that was registered but has never been Current or Ramping. The Temporal CLI's describe -o json output has no Version status. The script therefore counts a Version as inactive when its drainageStatus is unspecified and the output's routingConfig names it neither as the current Version nor as the ramping Version.

Running it locally​

Deploying is not how you test a change to the worker. The handler ignores every event except CREATE_INDEXES_EVENT. It reads only a deadline off its context, so an invocation needs nothing from AWS:

services/ai_ordering/scripts/local_temporal.sh
uv run --directory services/ai_ordering python scripts/local_lambda_worker.py \
--task-queue ordering-simulated

That runs the real composition — same configure, same converter, same registrations — against the dev server, until Ctrl-C.

Bound the invocation to reproduce Lambda's actual pattern:

... --invocation-seconds 60 --invocations 3

Each invocation starts a worker, polls, drains and exits, and the next one starts from nothing. That is what exposes state a worker held in memory for a later invocation to read. Below about 15s there is no working time left after the shutdown buffer, and the SDK says so.

The script sets AWS_LAMBDA_FUNCTION_VERSION to --build-id (default local), since locally there is no runtime to supply it. It then waits for the worker to poll and makes that version current — a Worker Deployment and its versions come into existence when a worker polls under them, and a versioned worker is sent nothing until its version is the queue's current one.

What local cannot tell you, and therefore what a deploy is still for:

  • whether Temporal can assume the invoke role and reach the function
  • whether the function reaches Nessie and Mongo from the private subnets
  • whether an activity fits in 900s and 8 GB against a real scope
  • whether the image is packaged for linux/amd64

Limits worth knowing​

Invocation ceiling900s, Lambda's maximum — and therefore the ceiling on one activity
Memory8192 MB, well above Temporal's 256 MB baseline: a live selection pipeline is resident, and Lambda scales CPU with memory
NetworkingThe function joins the worker's private subnets and security group, because Nessie and Mongo are VPC-internal

An activity that cannot finish in 900s cannot run here at all. construct_targets is the one to watch — it is blocked on the registry above regardless, but the time limit is the constraint that outlives that fix.

The worker also stops polling a drain window before that ceiling, so what an activity can actually have is less than 900s:

Invocation ceiling900s — lambda_worker_timeout_seconds, mirrored as WORKER_INVOCATION_CEILING
Drain, plus the hooks after it185s — GRACEFUL_SHUTDOWN + SHUTDOWN_HOOK_ALLOWANCE, time the worker spends not polling
Left for one activity~715s

An activity timeout above what that leaves bounds nothing, because the invocation ends first. PIPELINE_ACTIVITY_TIMEOUT and DATA_ACTIVITY_TIMEOUT are 10 minutes for that reason, and PIPELINE_HEARTBEAT_TIMEOUT is 5 so a stall is detected from the heartbeat rather than from waiting out the whole window. test_lambda_worker holds them to that arithmetic.

The drain does not cover a whole selection activity — nothing sized for a national scope finishes in three minutes. What it covers is an activity that has finished computing and is storing its result: heartbeating stops when the activity function returns, while the SDK is still writing. The server measures that silence against heartbeat_timeout. Storing the largest payload a run moves has taken up to a minute.

Payload storage needs s3:ListBucket​

The storage driver HEADs a content-addressed key before writing it, so a store is idempotent. S3 answers a HEAD of an absent key with 404 only when the caller holds s3:ListBucket. Without it the answer is 403, which is not a not-found code, so the first store of any payload fails the activity. The function's role carries the grant on the bucket. 403 stays out of the driver's not-found set deliberately — reading it as "not stored yet" would make a misconfigured role look like a cache miss.

A worker killed with its invocation — the 900s ceiling, or an out-of-memory kill — reports nothing to Temporal, so the run detects it only from a timeout. Every long activity therefore carries a heartbeat_timeout, and every activity that has one heartbeats across its whole body (activities/_heartbeat.py), not only inside its blocking sections. Without that the failure costs the full start_to_close_timeout: for a ten-minute data activity, ten minutes of silence before the retry.

Troubleshooting​

Nothing runs and the queue shows no pollers: the deployment version has to be current (set-current-version). Without that step Temporal never invokes the function.

AccessDenied assuming the invoke role: the external id a Version holds must match the one in /ai-ordering/serverless-worker.

Runs pinned to one Version keep a workflow task scheduled for hours, and their histories record no failure: the Version may hold an external id that the invoke role no longer accepts. See Changing the external id.

create-version returns code 13 INTERNAL: something went wrong, please retry with a support id: the namespace is at its Version cap. The error names nothing about Versions, and the count in the step's own log line is for one deployment, so it can read as low while the namespace is full. temporal worker deployment list plus a describe per deployment gives the real total.

The function starts and exits immediately: check the log group for a RuntimeError from configure — an unset TEMPORAL_TASK_QUEUE, or a queue with no registration, both fail there by design.