Serverless Workers
Temporal invokes a Lambda when a task arrives on a queue nothing is polling. The worker starts, polls, runs what it picks up, and drains before the invocation deadline — so a queue costs nothing while idle and needs no reconciler to scale it.
Every queue is served this way. One function per queue, and a rollback is pointing the deployment at the build id it had before.
What runs
python -m ai_ordering.lambda_worker is not the entrypoint — Lambda talks to the container
over its Runtime API, so Dockerfile.lambda runs awslambdaric against
ai_ordering.lambda_worker.lambda_handler.
One function serves one task queue, so there is a function per queue. The SDK builds a
single Worker from a single task_queue, and the invocation carries no queue identity —
nothing tells a function which queue woke it. TEMPORAL_TASK_QUEUE decides, before it
starts.
Every function runs the same image; only that variable differs. Workflows and activities
come from ORDERING_REGISTRATIONS, so queue isolation holds here as it does anywhere.
Naming a queue with no registration fails the invocation rather than polling a queue nobody
is watching.
The cold-start bootstrap — secrets, GCP credentials, data converter, deployment version,
the drain budget — is ai_temporal.lambda_worker.LambdaWorkerHarness, shared with every
other Temporal application in the repo. ai_ordering.lambda_worker supplies the parts only
this service has: its registrations, its payload converter, and a Mongo lifecycle entered
and exited inside the invocation that owns the event loop.
Each function gets its own Worker Deployment (ai-ordering-<name>), because Temporal makes
one version Current per deployment and these release independently.
Versioning
run_worker always enables Worker Deployment Versioning, and a versioned worker will not
register a workflow with no versioning behavior. The workflows declare none, so the Lambda
supplies one for all of them: pinned.
A run therefore finishes on the build it started on, and new runs pick up whatever version is current. Auto-upgrade would move an in-flight run to a new build at its next task, and these runs hold selection state and place real orders — a build that scores differently, or whose workflow code has moved, would change what a half-finished run does.
Because a run on the Lambda workers never leaves its build, a change to the workflow's
sequence of commands, such as which activities it schedules, needs no workflow.patched for
runs on those workers.
The long-running worker, python -m ai_ordering.worker, is not versioned. A run in flight
on it replays on whatever code the worker restarts with. If that code schedules different
commands, the replay fails with a nondeterminism error.
The cost is that a pinned run keeps its build alive: draining a version means waiting for its runs to finish rather than cutting over.
Nothing worker-local survives
Each invocation creates and destroys its Worker, and consecutive tasks in one run arrive in different processes. An activity cannot hold state in worker memory for a later activity to read.
Selection pipelines, OEM payloads, and the frames pull_data passes to
construct_targets are stored in external storage between activities. Any invocation can
therefore serve any activity in a run.
A warm container keeps imported modules, so a module-level client does survive — onto a
loop that does not. Each invocation runs under its own asyncio.run, and an
AsyncMongoClient or an aioboto3 client cannot leave the loop that built it. Reusing one
raises Cannot use AsyncMongoClient in different event loop on the next invocation. Every
event-loop-bound resource is therefore built and closed inside configure, around its
yield — the pattern the Temporal SDK prescribes for exactly this.
A cached workflow outlives its worker unless the worker evicts it. At the deadline,
run_worker cancels Worker.run, and the SDK shuts the worker down without evicting the
workflows it caches. A workflow that was waiting on an activity keeps its coroutines
suspended. Python's garbage collector later closes those coroutines on the thread where the
collection runs. In a warm container, that thread can be running another workflow's task.
A finally block in the closed coroutine then issues commands, and Temporal records them in
that other workflow's history. That workflow's own code did not produce those commands, so
every later replay of it fails with a nondeterminism error.
LambdaWorkerHarness therefore adds ai_temporal.eviction.EvictCachedWorkflows to every
worker. Once the worker stops, the plugin activates each workflow still cached with a
RemoveFromCache job, the same job the SDK processes when it evicts a workflow from a full
cache. The workflow's coroutines unwind inside the workflow itself, and the SDK drops every
command they issue. A workflow whose activation is deadlocked still holds its thread, so the
plugin leaves it cached.
The SDK has no public eviction call, so evict_cached_workflows reads the private
Worker._workflow_worker attribute and its _running_workflows dictionary.
packages/ai_temporal/tests/test_eviction.py fails when a Temporal release renames either
attribute.
lambda_worker_queues maps a short name to a queue, one entry per function. The name
suffixes both the function and its deployment, so changing it replaces both.
The run-kind queues (ordering-live, ordering-simulated) are the whole map.
Secrets
Terraform passes each secret as a reference and the function resolves <NAME> through
ai_core.secrets at cold start. An already-set variable wins, so a local run supplies
values directly. Putting secret values in the function's variables would store them in
its configuration as plaintext.
The function drops AWS_REGION and the other credential variables from the shared
environment: the runtime sets them itself and CreateFunction rejects a function that
declares them. The SDKs read the runtime's values.
GCP credentials
BigQuery is reached by workload identity federation, so no service-account key exists. The trust path:
GCP_CREDENTIAL_CONFIGcarries anexternal_accountconfig — the pool provider that trustsai-ordering-lambda-worker-role, and the service accountppm-ordering-worker@ai-app-386318that role may impersonate. It holds no secret, which is why it is passed as a plain environment variable.write_gcp_credential_configwrites it to/tmpat cold start and setsGOOGLE_APPLICATION_CREDENTIALS. Application Default Credentials resolution then finds it, so no BigQuery caller takes a credentials argument.- google-auth signs a
GetCallerIdentityrequest with the function's AWS credentials, GCP's STS verifies the caller is the trusted account and role, and the resulting token is exchanged for an impersonated access token.
Step 3 depends on the Lambda runtime placing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
AWS_SESSION_TOKEN and AWS_REGION in the environment. That is the one thing here
that does not carry to other compute. google-auth's default AWS credential source reads
those variables, and otherwise falls back to the EC2 metadata server — which an ECS task
does not have, its credentials coming from the container credentials endpoint instead.
Running this worker on ECS means supplying a custom
AwsSecurityCredentialsSupplier, which is a constructor argument and cannot be expressed
in the config file at all.
Terraform owns both ends: the GCP iam solution defines the pool, provider, service
account and its BigQuery grants, and the ai-ordering solution renders the config from
its ai_ordering_workload_identity_audience and ordering_service_account_email outputs.
Widening which projects the worker can read is a grant on that service account, not a
change here.
Known gaps
Terraform Deploy and CI are separate workflows with no ordering between them, so Terraform can create the function pointing at an image CI has not pushed yet, and on a first deploy CI pushes to an ECR repository Terraform has not created yet.
Deploying
Terraform creates the function, its role, and the role Temporal assumes to invoke it. It does not bind a queue to the function — that is a Temporal-side step.
Terraform owns the function's structure — role, VPC, memory, timeout, environment, the ECR repository, the invoke role. CI owns its code. Neither waits on the other.
-
CI builds and pushes the image. The service image job also builds
Dockerfile.lambdainto theai-ordering-lambdarepository — the same PEX as the ECS image behind a different entrypoint. It is tagged with the commit SHA. On a pull request that is the PR head, notgithub.sha, which on a pull request is a synthetic merge commit belonging to no branch. That repository is immutable, so re-running a workflow on a commit already pushed skips the push rather than failing. To build one by hand:docker build . -f services/ai_ordering/Dockerfile.lambda --platform linux/amd64 -t <repo>:<sha> -
The same job publishes the version and registers it. It reads
/ai-ordering/serverless-workerfrom SSM, runsupdate-function-code --publish, and registers the version that call returns:aws lambda update-function-code --function-name ai-ordering-worker-<name> \
--image-uri <repo>:<sha> --publish --query Version --output textOnce per function, from the
workerslist in the parameter. Both halves belong to the job that pushed the image: Lambda cannot run a tag ECR does not hold, and Temporal cannot invoke a version that was never published. Publishing an unchanged image returns the version already carrying it, so a re-run finds that version registered and registers nothing.A run on main deploys the service when Deploys from main selects it. A pull request builds the image but neither pushes nor deploys it. To run a branch on Lambda, repoint a function by hand.
-
Terraform applies only when the infrastructure changes.
image_uriis inignore_changes, so an apply never rolls the code and never needs an image that a given commit may not have built. Creating a function does need some image, so Terraform reads the repository's most recent one — CI replaces it on the next deploy.Check the five Temporal invoker account ids in
lambda_worker.tfagainst Temporal's own trust policy before the first apply — they are transcribed from that page, and a wrong digit either fails to assume or trusts an account that is not Temporal's.
The SSM handoff
/ai-ordering/serverless-worker is a SecureString holding a workers list — function
name, deployment name and queue for each — plus the invoke role ARN, its external id, and
how to reach Temporal.
Terraform writes it; CI reads it. The deploy therefore never runs Terraform, and the only
ordering left between them is that Terraform has run at least once.
The external id is why the parameter is encrypted. A role ARN is not a secret, so without
it another Temporal customer who learned this one could name it in their own deployment
version and have Temporal assume it for them — the
confused deputy problem.
AWS's guidance is that the third party mints the value. Temporal lets you choose one, which
puts uniqueness on us, hence 32 random characters. Nothing regenerates it. The
random_password resource has no keepers, and its prevent_destroy setting makes
Terraform reject any plan that would replace it.
Changing the external id describes what a new one requires.
Changing the external id
Temporal stores the invoke role ARN and the external id with each Version, in the Version's compute config. It assumes the role with that pair each time it invokes the Version. Once the role's trust policy has a new external id, Temporal cannot assume the role for a Version that holds the old one, so it never invokes that Version. A run pinned to the Version waits with a workflow task scheduled, and its history records no failure. Making a later Version current does not move a pinned run.
Before it registers the new Version, scripts/deploy_serverless_worker.sh runs
temporal worker deployment update-version-compute-config on every Version of each
worker's deployment that Temporal does not report as drained. The call sets the role ARN
and the external id from the SSM parameter. It sets the function ARN to the ARN that
Lambda returns for the function version whose number is the Version's build id. If the
function has no such version, the script does not update the Version and treats the
update as failed. describe-version shows neither the role ARN nor the external id, so
the script cannot tell which Versions hold the old pair, and it updates all of them. When
an update fails, the script writes a GitHub Actions warning annotation that names the
deployment and the build id, and continues. A failed update does not fail the step.
scripts/deploy_serverless_worker.sh --resync-invoke-role runs only these updates and
exits. It needs only the PARAMETER environment variable. It exits with status 1 if it
did not update every Version that is not drained. It runs with any AWS credentials that
can:
- read the SSM parameter;
- read the Temporal API key secret that the parameter names;
- call
lambda:GetFunctionConfigurationon the worker functions.
It also needs the AWS CLI, jq and a Temporal CLI of version 1.10.0 or later on PATH.
The script does not update a drained Version, so a drained Version keeps the pair it holds. A rollback makes a drained Version current again, so update the Version with the following command before the rollback:
CONFIG=$(aws ssm get-parameter --name /ai-ordering/serverless-worker \
--with-decryption --query Parameter.Value --output text)
TEMPORAL_API_KEY=$(aws secretsmanager get-secret-value \
--secret-id "$(jq -r .temporal_api_key_secret_arn <<<"$CONFIG")" \
--query SecretString --output text)
export TEMPORAL_API_KEY
temporal worker deployment update-version-compute-config \
--address "$(jq -r .temporal_address <<<"$CONFIG")" \
--namespace "$(jq -r .temporal_namespace <<<"$CONFIG")" --tls \
--deployment-name ai-ordering-<name> --build-id <build id> \
--aws-lambda-function-arn arn:aws:lambda:us-east-1:999655274916:function:ai-ordering-worker-<name>:<build id> \
--aws-lambda-assume-role-arn "$(jq -r .invoke_role_arn <<<"$CONFIG")" \
--aws-lambda-assume-role-external-id "$(jq -r .invoke_external_id <<<"$CONFIG")"
A Terraform apply does not start a deploy. Between the apply that replaces the external id and the update of a Version, Temporal does not invoke that Version, even when it is the current Version.
A deploy reads the external id from the SSM parameter when it starts. A deploy that started before the apply registers its new Version with the old external id, makes that Version current, and can apply the old external id to the other Versions again. Rotate the external id while no deploy of the application runs:
-
In a local checkout of
mainthat matches the commit Terraform last applied, removeprevent_destroyfromrandom_password.temporal_invoke_external_idinsolutions/ai-ordering. -
Initialize the solution as in
deployments/aws/terraform/CLAUDE.md, then runterraform apply -replace=random_password.temporal_invoke_external_idwith the solution's-var-file. Before you confirm, check that the plan replaces therandom_passwordand changes only the invoke role's trust policy and the SSM parameter. The apply writes the new id to the role's trust policy and to the SSM parameter. -
Right after the apply, update every Version that is not drained:
PARAMETER=/ai-ordering/serverless-worker \
scripts/deploy_serverless_worker.sh --resync-invoke-roleIf it exits with status 1, run it again, or run the command above for each build id that its warnings name. If a deploy of the application ran during the apply, run the command again after that deploy finishes.
-
Restore
prevent_destroyin the checkout.
The build id is the Lambda version
Temporal binds a build id to a qualified ARN — …:function:ai-ordering-worker-<name>:7 — not
to the function itself. So the build id is that version number, read at startup from
AWS_LAMBDA_FUNCTION_VERSION, which the runtime supplies and nothing can contradict. There
is no build id in the function's environment to drift out of step with the code, and no
value the deploy has to decide before it knows what it published.
An unqualified ARN follows $LATEST, which moves with every code update, so
deployment_version() refuses it rather than pinning runs to something that changes.
Tracing a build id back to a commit goes through the image: the published version's
ImageUri carries the SHA the code was built from.
aws lambda get-function --function-name ai-ordering-worker-<name>:7 \
--query Configuration.ImageUri
Rollback is a re-registration, not a rebuild: point the deployment back at an earlier build id, whose Lambda version still exists.
Creating the Worker Deployment
A Worker Deployment comes into existence when a worker polls under its name. A serverless
version never polls, so nothing creates it and create-version fails with no Worker
Deployment found. The deploy creates it when absent:
temporal worker deployment create --name ai-ordering-<name>
Cutting over on every deploy is safe because the workflows are pinned: set-current-version
routes new runs only, and anything in flight finishes on the version it started on.
Creating the Mongo indexes
A worker invocation creates no Mongo indexes. The ordering deploy step in
.github/workflows/ci.yml sets SETUP_EVENT to the value of
ai_ordering.lambda_worker.CREATE_INDEXES_EVENT. scripts/deploy_serverless_worker.sh
invokes the first worker version it publishes with that event, before it registers any
version with Temporal. lambda_handler then runs mongo_client.init_indexes() instead of
starting a Temporal worker. The function already has the Mongo secret and connects to
Mongo from the private subnets, so the deploy needs no Mongo access of its own.
The indexes therefore exist before any run reaches the new image. When the invocation
fails, the script does not register that worker's version. It invokes the next worker's
version with the event instead. The script registers a version only after one of these
invocations succeeds. When every invocation fails, every worker stays on its previous
version and the step fails. For each failed invocation, the step prints only the error
type; the function's CloudWatch log group has the traceback. init_indexes() is
idempotent, so re-running the deploy repeats it harmlessly.
The script invokes the qualified worker ARN, so CI's AWS identity needs
lambda:InvokeFunction on it. The CI user's policy, which is not in this repository, has
to grant it.
Worker invocations do not create indexes because init_indexes() calls create_index once
per index. A batch that starts dozens of runs at once causes dozens of cold starts, and each
one would repeat those calls. Mongo already refuses TLS handshakes under the load of those
cold starts.
To create them by hand:
aws lambda invoke --function-name ai-ordering-worker-<name>:<version> \
--cli-binary-format raw-in-base64-out \
--payload '{"ai_ordering": "create_mongo_indexes"}' /dev/stdout
The version budget is namespace-wide
Temporal caps the registered Versions that hold a compute config, counted across the whole
namespace rather than per Deployment. Only a Version registered with create-version --aws-lambda-function-arn holds one. A worker that registers by polling does not, so the ECS
workers are outside this budget. The cap on ppm-prod.obsii is 150. Temporal sets the number,
and raising it takes a support request. scripts/deploy_serverless_worker.sh deploys every
application's serverless workers, and each application has its own VERSION_BUDGET.
Ordering's budget is 105. A workflow application's budget is the version_budget in its
build.yaml. The budgets of all applications that deploy to the same namespace must
together stay under that namespace's cap. auto_swe deploys to a namespace of its own
(Deploying auto_swe).
docs.temporal.io/cloud/limits describes a
per-Deployment limit of 100 instead; the enforced limit is the namespace-wide one.
So the budget is split: each deployment is held to VERSION_BUDGET / <deployments> - 1
Versions, the spare slot being the registration it is about to receive, and the oldest
drained or inactive Versions are deleted to get there. Pruning is a pass of its own that
finishes before any deployment registers — a deployment still above its share would
otherwise hold the namespace at the cap for the ones registering ahead of it. Current,
Ramping and draining Versions are never deleted, so a deployment whose Versions are mostly
those cannot be brought under its share.
An inactive Version is one that was registered but has never been Current or Ramping. The
Temporal CLI's describe -o json output has no Version status. The script therefore counts a
Version as inactive when its drainageStatus is unspecified and the output's
routingConfig names it neither as the current Version nor as the ramping Version.
Running it locally
Deploying is not how you test a change to the worker. The handler ignores every event
except CREATE_INDEXES_EVENT. It reads only a deadline off
its context, so an invocation needs nothing from AWS:
services/ai_ordering/scripts/local_temporal.sh
uv run --directory services/ai_ordering python scripts/local_lambda_worker.py \
--task-queue ordering-simulated
That runs the real composition — same configure, same converter, same registrations —
against the dev server, until Ctrl-C.
Bound the invocation to reproduce Lambda's actual pattern:
... --invocation-seconds 60 --invocations 3
Each invocation starts a worker, polls, drains and exits, and the next one starts from nothing. That is what exposes state a worker held in memory for a later invocation to read. Below about 15s there is no working time left after the shutdown buffer, and the SDK says so.
The script sets AWS_LAMBDA_FUNCTION_VERSION to --build-id (default local), since
locally there is no runtime to supply it. It then waits for the worker to poll and makes
that version current — a Worker Deployment and its versions come into existence when a
worker polls under them, and a versioned worker is sent nothing until its version is the
queue's current one.
What local cannot tell you, and therefore what a deploy is still for:
- whether Temporal can assume the invoke role and reach the function
- whether the function reaches Nessie and Mongo from the private subnets
- whether an activity fits in 900s and 8 GB against a real scope
- whether the image is packaged for
linux/amd64
Limits worth knowing
| Invocation ceiling | 900s, Lambda's maximum — and therefore the ceiling on one activity |
| Memory | 8192 MB, well above Temporal's 256 MB baseline: a live selection pipeline is resident, and Lambda scales CPU with memory |
| Networking | The function joins the worker's private subnets and security group, because Nessie and Mongo are VPC-internal |
An activity that cannot finish in 900s cannot run here at all. construct_targets is the
one to watch — it is blocked on the registry above regardless, but the time limit is the
constraint that outlives that fix.
The worker also stops polling a drain window before that ceiling, so what an activity can actually have is less than 900s:
| Invocation ceiling | 900s — lambda_worker_timeout_seconds, mirrored as WORKER_INVOCATION_CEILING |
| Drain, plus the hooks after it | 185s — GRACEFUL_SHUTDOWN + SHUTDOWN_HOOK_ALLOWANCE, time the worker spends not polling |
| Left for one activity | ~715s |
An activity timeout above what that leaves bounds nothing, because the invocation ends
first. PIPELINE_ACTIVITY_TIMEOUT and DATA_ACTIVITY_TIMEOUT are 10 minutes for that
reason, and PIPELINE_HEARTBEAT_TIMEOUT is 5 so a stall is detected from the heartbeat rather
than from waiting out the whole window. test_lambda_worker holds them to that arithmetic.
The drain does not cover a whole selection activity — nothing sized for a national scope
finishes in three minutes. What it covers is an activity that has finished computing and is
storing its result: heartbeating stops when the activity function returns, while the SDK is
still writing. The server measures that silence against heartbeat_timeout. Storing the
largest payload a run moves has taken up to a minute.
Payload storage needs s3:ListBucket
The storage driver HEADs a content-addressed key before writing it, so a store is
idempotent. S3 answers a HEAD of an absent key with 404 only when the caller holds
s3:ListBucket. Without it the answer is 403, which is not a not-found code, so the first
store of any payload fails the activity. The function's role carries the grant on the bucket.
403 stays out of the driver's not-found set deliberately — reading it as "not stored yet"
would make a misconfigured role look like a cache miss.
A worker killed with its invocation — the 900s ceiling, or an out-of-memory kill — reports
nothing to Temporal, so the run detects it only from a timeout. Every long activity
therefore carries a heartbeat_timeout, and every activity that has one heartbeats across
its whole body (activities/_heartbeat.py), not only inside its blocking sections. Without
that the failure costs the full start_to_close_timeout: for a ten-minute data activity, ten
minutes of silence before the retry.
Troubleshooting
Nothing runs and the queue shows no pollers: the deployment version has to be current
(set-current-version). Without that step Temporal never invokes the function.
AccessDenied assuming the invoke role: the external id a Version holds must match the one
in /ai-ordering/serverless-worker.
Runs pinned to one Version keep a workflow task scheduled for hours, and their histories record no failure: the Version may hold an external id that the invoke role no longer accepts. See Changing the external id.
create-version returns code 13 INTERNAL: something went wrong, please retry with a support
id: the namespace is at its Version cap. The error names nothing about Versions, and the count
in the step's own log line is for one deployment, so it can read as low while the namespace is
full. temporal worker deployment list plus a describe per deployment gives the real total.
The function starts and exits immediately: check the log group for a RuntimeError from
configure — an unset TEMPORAL_TASK_QUEUE, or a queue with no registration, both fail
there by design.