Deploying auto_swe
workflows/auto_swe runs in AWS as the Lambda functions below. The worker has its own
image, and the webhook functions share one:
auto-swe-workeris the session worker, a Temporal serverless worker on theauto-swequeue. It also runs the Dagster triage workflows (The triage workflow). Temporal Cloud invokes it when a task arrives, as it doesai_ordering's workers (Serverless Workers).auto-swe-github-webhookis the GitHub App's webhook endpoint, a function URL with no AWS authentication. It runsauto_swe.adapters.github.webhook_lambda.handler. The handler checks the delivery's signature and reduces the delivery to a work item. It then submits the work item to the subject's session, if an enabled agent type would run on it. The handler answers 200 to a delivery it drops. One session per subject states which deliveries it drops. The function runs at most 10 invocations at once (github_webhook_reserved_concurrency), so requests to the public URL cannot use the concurrency the account's other functions share.auto-swe-linear-webhookis the Linear write app's webhook endpoint, a function URL with no AWS authentication. It runsauto_swe.adapters.linear.webhook_lambda.handler, which checks the delivery's signature and timestamp and submits its work items in the same way. It answers 200 to every delivery whose signature matches, because Linear retries a delivery answered with any other status. The implementation agent lists the deliveries it reduces. It runs at most 5 invocations at once (linear_webhook_reserved_concurrency).
GitHub waits ten seconds for a response and does not redeliver a delivery that timed out. A cold start of the webhook function must therefore finish within ten seconds. Two settings keep it within that limit:
- The webhook's image holds only what the handler imports.
Dockerfile.webhookbuilds it from thewebhook_pextarget inworkflows/auto_swe/src/BUILD. - The function has 2048 MB of memory. Lambda allocates CPU in proportion to memory.
With the worker's image, the webhook function's initialization took more than ten seconds. With its own image, it took 6.8 s at 512 MB and 1.9 s at 2048 MB.
Linear waits five seconds for a response. One minute after a delivery times out,
Linear retries it. The Linear webhook function has the same image and memory as the
GitHub webhook function. A retry has the delivery's Linear-Delivery id, so a session
ignores the retry of a delivery it received.
No function is connected to a VPC, because every service the functions call is on the public internet.
What deploys what
| Piece | Owner |
|---|---|
Functions, roles, ECR repository, SSM handoff, the Secret expiry table, the auto-swe-agent-readonly role for the AWS MCP Server, the auto-swe-code-factory AWS Budget | Terraform, solutions/auto-swe |
Secret shells under auto-swe/ | Terraform, solutions/auto-swe-secrets; values filled by hand |
| The two images, each worker version, its Temporal registration | CI, deploy_workflow_lambda in ci.yml, on main |
| Search attributes on the namespace | By hand, once per namespace |
CI builds both images on every pull request that changes the application. It tags the
worker's image with the commit SHA and the webhook's image with <commit SHA>-functions.
It pushes and deploys them only from main. The deploy runs scripts/deploy_serverless_worker.sh,
the same script as ai_ordering's Lambda deploy, with the parameter and the Version
budget named in workflows/auto_swe/build.yaml. The script:
- applies the current invoke role ARN and external id to every version of the worker that Temporal does not report as drained;
- publishes a version of the worker;
- registers the version with Temporal;
- makes the version current;
- updates the webhook functions to the webhook image, if steps 2 to 4 succeeded;
- sends the
check_versionsignal, named bycheck_version_signalinbuild.yaml, to every running workflow on the worker's task queue.
Versions
Sessions and triage workflows use the PINNED versioning behavior. A run finishes on the
version it started on, so a change to a workflow's control flow needs no
workflow.patched.
A run moves to a new version when it continues as new. At each workflow task, Temporal
tells a pinned run whether its deployment's target version differs from the version the run
is pinned to. When it differs, the run continues as new before it handles another work
item. _continue_as_new in agent_runs/workflow.py passes
initial_versioning_behavior=AUTO_UPGRADE, so the next run's first workflow task goes to
the target version. The next run is pinned to that version.
Temporal starts a workflow task only when an event arrives for the run, such as a signal.
An idle session would therefore stay on its version for up to its idle_timeout, 14 days
by default. The check_version signal gives every running workflow a workflow task right
after the deploy. A run with a step run in progress continues as new once that work
item is handled. A run that continues as new while it waits for a work item passes on
when the wait began, in RunState.idle_since, so the next run still ends idle_timeout
after the last work item.
Temporal reports a version as drained once no running workflow is pinned to it. The deploy
deletes only versions that Temporal reports as drained or inactive, so a version with
pinned runs keeps counting against version_budget.
Some payloads are written by one version and read by another:
- A signal from a newer webhook reaches a run pinned to an older version.
WorkItem,RepoRefandFeedbackignore unknown fields, so the older version handles the work item without a field it does not have. The webhook must not set a field whose absence would change how a session handles the item until every running workflow is on a version that has the field. A value that an older version'sLiteraldoes not list, such as a newFeedback.kind, fails validation there, and the older run drops the signal. - The input a run passes to continue-as-new,
SessionInputorTriageInput, is written by the older version and read by the newer one. A field the newer version adds needs a default. Both workflows fail on aValidationError, so an input that the newer version cannot read ends the workflow.
Activities run on the version of the run that scheduled them.
Upgrade on continue-as-new is in Public Preview, and its SDK API is experimental. It needs Temporal server 1.30.2 or later.
Changing the invoke role's external id
Temporal never invokes a version that holds an external id the invoke role's trust policy
no longer accepts.
Changing the external id
describes how Temporal uses the external id. Sessions and triage workflows pinned to such
a version wait with a workflow task scheduled, and their histories record no failure. The
check_version signal does not move them, because a run handles the signal only in a
workflow task.
Step 1 of each deploy updates every version that Temporal does not report as drained. When the step fails to update a version, it adds a warning annotation to the GitHub Actions run, and the deploy continues. Changing the external id also describes the drained versions that the update leaves out.
A Terraform apply does not start a deploy, so after an apply that replaces the external id, update the versions from a local checkout. A deploy that started before the apply registers its new version with the old external id, so rotate the external id while no deploy of auto_swe runs. To rotate the external id:
-
In a local checkout of
mainthat matches the commit Terraform last applied, removeprevent_destroyfromrandom_password.temporal_invoke_external_idinsolutions/auto-swe. -
Initialize the solution as in The first deploy, then run
terraform apply -replace=random_password.temporal_invoke_external_idwith the same-var-file. Before you confirm, check that the plan replaces therandom_passwordand changes only the invoke role's trust policy and the SSM parameter. The apply writes the new id to the role's trust policy and to the SSM parameter. -
Right after the apply, update every version that is not drained:
PARAMETER=/auto-swe/serverless-worker \
scripts/deploy_serverless_worker.sh --resync-invoke-roleChanging the external id lists the credentials and tools the command needs. If the command exits with status 1, run it again. For a version it still cannot update, run the
update-version-compute-configcommand from that section with the parameter/auto-swe/serverless-worker, the deploymentauto-sweand the functionauto-swe-worker. If a deploy of auto_swe ran during the apply, run the command again after that deploy finishes. -
Restore
prevent_destroyin the checkout.
Bedrock
The worker's role is allowed to call bedrock:InvokeModel on the bedrock: models
priced in workflows/auto_swe/src/auto_swe/harness/model_prices.json. For a model ID
with an inference profile prefix, such as us., the grant also covers the model's
foundation model in every Region, which Bedrock checks for a call through the profile.
solutions/auto-swe reads the file, and the Terraform workflow runs when the file
changes on main, so merging a new bedrock: price grants the model. See Agents for
how an agent names a model.
Before an agent uses a model through Bedrock, invoke the model once from an
administrator's role in the account. For a model that Bedrock sells through AWS
Marketplace, such as an Anthropic model, the first invocation subscribes the account to
it. That invocation needs aws-marketplace:Subscribe, which the worker's role does not
have. In an account that has never used an Anthropic model, an administrator first
submits Anthropic's use-case form in the Bedrock console.
Payloads and the codec server
auto_swe runs in its own Temporal namespace, auto-swe.obsii. Its functions write
offloaded payloads to ai-ordering-payloads-prod, the bucket that ai_ordering's codec
server reads. The ai-ordering solution lists auto-swe.obsii in
codec_server_extra_namespaces, so the codec server serves it as well as ppm-prod.obsii.
In Temporal Cloud, the codec server settings of auto-swe.obsii set the same endpoint
as those of ppm-prod.obsii. auto_swe's payloads expire under that bucket's 45-day
lifecycle rule. Each application's workers can read the other application's payloads.
Secrets
Fill each auto-swe/ secret in Secrets Manager after the auto-swe-secrets solution
is applied. The shells have their own solution, apart from the shared secrets
solution, so they move with auto_swe if it leaves this repository. A function resolves its secrets at cold start, and fails to start if one is
empty.
| Secret | Read by |
|---|---|
auto-swe/temporal/api-key | every function, and the CI deploy, which registers worker versions with it |
auto-swe/github/webhook-secret | GitHub webhook |
auto-swe/linear/webhook-secret | Linear webhook. It holds the signing secret of the Linear write app's webhook |
auto-swe/github/app-id, app-private-key, app-installation-id | worker |
auto-swe/daytona/api-key | worker. The key needs write:sandboxes, delete:sandboxes and manage:secrets |
auto-swe/anthropic/api-key | worker |
auto-swe/openai/api-key | worker |
auto-swe/langsmith/api-key | worker |
auto-swe/otlp/headers | worker. It holds the request headers of the worker's OTLP span export, as key=value,key=value. For LangSmith, the value is x-api-key=<LangSmith API key>,Langsmith-Project=auto-swe. See Exporting spans |
auto-swe/temporal/api-key holds a Temporal Cloud API key with access to the
auto-swe.obsii namespace.
The worker resolves the secrets above at cold start. The credential providers read the secrets below only when they mint a Daytona Secret, so an empty one fails only the agent types that declare its credential, except where a row says otherwise:
| Secret | Credential |
|---|---|
auto-swe/linear/read-oauth-client, auto-swe/linear/write-oauth-client | linear: read, linear: write. Each holds {"client_id": …, "client_secret": …} of its own Linear OAuth app with client-credentials tokens turned on. The worker mints the read app's tokens with scope read and the write app's with read,write,app:mentionable |
auto-swe/linear/write-token | None. The worker writes the write app's token here as {"value": …, "expires_at": …}, for its own Linear writes, and every worker reads it. Leave it empty |
auto-swe/dagster/read-token, auto-swe/dagster/write-token | dagster: read, dagster: write. Each holds a Dagster+ API token. The read token's user has the Viewer role, and the write token's user has the Launcher role. The triage context and the triage agent's Dagster+ tools read Dagster+ with the read token |
dagster/nessie/m2m-reader-token | nessie: read. The nessie-reader-token-refresher Lambda in the dagster-agent solution writes it; the secrets solution has its shell |
The worker reads auto-swe/typesafe/api-key, a TypeSafe API key, for each of its
TypeSafe requests; Tokens and cost lists
them. While it is empty, the pull-request agent's classify step fails, and every
mention goes to the review-or-answer step; see
What a mention asks for. The triage
context then leaves a failure's cause unclassified.
The triage agent files its issues as the Linear write app, through the shared token in
auto-swe/linear/write-token.
The worker records each credential's expiry in the DynamoDB table
auto-swe-credential-expiry, which solutions/auto-swe creates. The auto-swe job in
terraform.yml runs after secrets as well as auto-swe-secrets, because the
solution looks up the Nessie reader token's shell, which secrets creates.
The first deploy
Terraform creates the worker function from an image that already exists, and the ECR repository is in the same solution. The first apply therefore runs in two parts.
Before step 1, create the auto-swe.obsii namespace in Temporal Cloud with API-key
authentication. In its codec server settings, set the endpoint that ppm-prod.obsii
uses and turn on Pass the user access token.
-
Fill the
auto-swe/secrets. -
Create the repository alone:
cd deployments/aws/terraform/solutions/auto-swe
terraform init -backend-config="../../environments/us-east-1/auto-swe/prod/backend.tfvars"
terraform apply -target=aws_ecr_repository.lambda \
-var-file="../../environments/us-east-1/auto-swe/prod/environment.tfvars" -
Re-run the
deploy_workflow_lambdajob onmain. It pushes the image and then fails at the deploy step, since the SSM parameter does not exist yet. -
Run the full apply, or re-run the Terraform workflow.
-
Re-run
deploy_workflow_lambda. It registers the first version with Temporal. -
Run
tcld login, then register the search attributes:tcld namespace search-attributes add --namespace auto-swe.obsii \
--sa "AutoSweSubject=Keyword" --sa "AutoSweWorkItemType=Keyword" \
--sa "AutoSweAgentTypes=KeywordList" --sa "AutoSweHarness=Keyword" \
--sa "AutoSweOutcome=Keyword"The list is copied from
scripts/register_search_attributes.sh, whose list a test keeps equal toauto_swe.agent_runs.search_attributes. A workflow that upserts an unregistered attribute fails its workflow task. -
Set the GitHub App's webhook URL to the
webhook_urloutput, with the webhook secret from step 1. -
Grant the App these repository permissions:
Permission Access Contents, Metadata Read-only Pull requests Read and write Issues Read and write Checks, Actions Read-only An agent's
githubcredential can hold only permissions the App has. Thegithub_post_comment,github_post_reviewandgithub_reply_in_threadtools need Pull requests write, and so does the 👀 reaction on a comment in the pull request's conversation. Reading the linked issues and receiving the issue comment event need Issues read. Thepull-requestagent'sgithubcredential has read access only. GitHub lets an App subscribe to an event only when the App has the permission the event requires, so the workflow run event needs Actions. An agent that pushes needs Contents write. The ruleset onmain, described in Branch protection, blocks its pushes tomain. -
Subscribe the App to the events the adapter reduces to work items:
- pull request
- pull request review comment
- issue comment
- check suite
- workflow run
-
Open a test pull request and comment
@ai-auto-swe review. The App's recent deliveries show a 202 from the webhook. The comment gets 👀. The Cloud UI shows the pull request's session, with its payloads decoded. Thepull-requestagent posts a review.
The Linear webhook
The implementation agent runs on the deliveries of the
Linear write app's own webhook. A workspace admin sets the webhook up once, after the
auto-swe solution is applied:
- In the write app's settings in Linear, turn on webhooks, set the URL to the
linear_webhook_urloutput, and subscribe to the Issues and Agent session events categories. As soon as Linear agent session events are on, people in the workspace can mention the app, and they see its Linear agent sessions. - Put the webhook's signing secret in
auto-swe/linear/webhook-secret. - Mention the app in a comment on a test issue. Its webhook shows a 200, and the Temporal Cloud UI shows the issue's session.
The worker mints the write app's tokens with the app:mentionable scope, which lets
people mention the app. Minting a token with a different scope revokes the app's other
tokens. A worker whose token is refused mints a new one.