Skip to main content

Deploying auto_swe

workflows/auto_swe runs in AWS as the Lambda functions below. The worker has its own image, and the webhook functions share one:

  • auto-swe-worker is the session worker, a Temporal serverless worker on the auto-swe queue. It also runs the Dagster triage workflows (The triage workflow). Temporal Cloud invokes it when a task arrives, as it does ai_ordering's workers (Serverless Workers).
  • auto-swe-github-webhook is the GitHub App's webhook endpoint, a function URL with no AWS authentication. It runs auto_swe.adapters.github.webhook_lambda.handler. The handler checks the delivery's signature and reduces the delivery to a work item. It then submits the work item to the subject's session, if an enabled agent type would run on it. The handler answers 200 to a delivery it drops. One session per subject states which deliveries it drops. The function runs at most 10 invocations at once (github_webhook_reserved_concurrency), so requests to the public URL cannot use the concurrency the account's other functions share.
  • auto-swe-linear-webhook is the Linear write app's webhook endpoint, a function URL with no AWS authentication. It runs auto_swe.adapters.linear.webhook_lambda.handler, which checks the delivery's signature and timestamp and submits its work items in the same way. It answers 200 to every delivery whose signature matches, because Linear retries a delivery answered with any other status. The implementation agent lists the deliveries it reduces. It runs at most 5 invocations at once (linear_webhook_reserved_concurrency).

GitHub waits ten seconds for a response and does not redeliver a delivery that timed out. A cold start of the webhook function must therefore finish within ten seconds. Two settings keep it within that limit:

  • The webhook's image holds only what the handler imports. Dockerfile.webhook builds it from the webhook_pex target in workflows/auto_swe/src/BUILD.
  • The function has 2048 MB of memory. Lambda allocates CPU in proportion to memory.

With the worker's image, the webhook function's initialization took more than ten seconds. With its own image, it took 6.8 s at 512 MB and 1.9 s at 2048 MB.

Linear waits five seconds for a response. One minute after a delivery times out, Linear retries it. The Linear webhook function has the same image and memory as the GitHub webhook function. A retry has the delivery's Linear-Delivery id, so a session ignores the retry of a delivery it received.

No function is connected to a VPC, because every service the functions call is on the public internet.

What deploys what​

PieceOwner
Functions, roles, ECR repository, SSM handoff, the Secret expiry table, the auto-swe-agent-readonly role for the AWS MCP Server, the auto-swe-code-factory AWS BudgetTerraform, solutions/auto-swe
Secret shells under auto-swe/Terraform, solutions/auto-swe-secrets; values filled by hand
The two images, each worker version, its Temporal registrationCI, deploy_workflow_lambda in ci.yml, on main
Search attributes on the namespaceBy hand, once per namespace

CI builds both images on every pull request that changes the application. It tags the worker's image with the commit SHA and the webhook's image with <commit SHA>-functions. It pushes and deploys them only from main. The deploy runs scripts/deploy_serverless_worker.sh, the same script as ai_ordering's Lambda deploy, with the parameter and the Version budget named in workflows/auto_swe/build.yaml. The script:

  1. applies the current invoke role ARN and external id to every version of the worker that Temporal does not report as drained;
  2. publishes a version of the worker;
  3. registers the version with Temporal;
  4. makes the version current;
  5. updates the webhook functions to the webhook image, if steps 2 to 4 succeeded;
  6. sends the check_version signal, named by check_version_signal in build.yaml, to every running workflow on the worker's task queue.

Versions​

Sessions and triage workflows use the PINNED versioning behavior. A run finishes on the version it started on, so a change to a workflow's control flow needs no workflow.patched.

A run moves to a new version when it continues as new. At each workflow task, Temporal tells a pinned run whether its deployment's target version differs from the version the run is pinned to. When it differs, the run continues as new before it handles another work item. _continue_as_new in agent_runs/workflow.py passes initial_versioning_behavior=AUTO_UPGRADE, so the next run's first workflow task goes to the target version. The next run is pinned to that version.

Temporal starts a workflow task only when an event arrives for the run, such as a signal. An idle session would therefore stay on its version for up to its idle_timeout, 14 days by default. The check_version signal gives every running workflow a workflow task right after the deploy. A run with a step run in progress continues as new once that work item is handled. A run that continues as new while it waits for a work item passes on when the wait began, in RunState.idle_since, so the next run still ends idle_timeout after the last work item.

Temporal reports a version as drained once no running workflow is pinned to it. The deploy deletes only versions that Temporal reports as drained or inactive, so a version with pinned runs keeps counting against version_budget.

Some payloads are written by one version and read by another:

  • A signal from a newer webhook reaches a run pinned to an older version. WorkItem, RepoRef and Feedback ignore unknown fields, so the older version handles the work item without a field it does not have. The webhook must not set a field whose absence would change how a session handles the item until every running workflow is on a version that has the field. A value that an older version's Literal does not list, such as a new Feedback.kind, fails validation there, and the older run drops the signal.
  • The input a run passes to continue-as-new, SessionInput or TriageInput, is written by the older version and read by the newer one. A field the newer version adds needs a default. Both workflows fail on a ValidationError, so an input that the newer version cannot read ends the workflow.

Activities run on the version of the run that scheduled them.

Upgrade on continue-as-new is in Public Preview, and its SDK API is experimental. It needs Temporal server 1.30.2 or later.

Changing the invoke role's external id​

Temporal never invokes a version that holds an external id the invoke role's trust policy no longer accepts. Changing the external id describes how Temporal uses the external id. Sessions and triage workflows pinned to such a version wait with a workflow task scheduled, and their histories record no failure. The check_version signal does not move them, because a run handles the signal only in a workflow task.

Step 1 of each deploy updates every version that Temporal does not report as drained. When the step fails to update a version, it adds a warning annotation to the GitHub Actions run, and the deploy continues. Changing the external id also describes the drained versions that the update leaves out.

A Terraform apply does not start a deploy, so after an apply that replaces the external id, update the versions from a local checkout. A deploy that started before the apply registers its new version with the old external id, so rotate the external id while no deploy of auto_swe runs. To rotate the external id:

  1. In a local checkout of main that matches the commit Terraform last applied, remove prevent_destroy from random_password.temporal_invoke_external_id in solutions/auto-swe.

  2. Initialize the solution as in The first deploy, then run terraform apply -replace=random_password.temporal_invoke_external_id with the same -var-file. Before you confirm, check that the plan replaces the random_password and changes only the invoke role's trust policy and the SSM parameter. The apply writes the new id to the role's trust policy and to the SSM parameter.

  3. Right after the apply, update every version that is not drained:

    PARAMETER=/auto-swe/serverless-worker \
    scripts/deploy_serverless_worker.sh --resync-invoke-role

    Changing the external id lists the credentials and tools the command needs. If the command exits with status 1, run it again. For a version it still cannot update, run the update-version-compute-config command from that section with the parameter /auto-swe/serverless-worker, the deployment auto-swe and the function auto-swe-worker. If a deploy of auto_swe ran during the apply, run the command again after that deploy finishes.

  4. Restore prevent_destroy in the checkout.

Bedrock​

The worker's role is allowed to call bedrock:InvokeModel on the bedrock: models priced in workflows/auto_swe/src/auto_swe/harness/model_prices.json. For a model ID with an inference profile prefix, such as us., the grant also covers the model's foundation model in every Region, which Bedrock checks for a call through the profile. solutions/auto-swe reads the file, and the Terraform workflow runs when the file changes on main, so merging a new bedrock: price grants the model. See Agents for how an agent names a model.

Before an agent uses a model through Bedrock, invoke the model once from an administrator's role in the account. For a model that Bedrock sells through AWS Marketplace, such as an Anthropic model, the first invocation subscribes the account to it. That invocation needs aws-marketplace:Subscribe, which the worker's role does not have. In an account that has never used an Anthropic model, an administrator first submits Anthropic's use-case form in the Bedrock console.

Payloads and the codec server​

auto_swe runs in its own Temporal namespace, auto-swe.obsii. Its functions write offloaded payloads to ai-ordering-payloads-prod, the bucket that ai_ordering's codec server reads. The ai-ordering solution lists auto-swe.obsii in codec_server_extra_namespaces, so the codec server serves it as well as ppm-prod.obsii. In Temporal Cloud, the codec server settings of auto-swe.obsii set the same endpoint as those of ppm-prod.obsii. auto_swe's payloads expire under that bucket's 45-day lifecycle rule. Each application's workers can read the other application's payloads.

Secrets​

Fill each auto-swe/ secret in Secrets Manager after the auto-swe-secrets solution is applied. The shells have their own solution, apart from the shared secrets solution, so they move with auto_swe if it leaves this repository. A function resolves its secrets at cold start, and fails to start if one is empty.

SecretRead by
auto-swe/temporal/api-keyevery function, and the CI deploy, which registers worker versions with it
auto-swe/github/webhook-secretGitHub webhook
auto-swe/linear/webhook-secretLinear webhook. It holds the signing secret of the Linear write app's webhook
auto-swe/github/app-id, app-private-key, app-installation-idworker
auto-swe/daytona/api-keyworker. The key needs write:sandboxes, delete:sandboxes and manage:secrets
auto-swe/anthropic/api-keyworker
auto-swe/openai/api-keyworker
auto-swe/langsmith/api-keyworker
auto-swe/otlp/headersworker. It holds the request headers of the worker's OTLP span export, as key=value,key=value. For LangSmith, the value is x-api-key=<LangSmith API key>,Langsmith-Project=auto-swe. See Exporting spans

auto-swe/temporal/api-key holds a Temporal Cloud API key with access to the auto-swe.obsii namespace.

The worker resolves the secrets above at cold start. The credential providers read the secrets below only when they mint a Daytona Secret, so an empty one fails only the agent types that declare its credential, except where a row says otherwise:

SecretCredential
auto-swe/linear/read-oauth-client, auto-swe/linear/write-oauth-clientlinear: read, linear: write. Each holds {"client_id": …, "client_secret": …} of its own Linear OAuth app with client-credentials tokens turned on. The worker mints the read app's tokens with scope read and the write app's with read,write,app:mentionable
auto-swe/linear/write-tokenNone. The worker writes the write app's token here as {"value": …, "expires_at": …}, for its own Linear writes, and every worker reads it. Leave it empty
auto-swe/dagster/read-token, auto-swe/dagster/write-tokendagster: read, dagster: write. Each holds a Dagster+ API token. The read token's user has the Viewer role, and the write token's user has the Launcher role. The triage context and the triage agent's Dagster+ tools read Dagster+ with the read token
dagster/nessie/m2m-reader-tokennessie: read. The nessie-reader-token-refresher Lambda in the dagster-agent solution writes it; the secrets solution has its shell

The worker reads auto-swe/typesafe/api-key, a TypeSafe API key, for each of its TypeSafe requests; Tokens and cost lists them. While it is empty, the pull-request agent's classify step fails, and every mention goes to the review-or-answer step; see What a mention asks for. The triage context then leaves a failure's cause unclassified.

The triage agent files its issues as the Linear write app, through the shared token in auto-swe/linear/write-token.

The worker records each credential's expiry in the DynamoDB table auto-swe-credential-expiry, which solutions/auto-swe creates. The auto-swe job in terraform.yml runs after secrets as well as auto-swe-secrets, because the solution looks up the Nessie reader token's shell, which secrets creates.

The first deploy​

Terraform creates the worker function from an image that already exists, and the ECR repository is in the same solution. The first apply therefore runs in two parts.

Before step 1, create the auto-swe.obsii namespace in Temporal Cloud with API-key authentication. In its codec server settings, set the endpoint that ppm-prod.obsii uses and turn on Pass the user access token.

  1. Fill the auto-swe/ secrets.

  2. Create the repository alone:

    cd deployments/aws/terraform/solutions/auto-swe
    terraform init -backend-config="../../environments/us-east-1/auto-swe/prod/backend.tfvars"
    terraform apply -target=aws_ecr_repository.lambda \
    -var-file="../../environments/us-east-1/auto-swe/prod/environment.tfvars"
  3. Re-run the deploy_workflow_lambda job on main. It pushes the image and then fails at the deploy step, since the SSM parameter does not exist yet.

  4. Run the full apply, or re-run the Terraform workflow.

  5. Re-run deploy_workflow_lambda. It registers the first version with Temporal.

  6. Run tcld login, then register the search attributes:

    tcld namespace search-attributes add --namespace auto-swe.obsii \
    --sa "AutoSweSubject=Keyword" --sa "AutoSweWorkItemType=Keyword" \
    --sa "AutoSweAgentTypes=KeywordList" --sa "AutoSweHarness=Keyword" \
    --sa "AutoSweOutcome=Keyword"

    The list is copied from scripts/register_search_attributes.sh, whose list a test keeps equal to auto_swe.agent_runs.search_attributes. A workflow that upserts an unregistered attribute fails its workflow task.

  7. Set the GitHub App's webhook URL to the webhook_url output, with the webhook secret from step 1.

  8. Grant the App these repository permissions:

    PermissionAccess
    Contents, MetadataRead-only
    Pull requestsRead and write
    IssuesRead and write
    Checks, ActionsRead-only

    An agent's github credential can hold only permissions the App has. The github_post_comment, github_post_review and github_reply_in_thread tools need Pull requests write, and so does the 👀 reaction on a comment in the pull request's conversation. Reading the linked issues and receiving the issue comment event need Issues read. The pull-request agent's github credential has read access only. GitHub lets an App subscribe to an event only when the App has the permission the event requires, so the workflow run event needs Actions. An agent that pushes needs Contents write. The ruleset on main, described in Branch protection, blocks its pushes to main.

  9. Subscribe the App to the events the adapter reduces to work items:

    • pull request
    • pull request review comment
    • issue comment
    • check suite
    • workflow run
  10. Open a test pull request and comment @ai-auto-swe review. The App's recent deliveries show a 202 from the webhook. The comment gets 👀. The Cloud UI shows the pull request's session, with its payloads decoded. The pull-request agent posts a review.

The Linear webhook​

The implementation agent runs on the deliveries of the Linear write app's own webhook. A workspace admin sets the webhook up once, after the auto-swe solution is applied:

  1. In the write app's settings in Linear, turn on webhooks, set the URL to the linear_webhook_url output, and subscribe to the Issues and Agent session events categories. As soon as Linear agent session events are on, people in the workspace can mention the app, and they see its Linear agent sessions.
  2. Put the webhook's signing secret in auto-swe/linear/webhook-secret.
  3. Mention the app in a comment on a test issue. Its webhook shows a 200, and the Temporal Cloud UI shows the issue's session.

The worker mints the write app's tokens with the app:mentionable scope, which lets people mention the app. Minting a token with a different scope revokes the app's other tokens. A worker whose token is refused mints a new one.