Skip to main content

How a session runs

workflows/auto_swe runs coding agents as Temporal workflows. An inbound adapter reduces an outside event to a work item. The work item goes to the session for its subject. For each agent with a trigger that the work item matches, the session starts an agent run and runs its steps. Agents describes the agent file and its design. An agent writes to the session's subject with registered tools, which run on the worker. Its sandbox's credentials grant read-only access unless the agent sets allow_sandbox_writes.

Dagster failures do not go to a session. Each Dagster+ deployment has its own DagsterTriage workflow, in auto_swe/triage/workflow.py, which runs the triage agent's diagnose step on each failure. It runs the step with StepRunner, in auto_swe/agent_runs/runner.py, which the session also uses.

Both workflows subclass AgentWorkflow, in auto_swe/agent_runs/workflow.py. The base class holds the work-item queue, the lifecycle events, the run ledger, the StepRunner and the recent step-run records. It builds the fields that the two workflows' resume states, statuses and outcomes share. It also handles work items until the workflow ends or continues as new, and deletes the sandboxes when the workflow is cancelled. Each workflow defines run, its signals, its status query, how it handles a work item and the input of its next run.

Cancellation​

When a workflow is cancelled, at whatever step the cancellation arrives, it:

  1. emits the cancelled lifecycle event;
  2. sets AutoSweOutcome to cancelled;
  3. deletes its sandboxes;
  4. ends.

Temporal delivers the cancellation to an activity that the workflow awaits. The workflow receives it from that activity as an ActivityError, which every handler of a failed activity would catch. The worker's ActivityCancellation interceptor, in auto_swe/worker/activity_cancellation.py, raises it in the workflow as asyncio.CancelledError instead, so no handler needs its own check. On a worker or test environment without activity_cancellation_plugin(), the workflow handles a cancelled activity as a failed one and keeps running.

A sandbox whose deletion is in progress when the cancellation arrives is not deleted a second time. The sandbox_delete activity does not heartbeat, so the cancellation does not stop it.

sandbox_start_or_create does not heartbeat either, so a cancellation does not stop a step run's sandbox start, and the teardown runs while the start continues:

  • A sandbox that the activity is creating is left to Daytona's auto-delete interval, because the workflow never receives its id.
  • A resumed sandbox that the activity is starting is deleted by the teardown during its start. If Daytona refuses that deletion, the teardown logs a warning, and the sandbox stops after its auto-stop interval and is deleted after its auto-delete interval.

Work items and subjects​

An inbound adapter reduces an outside event to a WorkItem (auto_swe.work_items.models):

FieldMeaning
idUnique per event. A session ignores an id it has handled.
typeWhat the subject is, such as github.pull_request. It selects the context type.
subjectWhat the session is about, such as github/<owner>/<repo>/pr/<n>.
repo, refThe repository and the commit a sandbox checks out.
event, action, payloadThe event's name and action, and the adapter's model of the event. Agents' triggers match against them.
terminalThe subject has ended, such as a closed pull request. The session ends.

The GitHub adapter reduces each of these deliveries about a pull request to a github.pull_request work item on that pull request's subject:

  • pull_request
  • pull_request_review_comment
  • issue_comment on a pull request
  • check_suite
  • workflow_run

It drops a pull_request, pull_request_review_comment, check_suite or workflow_run delivery whose head branch is in another repository, such as a fork. It also drops a check_suite or workflow_run delivery that names no pull request, and a comment from a bot, which includes the GitHub App's own comments.

An issue_comment delivery does not name the head repository, so the webhook cannot drop a fork's comment. The github_pull_request_context activity checks the head repository instead, and the session runs no agent loop for a fork's pull request.

A linear.issue work item is about one Linear issue. Its subject is linear/issue/<issue id>, and its repository is ppm-data-platform. Its event is issue or comment, as IssueEvent (adapters/linear/models.py) names them. The linear_issue_context activity reads the issue as it is now, and skips the item when the issue is completed, canceled or marked a duplicate. An agent run's later steps read the branch, base commit, commit, findings and review that earlier steps' outcomes added to the payload. The checkout is the payload's commit, or else its base_commit, or else the current head of main.

Comments that address the agents​

A comment that mentions the App's handle, @ai-auto-swe, addresses the agents. The match is case-insensitive. The handle must not be part of a longer handle, such as @ai-auto-swe-bot, or of an email address. A mention in any of these places does not count:

  • a quoted line
  • a code span
  • a fenced code block

A comment that quotes someone's request therefore does not trigger the agents a second time.

For a comment that mentions the handle, the adapter sets one of two flags on the payload, and its command to the first word after the mention, such as review. A trigger's when can match on any of them:

FlagThe comment
mentionedMentions the handle, and the next word is not feedback
feedbackMentions the handle, and the next word is feedback

Comment.mention_text holds the comment's text after its first mention. For a labeled action, the payload's label names the label that was added.

What a mention asks for​

A mentioned comment starts the pull-request agent at its classify step, whose github_classify_intent activity decides what the comment asks for, however it is worded. Each intent is an outcome of the step, and leads to the step that does what the comment asks. The activity asks TypeSafe's Jev one Choice question about the comment's text and where it was written: the pull request's conversation, or a review thread and its file. For a reply in a review thread, the activity also sends the thread's first comment and the last comments before the reply, up to MAX_EARLIER_COMMENTS in all. Jev then classifies a reply such as "why?" by the comment it answers. The answer is one of:

IntentThe comment
reviewAsks for a review in any wording, or holds only the mention
answerAsks a question or makes a remark about the pull request, its code or an earlier comment, including whether something should change
fixAsks for the feedback already given to be fixed, or for the change the comment names
review_and_fixAsks for a review and for its findings to be fixed

The fix and review_and_fix intents change code. The activity returns either one only when Jev's confidence is at least INTENT_SETTINGS.min_confidence_to_change_code, and returns answer below it. The pull-request agent's answer step then tells the agent to say what it would change, and that the person can ask with @ai-auto-swe fix.

The step's step_finished lifecycle event records Jev's choice and its confidence. When the activity fails, such as when TypeSafe is unreachable or GitHub returns an error for a reply's earlier thread comments, the failed lifecycle event records the failure, and the step ends with the failed outcome, which leads to the review-or-answer step.

The activity reads the TypeSafe key from auto-swe/typesafe/api-key, or from TYPESAFE_API_KEY in a local run. INTENT_SETTINGS in auto_swe.adapters.github.intent sets the model. The eval in evals/intent/ measures the question's accuracy on a pydantic-evals dataset of labeled comments. For each intent Jev answered, its report gives the lowest confidence of a right answer and the highest confidence of a wrong one. It also gives the requests' tokens by model, and their estimated cost. It needs TYPESAFE_API_KEY. It exits with status 1 when an answer is wrong, or when a case, an evaluator or a report evaluator fails:

uv run --directory workflows/auto_swe python -m evals.intent.run --repeat 3

--case <name> runs only that case and is repeatable. --max-concurrency <n> limits the requests made at once, which are otherwise all made together. --no-earlier leaves out each case's earlier comments in its thread.

The context activity checks the comment's author before each agent loop. For an author without write access to the repository, it skips the item with the reason <login> has <permission> access, not write. The agent run then ends without an agent loop. The maintain role counts as write. The session acknowledges the item before this check, so a comment from such an author still gets 👀.

The GitHub context type's acknowledge activity, github_acknowledge, reacts 👀 to the comment that starts an agent run, and to a comment that arrives during an agent run and waits for it. A work item from an added label gets no reaction.

The adapter sets WorkItem.feedback on an @ai-auto-swe feedback comment and on a reply in a review thread. Before it routes the item, the session records the feedback in LangSmith. It records a reply only when the reply is in the thread of a comment the agents posted. A reply that also mentions the App gets an answer as well. When the session ends, it reads the 👍 and 👎 on the agents' comments and records them too. See Feedback.

One session per subject​

auto_swe.work_items.dispatch.submit starts the subject's session if it has none, and signals it the work item, in one call. The session is the workflow AutoSweSession with id auto-swe/<subject> on the queue auto-swe. Its SessionConfig applies only when the call starts it.

The webhook function submits a work item only when the work item matches a trigger of an agent that the default SessionConfig enables. It reads the agents for this routing with auto_swe.agents.packaged. A terminal work item that no enabled agent matches, such as a pull request's closed event, goes instead to dispatch.signal_existing. That function signals the subject's running session and never starts one. The session then ends and deletes its sandboxes. The webhook function drops every other work item with a 200, so no session starts and the worker does not run.

A session handles its work items one at a time, in the order they arrive. For each one it:

  1. Loads the agents, in an activity. A running session therefore uses a deployed change to an agent from its next work item.
  2. Finds each agent with a trigger that the item matches, in name order. An agent that is off by default runs only when the session's config names it.
  3. For each of these agents, starts an agent run at the step that the agent's first matching trigger names. Before the first step, it runs the context type's acknowledge activity with the stage started, unless the session is a dry run. A failed acknowledgement does not stop the agent run.
  4. Runs the agent run's steps until the agent run ends. Each step's outcome selects the next step in the step's on. Then it starts the next agent's run.

The steps of an agent run​

Each step runs on the work item that started the agent run, with the agent run's data merged over the item's payload. data holds the data of every outcome so far, each outcome's data merged over the previous.

A step that runs an agent loop:

  1. Ends with the over_budget outcome when the session is over its run budget.
  2. Loads the item's context through the context type's activity. The context can skip the item, for example a pull request that is no longer open. A context that lacks a field its context type declares fails the step.
  3. Checks the run budget again, since the context's model requests count toward it.
  4. Starts a step run. A step run that fails emits a failed lifecycle event, and the step ends with the failed outcome.

An activity step runs its activity, with up to three attempts. An activity that fails emits a failed lifecycle event, and the step ends with the failed outcome.

After each step, the session emits the step_finished lifecycle event and moves the agent run to the step that the outcome leads to. When the outcome leads to a step that the agent run has entered max_rounds times, the agent run enters the step's after_max_rounds instead. That step's own max_rounds applies in the same way. The agent run ends when:

  • the next step is end;
  • the round caps lead back to a step whose cap they already passed;
  • the step's on does not map the outcome;
  • the context skipped the item;
  • the item is terminal, after the first step;
  • the session is stopping.

Agents describes the design of outcomes, round caps and the agent run's state.

Work items during an agent run​

A work item that arrives during an agent run waits in the queue until the session has handled the current item. When the arriving item matches a trigger of an agent the session last loaded, the session acknowledges it with the stage queued as it arrives.

A terminal work item or end_session that arrives during an agent run makes the session stop:

  • Every tool whose spec sets writes refuses its calls from then on, and tells the agent that the session is stopping. Commands in the sandbox are not covered, so an agent with allow_sandbox_writes can still push.
  • The step run in progress is not cancelled.
  • When the step ends, the agent run ends without running its next step. The step_finished event's detail names the step that did not run.
  • The session starts no further agent run for the current item, and handles the queue.

Ending and continuing as new​

The session ends on any of:

  • a terminal work item;
  • the end_session signal;
  • idle_timeout with no work item.

It then deletes its sandboxes. A work item that arrives while the session deletes them starts a new run of the session, which creates new sandboxes. When work items were still pending as the session ended, the new run handles them and then ends for the same reason, without waiting for more.

The session also continues as new when Temporal suggests it, or when the deployment's target version changes. The session checks after each work item, and between the steps of an agent run. It passes the new run its:

  • agents' named conversations;
  • sandboxes;
  • seen ids;
  • pending items;
  • counters;
  • recent events;
  • the records of its recent step runs;
  • the start time and cost of the last day's step runs, which the run budget counts;
  • the comments the agents posted;
  • the last work item;
  • the AgentRun in progress, when it continues as new between steps, and the agents its work item matched that had not started yet.

The new run first loads the agents and resumes the saved AgentRun at its step. It then starts the agent runs of the agents that had not started, on the same work item, and then handles the pending items. When an agent, or its step, no longer exists in the loaded agents, its agent run ends with a failed lifecycle event.

A step run​

A step run is everything the session does for one agent step of an agent run. The step runs its agent with the step's changes to the agent's fields; see Steps. A step run has these stages, and the last one is its agent loop:

  1. Sandbox. The session creates or refreshes the Daytona Secrets for the step's credentials:, which are the agent's unless the step narrows them. It keeps one Daytona sandbox per snapshot and set of credentials, and reuses it across step runs. It creates one when it has none, or when Daytona has deleted it. Before each command in the step run, the session gives a Secret a new value if its value is about to expire. See Security model.
  2. Checkout. The session checks out the context's commit in the sandbox, such as a pull request's current head. When the context names no commit, it checks out the work item's commit. The first checkout in a sandbox clones the repository with its full history. Each later one fetches the commit, and the context's history_from, such as a pull request's merge base.
  3. Setup. The agent's setup script runs once per sandbox. A marker file records that it ran. Then the script that repo_context.base_setup_script names, read from the base branch, runs in each step run until it succeeds in the sandbox, unless a run does not finish. Its failure does not fail the step run.
  4. Repository context. The session finds the checkout's instructions, skills, plugins and hooks by convention, and applies the agent's changes to them. See Configuration.
  5. Prepare. The step's prepare script, or else the agent's, runs in every step run that has one, with the context's prepare_args. The tail of its output fills $prepare_output. The harness then writes the context's files to the sandbox's context_files directory, outside the checkout, and lists their paths in $context_files.
  6. Agent loop. The harness runs the agent loop on Deep Agents. Model calls and tool calls repeat until the model replies without a tool call. A step that sets conversation starts from the agent's conversation of that name in this session; see Conversations. Any other step starts from its prompt alone. The system prompt is the agent's instructions, then the step's instructions, then the agent's base_instructions read from the base branch. Every model call and every sandbox operation is an activity. The agent writes to the session's subject with a registered tool, such as github_post_review, which writes nothing in a dry run. The tool gets the agent's settings with the step's settings over them, and the overrides in the repository's .auto-swe/config.yaml on the base branch over both. An agent that sets allow_sandbox_writes also writes with commands in the sandbox, such as git push. Each tool call, each command and each command's output are in the session's history. In a dry run, such an agent's prompt tells it that its sandbox credentials are read-only and that a refused write is expected.

The session stops the sandbox after the agent loop, and the next step run starts it again. If the session cannot stop the sandbox, Daytona stops it after its auto-stop interval.

The run budget​

run_budget limits the step runs a workflow starts, and their cost, in a rolling hour and a rolling day. It is a field of SessionConfig for the session, and of TriageConfig for the DagsterTriage workflow. RunBudget and RunSpend are in auto_swe/agent_runs/budget.py. A RunSpend records the start time and cost of one step run, or of model requests made outside a step run. Each workflow counts only its own step runs. A limit left unset does not apply. TriageConfig.run_budget defaults to RunBudget(), which sets no limit. The default of SessionConfig.run_budget is listed in SessionConfig. A config that sets its own run_budget replaces both default limits.

Each workflow keeps its counters and its RunSpends in one RunLedger, in auto_swe/agent_runs/ledger.py. StepRunner.run records each step run in the ledger. The workflow records its other counts, such as the work items it handled, and its model requests made outside step runs.

LimitHourDay
Step runs startedper_hourper_day
Dollars spent on step runs and on model requests outside themusd_per_hourusd_per_day

A run's cost is the cost_usd of its step run record, estimated from the model's token usage at list prices. A run's cost is known only when the run ends. The workflow checks the limits before it starts each run, so the run that crosses a dollar limit finishes. After that run, the workflow starts no step run, for the same work item or a later one, until the spend in the rolling hour or day falls below the limit. In a rolling hour or day, the workflow therefore spends at most the limit plus one run's cost. The dollar limits leave out a run that fails before its agent loop makes a model call. Such a run reports no cost, and the run limits still count it.

The dollar limits also count each model request a workflow makes outside its step runs. For the session, these are a mention's intent classification and the model requests of the context type's activity. The triage workflow's requests are listed in The run budget.

A session checks the limits before it loads the context of each agent step, and again after the context loads. Over a limit, the session:

  • runs no step run for the step, which ends with the over_budget outcome;
  • emits an over_budget lifecycle event, which names the step's agent and whose detail names the limit;
  • counts the refusal in counters.over_budget.

The agent run then ends, unless the step's on maps over_budget. When the check before the context finds the session over a limit, the session does not load the context. What the triage workflow does with a failure over its budget is in The run budget on the triage agent's page.

When a workflow continues as new, it passes the start time and cost of the last day's step runs, and of the model requests it charged, to the new run in run_spend.

Running a session locally​

workflows/auto_swe/scripts/README.md is the local runbook. It covers:

  • the dev server;
  • the worker;
  • sending a captured delivery from tests/data/ as a dry run;
  • watching the session;
  • ending the session.

Deployment to AWS is in Deploying auto_swe.