Observability
Lifecycle events
A session emits the same lifecycle events for every agent:
| Event | When |
|---|---|
received | A work item arrived |
context_loaded | The context type's activity returned |
sandbox_ready | The sandbox was created or reused |
agent_loop_started, agent_loop_finished | The agent loop started or finished. agent_loop_finished includes the agent loop's model-call count |
over_budget | The run budget refused a work item or one of its step runs; the detail names the limit |
agent_run_started | An agent run started. The detail names its first step. |
agent_run_finished | An agent run ended. The detail gives its number of steps, its estimated cost and how it ended. |
step_finished | A step of an agent run ended. The detail is <step>: <outcome> -> <next>, where <next> is the next step or end. When the session is stopping, the detail names the next step, which did not run. |
failed | A context load, a step run or a step's activity failed, or the agent or the step of a resumed agent run does not exist; the session goes on |
completed, cancelled | The session ended |
The DagsterTriage workflow also emits prematch when its prematch asked Jev about
any error of a work item. The detail has one part per distinct error, and is cut to
its first 500 characters. Each part starts with the first 100 characters of the
error's description. It then names the issues Jev was asked about, with its
probability of each, and says whether the error's failures were recorded on one of
those issues or left to the agent. The line the worker logs for each prematch has the
whole detail, with each error's whole description. The workflow's status counts the
outcomes of the distinct errors in these counters, which
The prematch defines:
counters.prematched;counters.prematch_refused;counters.prematch_ambiguous;counters.prematch_search_refused;counters.prematch_unanswered;counters.prematch_not_written.
It emits blocked_retry when the agent marked a work item blocked and the workflow
schedules the work item to run again. The detail names the retry, when it is due and
the agent's reason. counters.blocked_retries counts the retries. See
Retries of a blocked failure.
Each event goes to three places:
- the worker log;
- the
statusquery, which also returns the counters; - the workflow's current details in the Temporal UI, which show the latest ten.
Temporal history
Every activity has a summary, for example:
exec: pants test …checkout <owner>/<repo>@<sha>prepare: <script>tool: github_post_review
The session has a static summary that names its subject. A payload over 64 KB, such as the conversation so far or a long command output, is written to external storage, and history holds a reference to it. The codec server resolves those references for the UI and CLI.
Sandbox spans
Each attempt of a sandbox activity records one OpenTelemetry span:
| Activity | Span |
|---|---|
sandbox_start_or_create | start sandbox |
sandbox_exec | run sandbox command |
sandbox_upload | upload files to sandbox |
sandbox_upload_scripts | upload scripts to sandbox |
sandbox_download | download files from sandbox |
sandbox_stop | stop sandbox |
sandbox_delete | delete sandbox |
The span's duration is the attempt's duration. LangSmith keeps a child run's start at or after its parent's, so a span with a LangSmith parent starts no earlier than that parent. When the worker's clock is behind the clock that started the parent, the span starts at the parent's start, and its duration is that much shorter.
Every span has these attributes:
| Attribute | Value |
|---|---|
temporal.workflow_id, temporal.run_id | The session's workflow id and run id |
temporal.activity_id, temporal.activity_type, temporal.attempt | The activity and its attempt number |
sandbox.id | The Daytona sandbox |
langsmith.trace.id, langsmith.span.parent_id | The LangSmith trace, and the run that scheduled the activity, such as the agent's tool call |
langsmith.span.id, langsmith.span.dotted_order | The span's own LangSmith run id, and its position in the trace |
langsmith.trace.session_name | The LangSmith project |
langsmith.trace.id is the id of the work item's run, which is the trace's root run.
Every sandbox span of a step run has the LangSmith attributes, including the spans of
the checkout and the setup script. LangSmith reads them, so each span appears in the
work item's trace as a child of the run that scheduled the activity. A sandbox span
outside a step run has no LangSmith attributes.
Each span's input.value and output.value attributes hold JSON, which LangSmith shows
as the input and output of the span's run:
| Span | Input | Output |
|---|---|---|
start sandbox | The snapshot, the egress, and the id of the sandbox to start, which is null when the session has no sandbox. The input leaves out the secrets and the environment variables | The sandbox id, and whether the sandbox is new |
run sandbox command | The command, truncated to its first 2,000 characters, and its timeout | The exit code, whether the command timed out, and the last 4,000 characters of stdout and of stderr |
upload files to sandbox | The files' paths | None |
upload scripts to sandbox | The scripts' directory and names | None |
download files from sandbox | The paths | Each file's size in bytes, or its error. The output leaves out the files' content |
stop sandbox, delete sandbox | The sandbox id | None |
LangSmith records a span's other attributes in the metadata of the span's run, as
otel.span.<attribute>.
To find a session's sandbox commands, filter runs on the metadata key
otel.span.temporal.workflow_id.
A run sandbox command span also has these attributes:
| Attribute | Value |
|---|---|
process.command_line | The command, cut to its first 2,000 characters |
sandbox.command.timeout_seconds | The command's own time limit |
process.exit.code | The command's exit code |
sandbox.command.timed_out | Whether timeout stopped the command |
sandbox.command.output_truncated | Whether stdout or stderr was cut |
sandbox.command.resumed | Whether the attempt waited for a command an earlier attempt started |
auto_swe.agent, auto_swe.work_item | The agent and work item the command runs for |
A span has an error status when the command timed out, the activity raised, or Temporal cancelled the attempt. A non-zero exit code alone does not set an error status.
A span is exported when its attempt ends. An attempt ends within the sandbox settings'
attempt_timeout, 10 minutes by default. A command that runs longer shows as one span
per attempt, each with the same temporal.activity_id. The first span has
sandbox.command.resumed false, and the later spans have it true. To find a slow
command, filter spans on temporal.workflow_id and sort them by duration.
Exporting spans
The worker exports spans over OTLP/HTTP when OTEL_EXPORTER_OTLP_ENDPOINT or
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. Without either, the worker records no spans.
The exporter reads the standard OpenTelemetry variables:
| Variable | Meaning |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | The collector's base URL. The exporter sends to <endpoint>/v1/traces |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | The full traces URL, used in place of the base URL |
OTEL_EXPORTER_OTLP_HEADERS | Request headers, such as an API key, as key=value,key=value |
OTEL_SERVICE_NAME | The service.name resource attribute; auto-swe by default |
OTEL_RESOURCE_ATTRIBUTES | More resource attributes, such as deployment.environment=prod |
OTEL_SDK_DISABLED | true turns export off when an endpoint is set |
The Lambda worker flushes the exporter at the end of each invocation, within the
5-second shutdown-hook allowance of LambdaWorkerHarness. The OpenTelemetry SDK's
flush has no time limit of its own, and it can first wait for an export already in
progress. The Lambda worker therefore limits each export, retries included, to a third
of the allowance, in place of OTEL_EXPORTER_OTLP_TIMEOUT. A backend that does not
answer in time loses that invocation's last spans.
The deployed worker exports to LangSmith. The auto-swe Terraform solution sets two of
the variables:
OTEL_EXPORTER_OTLP_ENDPOINTishttps://api.smith.langchain.com/otel. Its value is theotlp_endpointvariable inenvironments/us-east-1/auto-swe/prod/environment.tfvars.OTEL_EXPORTER_OTLP_HEADERScomes from theauto-swe/otlp/headerssecret. Secrets gives its value for LangSmith.resolve_secrets()reads it beforeconfigure_tracing()creates the exporter.
LangSmith returns HTTP 200 before it processes a span, and it drops a span whose parent run never reaches it. A span missing from a trace therefore leaves no error in the worker's logs.
To export to another OTLP backend, change otlp_endpoint and the secret's value. Another
backend keeps the langsmith.* attributes as plain attributes, so it can still group the
spans of a step run by langsmith.trace.id.
The worker's tracer provider is the global tracer provider. It exports only the spans of
the auto_swe.sandbox tracer. It does not export the spans that libraries record through
the global tracer provider, such as the Anthropic SDK's span of each model call and the
Daytona SDK's span of each API call. Each model call is already in the trace as its
ChatAnthropic run.
Leave DAYTONA_OTEL_ENABLED unset. With it set, each Daytona client the activities open
creates its own tracer provider and instruments every HTTP request the SDK makes.
Why the worker records its own spans
Daytona's own OpenTelemetry support does not give one span per command with the session's context:
- The SDK's tracing (
DAYTONA_OTEL_ENABLED) records one span per SDK call, such as each status poll of a running command. It also replaces the global tracer provider. - The organization's OpenTelemetry configuration in Daytona exports telemetry from inside the sandbox. That telemetry has no Temporal workflow id or agent name.
Temporal's OpenTelemetryPlugin exports a workflow's spans only when the span opened in
the workflow ends. A session lives as long as its subject. The sandbox spans therefore
link to the workflow through attributes instead.
Search attributes
The session and the triage workflow set these attributes, so
temporal workflow list --query and the UI can filter them:
| Attribute | Type | Value |
|---|---|---|
AutoSweSubject | Keyword | The subject |
AutoSweWorkItemType | Keyword | The latest work item's type |
AutoSweAgentTypes | KeywordList | The agents that have run |
AutoSweHarness | Keyword | deep_agents |
AutoSweOutcome | Keyword | running, then why the workflow ended: terminal, ended, idle or cancelled |
The namespace must have them registered before a workflow runs. See
Deploying auto_swe for Temporal Cloud, and
scripts/register_search_attributes.sh for the dev server.
Tokens and cost
The workflows count tokens by model. A usage field maps a model id to a LangChain
UsageMetadata. The model ids are:
- the agent's
modelfor the agent's model calls, such asbedrock:us.anthropic.claude-opus-5-5; typesafe:and the Jev model for the worker's TypeSafe requests, such astypesafe:jev-1.14.0.
auto_swe.usage.add_usages adds usage fields model by model, with LangChain's
add_usage.
Each model reply reports its token usage. The workflow sums an agent loop's usage under
the agent's model. A model's UsageMetadata has:
input_tokens, the sum of all input tokens, cache reads and cache creation included;output_tokensandtotal_tokens;input_token_details, withcache_readandcache_creation.
For Anthropic models, cache_creation is 0. langchain-anthropic reports cache writes
by TTL, in keys that LangChain's UsageMetadata does not declare. The
workflow deserializes each reply from the model activity's result, and the
deserialization drops those keys.
The workflow estimates the cost of each reply and sums the estimates over the step run.
The prices are in auto_swe/harness/model_prices.json, in dollars per million tokens.
For an Anthropic reply, the workflow reads Anthropic's usage object in the reply's
response_metadata, which includes the cache writes by TTL. Each kind of token has its
own price:
- uncached input;
- cache reads;
- cache writes with a 5-minute TTL;
- cache writes with a 1-hour TTL;
- output.
The prices come from LiteLLM's price table.
scripts/update_model_prices.py rewrites the prices of every model in
model_prices.json, and --add adds a model:
uv run --directory workflows/auto_swe python scripts/update_model_prices.py --add anthropic:<model>
A model id is <provider>:<LiteLLM model name>. A provider that charges no premium for
cache writes, such as OpenAI's own API, has its cache writes priced as input.
When Bedrock reports a Converse reply's cache writes by TTL, ChatBedrockConverse puts
them in keys of input_token_details that the workflow's copy of the reply does not
have. The worker's ConverseUsage interceptor therefore sets the reply's
cache_creation to their sum before the model activity returns it. The workflow prices
cache_creation at the 5-minute cache-write price.
The cost_usd counter sums the estimates. LangSmith also computes a cost for each trace
from each model call's tokens and LangSmith's own model prices. See
LangSmith. A step run's estimate and its trace's LangSmith cost differ
when model_prices.json and LangSmith have different prices for the model.
The session and the triage workflow each record one StepRunRecord
(auto_swe/agent_runs/records.py) for every step run:
| Field | Value |
|---|---|
subject, repository | The work item's subject, and its repository as owner/name |
agent, step | The agent, and the step that ran |
event, action | The event and action of the work item that started the agent run |
model, cost_usd | The agent's model, and the run's estimated cost in dollars |
usage | The run's tokens by model. The tokens of its model calls are under model, and those of its registered tools' TypeSafe requests are under typesafe:<model> |
model_calls, started_at, duration_seconds | The number of model calls the run made, when it started, and how long it took |
commit | The context's commit, or else the work item's |
langsmith_run_id | The id of the step run's LangSmith run |
agent_run_id | The id of the agent run the step ran in, or null for a step that DagsterTriage runs. |
dry_run | Whether the workflow is a dry run |
The worker logs each record as JSON, in a line starting with step run. The workflow
keeps the last 20 records across continue-as-new. Its status query and its result
return them as step_runs. The agent_loop_finished lifecycle event shows the
number of model calls and the estimated cost.
The session also records one AgentRunRecord for every agent run, so the cost of a
review or a fix is one number:
| Field | Value |
|---|---|
id | The agent run's id, which its step runs' records and traces name as agent_run_id |
agent, work_item | The agent, and the id of the work item that started the agent run |
steps, ended | The steps run, in order, and the step_finished detail of the last one |
usage, cost_usd | The tokens and estimated cost that the session spent during the agent run: its step runs, and its contexts' and activity steps' model requests |
started_at, duration_seconds, dry_run | When it started, how long it took, and whether the session is a dry run |
The worker logs each record as JSON, in a line starting with agent run. The session
keeps the last 20 across continue-as-new, as recent_agent_runs in its state, and its
status query and its result return them as agent_runs. The ended_agent_runs
counter counts them.
The status query and the result also return the usage and cost_usd counters.
usage adds up the tokens of the workflow's step runs and of the worker's own TypeSafe
requests, by model. The worker makes these TypeSafe requests:
- the classifications of comments' intent;
- the contexts' requests, such as a Dagster failure's cause classification;
- the requests of the step runs' tools.
cost_usd also includes these requests. Each activity that makes one reports its
tokens by model as usage. The workflow computes their cost with
auto_swe.harness.pricing.usage_cost when it records the usage, at the model's prices
in model_prices.json, as for any other model. A Jev
model's entry charges $0.042 per million input tokens and nothing for output tokens,
the same price as LangSmith's. A step run's cost_usd includes its tools' requests,
also when the run fails, and the cost_usd counter adds the classifications' and
contexts' requests.
Spend on the AWS bill
The auto-swe and auto-swe-secrets Terraform solutions set the tag
autointel:group=code-factory on each of their resources through the provider's
default tags. The worker's IAM role is one of those resources. Bedrock records a model
call's cost with the tags of the IAM principal that made the call, under keys prefixed
iamPrincipal/. The cost of the worker's
Bedrock calls therefore has the tag iamPrincipal/autointel:group=code-factory.
The auto-swe-code-factory AWS Budget counts the spend with either tag. It emails
var.budget_alert_emails when a month's forecast spend exceeds
var.aws_monthly_budget_usd, and again when its actual spend does. Billing data reaches
Budgets hours after the spend, so the actual-spend alert arrives after the budget is
already exceeded. The budget does not count:
- a call to an
anthropic:oropenai:model, or a TypeSafe request, none of which is on the AWS bill; - a Bedrock call from a local worker, which uses its developer's credentials.
The budget counts the iamPrincipal/autointel:group key only while that key is active
as a cost allocation tag of type IAM principal. Someone activates it by hand in the
payer account's Billing console. See Activation.
LangSmith
Tracing is required: the worker does not start without LANGSMITH_API_KEY. It traces
each work item to LangSmith, in the project LANGSMITH_PROJECT, as one trace. A work
item's trace has these runs:
- the work item, named
<event> work item, which is the root run. Its input is the work item's id. It holds the item's classification and context, which include the worker's TypeSafe requests, and its agent runs; - each agent run, named
<agent> agent run, a child run of the work item. It holds the agent run's context loads and step runs, and its output is the agent run's steps, how it ended and its cost. After the session continues as new partway through an agent run, the rest of its step runs are under a second<agent> agent runwith the sameagent_run_id; - each step run, named
<agent>/<step> step run, a child run of the agent run. It lasts from the sandbox's start to its stop. Its input is the work item's id, and its output is the agent loop's final text; - the agent loop, named
<agent> agent loop, which is a child run of the step run. Its input and output are the loop's messages; - each model call, a child run of the agent loop;
- the sandbox spans, when the worker exports them to LangSmith.
The work item's run has the metadata keys session, thread_id, subject,
work_item, event and action. Each step run has these metadata keys:
session, the session's workflow id;subject,repository,agent,step,agent_run_id,eventandaction, with the step run record's values;dry_run.
An agent run has the metadata keys session, thread_id, subject, agent,
agent_run_id and work_item. Filter runs on them in LangSmith. The workflow
generates the step run's run id and records it as the step run's langsmith_run_id,
and generates the agent run's first run id as the agent run's id.
LangSmithPlugin gives a version 4 id to each run that the workflow creates without a
run id. The plugin draws those ids from a generator with the same seed as
workflow.uuid4(), so it generates the same sequence of ids that workflow.uuid4()
returns. The workflow therefore derives the step run's id from a workflow.uuid4()
value as a version 5 UUID. No id that the plugin generates can equal a version 5 UUID.
The work items of one session or triage workflow form one LangSmith thread. Its id is
derived from the workflow id, so a continued run stays in it. When the workflow starts,
it writes a link to the thread at the top of its current details in the Temporal UI,
and the status query returns it as langsmith_thread. A workflow that cannot reach
LangSmith as it starts runs without the link. The thread's traces include LangSmith's
cost for each model call.
A Lambda worker evicts the workflow from its cache when its invocation ends, and the
next worker replays the workflow's history, agent loop included.
workflow.unsafe.is_replaying_history_events() also returns True while a worker
evicts the workflow. The agent loop's runs are therefore created by
auto_swe.harness.deep_agents.tracer:
ReplaySafeTracerrecords the LangChain runs of the loop, such asLangGraphand each node.- Each of those runs, and each run of a
@traceablefunction inside one, is aReplaySafeRun. Its id is derived from its parent's id and its position among the parent's child runs. Its start time is the workflow's time. A replay therefore creates each run again with the same id and start time. - A
ReplaySafeRunsends nothing to LangSmith while the workflow replays its history.
A run that one worker starts and another worker ends is one run in LangSmith. The agent
loop has one LangGraph run. A replay records no error and no second llm run
for a model call. LangSmithPlugin gives the same guarantee for the runs that the
workflow creates with @traceable, such as the step run and the agent loop.
For a bedrock: model, the activity's run has the model's Bedrock ID as its model name.
For a Claude model, the run is ChatAnthropicBedrock, with the provider
anthropic-bedrock. For any other bedrock: model, the run is ChatBedrockConverse,
with the provider amazon_bedrock. If LangSmith has no price for that provider and
name, the run has no cost. To give it one, add a model price in the LangSmith
workspace's settings that matches them.
TypeSafe requests
auto_swe.credentials.typesafe.system_one records each of the worker's TypeSafe requests
as a LangSmith llm run:
| Run | Request |
|---|---|
Jev: comment intent | The classification of what a comment that mentions the App asks for |
Jev: failure cause | The triage context's classification of a Dagster failure's cause |
Jev: diagnosis grade | dagster_submit_diagnosis's grade of a diagnosis |
Jev: rubric <step> | A rubric step's questions, one Noul per criterion |
The run's input is the request's state and questions. Its output is the response. Its metadata has these keys:
ls_provider, set totypesafe;ls_model_name, set to the Jev model;usage_metadata, set to the request's tokens.
A classification is a child run of its work item's run, and a grade is a child run of the step run whose tool call asked for it. An eval's request has no parent run.
LangSmith computes each run's cost from its built-in typesafe_jev model price. The
price matches the provider typesafe and every Jev model's name. It charges $0.042 per
million input tokens and nothing for output tokens.
Feedback
The session records people's feedback on the agents' work as LangSmith feedback on the
trace of the step run it is about. Each record's id is derived from the reaction, reply
or command it records, so LangSmith stores a record sent twice only once. The feedback
counter counts the records sent.
| Key | Feedback | Score | Recorded |
|---|---|---|---|
reaction | 👍 or 👎 on an agent's inline comment, thread reply or comment in the pull request's conversation | 1 for 👍, 0 for 👎 | When the session ends: the pull request closes, the session is idle for idle_timeout, or it gets end_session |
reply | A reply in the thread of an agent's comment, with or without a mention | none | When the reply arrives |
user_feedback | @ai-auto-swe feedback <text>, from someone with write access | none | When the comment arrives, on the session's latest step run |
Each record's comment is the person's text, or for a reaction the title of the finding
or consideration. A record of a reaction on a thread reply or on a comment in the pull
request's conversation has no comment. Each record's source_info has:
- the person's login and the comment's URL
- for a reaction, the comment's kind, which is
finding,consideration,replyoranswer. Anansweris a comment in the pull request's conversation. - for a reaction on a finding, the finding's severity and category
GitHub sends no webhook for a reaction, so the session reads the reactions through the
context type's collect_ratings_activity before it deletes its sandboxes. The App's
own 👍 and 👎, which it adds to each comment it posts, are left out. A comment that
someone deleted before the session ends has no recorded reactions.
In LangSmith, filter an agent's runs by the agent metadata key and open the
Feedback column, or filter by feedback key.