Skip to main content

Observability

Lifecycle events​

A session emits the same lifecycle events for every agent:

EventWhen
receivedA work item arrived
context_loadedThe context type's activity returned
sandbox_readyThe sandbox was created or reused
agent_loop_started, agent_loop_finishedThe agent loop started or finished. agent_loop_finished includes the agent loop's model-call count
over_budgetThe run budget refused a work item or one of its step runs; the detail names the limit
agent_run_startedAn agent run started. The detail names its first step.
agent_run_finishedAn agent run ended. The detail gives its number of steps, its estimated cost and how it ended.
step_finishedA step of an agent run ended. The detail is <step>: <outcome> -> <next>, where <next> is the next step or end. When the session is stopping, the detail names the next step, which did not run.
failedA context load, a step run or a step's activity failed, or the agent or the step of a resumed agent run does not exist; the session goes on
completed, cancelledThe session ended

The DagsterTriage workflow also emits prematch when its prematch asked Jev about any error of a work item. The detail has one part per distinct error, and is cut to its first 500 characters. Each part starts with the first 100 characters of the error's description. It then names the issues Jev was asked about, with its probability of each, and says whether the error's failures were recorded on one of those issues or left to the agent. The line the worker logs for each prematch has the whole detail, with each error's whole description. The workflow's status counts the outcomes of the distinct errors in these counters, which The prematch defines:

  • counters.prematched;
  • counters.prematch_refused;
  • counters.prematch_ambiguous;
  • counters.prematch_search_refused;
  • counters.prematch_unanswered;
  • counters.prematch_not_written.

It emits blocked_retry when the agent marked a work item blocked and the workflow schedules the work item to run again. The detail names the retry, when it is due and the agent's reason. counters.blocked_retries counts the retries. See Retries of a blocked failure.

Each event goes to three places:

  • the worker log;
  • the status query, which also returns the counters;
  • the workflow's current details in the Temporal UI, which show the latest ten.

Temporal history​

Every activity has a summary, for example:

  • exec: pants test …
  • checkout <owner>/<repo>@<sha>
  • prepare: <script>
  • tool: github_post_review

The session has a static summary that names its subject. A payload over 64 KB, such as the conversation so far or a long command output, is written to external storage, and history holds a reference to it. The codec server resolves those references for the UI and CLI.

Sandbox spans​

Each attempt of a sandbox activity records one OpenTelemetry span:

ActivitySpan
sandbox_start_or_createstart sandbox
sandbox_execrun sandbox command
sandbox_uploadupload files to sandbox
sandbox_upload_scriptsupload scripts to sandbox
sandbox_downloaddownload files from sandbox
sandbox_stopstop sandbox
sandbox_deletedelete sandbox

The span's duration is the attempt's duration. LangSmith keeps a child run's start at or after its parent's, so a span with a LangSmith parent starts no earlier than that parent. When the worker's clock is behind the clock that started the parent, the span starts at the parent's start, and its duration is that much shorter.

Every span has these attributes:

AttributeValue
temporal.workflow_id, temporal.run_idThe session's workflow id and run id
temporal.activity_id, temporal.activity_type, temporal.attemptThe activity and its attempt number
sandbox.idThe Daytona sandbox
langsmith.trace.id, langsmith.span.parent_idThe LangSmith trace, and the run that scheduled the activity, such as the agent's tool call
langsmith.span.id, langsmith.span.dotted_orderThe span's own LangSmith run id, and its position in the trace
langsmith.trace.session_nameThe LangSmith project

langsmith.trace.id is the id of the work item's run, which is the trace's root run. Every sandbox span of a step run has the LangSmith attributes, including the spans of the checkout and the setup script. LangSmith reads them, so each span appears in the work item's trace as a child of the run that scheduled the activity. A sandbox span outside a step run has no LangSmith attributes.

Each span's input.value and output.value attributes hold JSON, which LangSmith shows as the input and output of the span's run:

SpanInputOutput
start sandboxThe snapshot, the egress, and the id of the sandbox to start, which is null when the session has no sandbox. The input leaves out the secrets and the environment variablesThe sandbox id, and whether the sandbox is new
run sandbox commandThe command, truncated to its first 2,000 characters, and its timeoutThe exit code, whether the command timed out, and the last 4,000 characters of stdout and of stderr
upload files to sandboxThe files' pathsNone
upload scripts to sandboxThe scripts' directory and namesNone
download files from sandboxThe pathsEach file's size in bytes, or its error. The output leaves out the files' content
stop sandbox, delete sandboxThe sandbox idNone

LangSmith records a span's other attributes in the metadata of the span's run, as otel.span.<attribute>. To find a session's sandbox commands, filter runs on the metadata key otel.span.temporal.workflow_id.

A run sandbox command span also has these attributes:

AttributeValue
process.command_lineThe command, cut to its first 2,000 characters
sandbox.command.timeout_secondsThe command's own time limit
process.exit.codeThe command's exit code
sandbox.command.timed_outWhether timeout stopped the command
sandbox.command.output_truncatedWhether stdout or stderr was cut
sandbox.command.resumedWhether the attempt waited for a command an earlier attempt started
auto_swe.agent, auto_swe.work_itemThe agent and work item the command runs for

A span has an error status when the command timed out, the activity raised, or Temporal cancelled the attempt. A non-zero exit code alone does not set an error status.

A span is exported when its attempt ends. An attempt ends within the sandbox settings' attempt_timeout, 10 minutes by default. A command that runs longer shows as one span per attempt, each with the same temporal.activity_id. The first span has sandbox.command.resumed false, and the later spans have it true. To find a slow command, filter spans on temporal.workflow_id and sort them by duration.

Exporting spans​

The worker exports spans over OTLP/HTTP when OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. Without either, the worker records no spans. The exporter reads the standard OpenTelemetry variables:

VariableMeaning
OTEL_EXPORTER_OTLP_ENDPOINTThe collector's base URL. The exporter sends to <endpoint>/v1/traces
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTThe full traces URL, used in place of the base URL
OTEL_EXPORTER_OTLP_HEADERSRequest headers, such as an API key, as key=value,key=value
OTEL_SERVICE_NAMEThe service.name resource attribute; auto-swe by default
OTEL_RESOURCE_ATTRIBUTESMore resource attributes, such as deployment.environment=prod
OTEL_SDK_DISABLEDtrue turns export off when an endpoint is set

The Lambda worker flushes the exporter at the end of each invocation, within the 5-second shutdown-hook allowance of LambdaWorkerHarness. The OpenTelemetry SDK's flush has no time limit of its own, and it can first wait for an export already in progress. The Lambda worker therefore limits each export, retries included, to a third of the allowance, in place of OTEL_EXPORTER_OTLP_TIMEOUT. A backend that does not answer in time loses that invocation's last spans.

The deployed worker exports to LangSmith. The auto-swe Terraform solution sets two of the variables:

  • OTEL_EXPORTER_OTLP_ENDPOINT is https://api.smith.langchain.com/otel. Its value is the otlp_endpoint variable in environments/us-east-1/auto-swe/prod/environment.tfvars.
  • OTEL_EXPORTER_OTLP_HEADERS comes from the auto-swe/otlp/headers secret. Secrets gives its value for LangSmith. resolve_secrets() reads it before configure_tracing() creates the exporter.

LangSmith returns HTTP 200 before it processes a span, and it drops a span whose parent run never reaches it. A span missing from a trace therefore leaves no error in the worker's logs.

To export to another OTLP backend, change otlp_endpoint and the secret's value. Another backend keeps the langsmith.* attributes as plain attributes, so it can still group the spans of a step run by langsmith.trace.id.

The worker's tracer provider is the global tracer provider. It exports only the spans of the auto_swe.sandbox tracer. It does not export the spans that libraries record through the global tracer provider, such as the Anthropic SDK's span of each model call and the Daytona SDK's span of each API call. Each model call is already in the trace as its ChatAnthropic run.

Leave DAYTONA_OTEL_ENABLED unset. With it set, each Daytona client the activities open creates its own tracer provider and instruments every HTTP request the SDK makes.

Why the worker records its own spans​

Daytona's own OpenTelemetry support does not give one span per command with the session's context:

  • The SDK's tracing (DAYTONA_OTEL_ENABLED) records one span per SDK call, such as each status poll of a running command. It also replaces the global tracer provider.
  • The organization's OpenTelemetry configuration in Daytona exports telemetry from inside the sandbox. That telemetry has no Temporal workflow id or agent name.

Temporal's OpenTelemetryPlugin exports a workflow's spans only when the span opened in the workflow ends. A session lives as long as its subject. The sandbox spans therefore link to the workflow through attributes instead.

Search attributes​

The session and the triage workflow set these attributes, so temporal workflow list --query and the UI can filter them:

AttributeTypeValue
AutoSweSubjectKeywordThe subject
AutoSweWorkItemTypeKeywordThe latest work item's type
AutoSweAgentTypesKeywordListThe agents that have run
AutoSweHarnessKeyworddeep_agents
AutoSweOutcomeKeywordrunning, then why the workflow ended: terminal, ended, idle or cancelled

The namespace must have them registered before a workflow runs. See Deploying auto_swe for Temporal Cloud, and scripts/register_search_attributes.sh for the dev server.

Tokens and cost​

The workflows count tokens by model. A usage field maps a model id to a LangChain UsageMetadata. The model ids are:

  • the agent's model for the agent's model calls, such as bedrock:us.anthropic.claude-opus-5-5;
  • typesafe: and the Jev model for the worker's TypeSafe requests, such as typesafe:jev-1.14.0.

auto_swe.usage.add_usages adds usage fields model by model, with LangChain's add_usage.

Each model reply reports its token usage. The workflow sums an agent loop's usage under the agent's model. A model's UsageMetadata has:

  • input_tokens, the sum of all input tokens, cache reads and cache creation included;
  • output_tokens and total_tokens;
  • input_token_details, with cache_read and cache_creation.

For Anthropic models, cache_creation is 0. langchain-anthropic reports cache writes by TTL, in keys that LangChain's UsageMetadata does not declare. The workflow deserializes each reply from the model activity's result, and the deserialization drops those keys.

The workflow estimates the cost of each reply and sums the estimates over the step run. The prices are in auto_swe/harness/model_prices.json, in dollars per million tokens. For an Anthropic reply, the workflow reads Anthropic's usage object in the reply's response_metadata, which includes the cache writes by TTL. Each kind of token has its own price:

  • uncached input;
  • cache reads;
  • cache writes with a 5-minute TTL;
  • cache writes with a 1-hour TTL;
  • output.

The prices come from LiteLLM's price table. scripts/update_model_prices.py rewrites the prices of every model in model_prices.json, and --add adds a model:

uv run --directory workflows/auto_swe python scripts/update_model_prices.py --add anthropic:<model>

A model id is <provider>:<LiteLLM model name>. A provider that charges no premium for cache writes, such as OpenAI's own API, has its cache writes priced as input.

When Bedrock reports a Converse reply's cache writes by TTL, ChatBedrockConverse puts them in keys of input_token_details that the workflow's copy of the reply does not have. The worker's ConverseUsage interceptor therefore sets the reply's cache_creation to their sum before the model activity returns it. The workflow prices cache_creation at the 5-minute cache-write price.

The cost_usd counter sums the estimates. LangSmith also computes a cost for each trace from each model call's tokens and LangSmith's own model prices. See LangSmith. A step run's estimate and its trace's LangSmith cost differ when model_prices.json and LangSmith have different prices for the model.

The session and the triage workflow each record one StepRunRecord (auto_swe/agent_runs/records.py) for every step run:

FieldValue
subject, repositoryThe work item's subject, and its repository as owner/name
agent, stepThe agent, and the step that ran
event, actionThe event and action of the work item that started the agent run
model, cost_usdThe agent's model, and the run's estimated cost in dollars
usageThe run's tokens by model. The tokens of its model calls are under model, and those of its registered tools' TypeSafe requests are under typesafe:<model>
model_calls, started_at, duration_secondsThe number of model calls the run made, when it started, and how long it took
commitThe context's commit, or else the work item's
langsmith_run_idThe id of the step run's LangSmith run
agent_run_idThe id of the agent run the step ran in, or null for a step that DagsterTriage runs.
dry_runWhether the workflow is a dry run

The worker logs each record as JSON, in a line starting with step run. The workflow keeps the last 20 records across continue-as-new. Its status query and its result return them as step_runs. The agent_loop_finished lifecycle event shows the number of model calls and the estimated cost.

The session also records one AgentRunRecord for every agent run, so the cost of a review or a fix is one number:

FieldValue
idThe agent run's id, which its step runs' records and traces name as agent_run_id
agent, work_itemThe agent, and the id of the work item that started the agent run
steps, endedThe steps run, in order, and the step_finished detail of the last one
usage, cost_usdThe tokens and estimated cost that the session spent during the agent run: its step runs, and its contexts' and activity steps' model requests
started_at, duration_seconds, dry_runWhen it started, how long it took, and whether the session is a dry run

The worker logs each record as JSON, in a line starting with agent run. The session keeps the last 20 across continue-as-new, as recent_agent_runs in its state, and its status query and its result return them as agent_runs. The ended_agent_runs counter counts them.

The status query and the result also return the usage and cost_usd counters. usage adds up the tokens of the workflow's step runs and of the worker's own TypeSafe requests, by model. The worker makes these TypeSafe requests:

  • the classifications of comments' intent;
  • the contexts' requests, such as a Dagster failure's cause classification;
  • the requests of the step runs' tools.

cost_usd also includes these requests. Each activity that makes one reports its tokens by model as usage. The workflow computes their cost with auto_swe.harness.pricing.usage_cost when it records the usage, at the model's prices in model_prices.json, as for any other model. A Jev model's entry charges $0.042 per million input tokens and nothing for output tokens, the same price as LangSmith's. A step run's cost_usd includes its tools' requests, also when the run fails, and the cost_usd counter adds the classifications' and contexts' requests.

Spend on the AWS bill​

The auto-swe and auto-swe-secrets Terraform solutions set the tag autointel:group=code-factory on each of their resources through the provider's default tags. The worker's IAM role is one of those resources. Bedrock records a model call's cost with the tags of the IAM principal that made the call, under keys prefixed iamPrincipal/. The cost of the worker's Bedrock calls therefore has the tag iamPrincipal/autointel:group=code-factory.

The auto-swe-code-factory AWS Budget counts the spend with either tag. It emails var.budget_alert_emails when a month's forecast spend exceeds var.aws_monthly_budget_usd, and again when its actual spend does. Billing data reaches Budgets hours after the spend, so the actual-spend alert arrives after the budget is already exceeded. The budget does not count:

  • a call to an anthropic: or openai: model, or a TypeSafe request, none of which is on the AWS bill;
  • a Bedrock call from a local worker, which uses its developer's credentials.

The budget counts the iamPrincipal/autointel:group key only while that key is active as a cost allocation tag of type IAM principal. Someone activates it by hand in the payer account's Billing console. See Activation.

LangSmith​

Tracing is required: the worker does not start without LANGSMITH_API_KEY. It traces each work item to LangSmith, in the project LANGSMITH_PROJECT, as one trace. A work item's trace has these runs:

  • the work item, named <event> work item, which is the root run. Its input is the work item's id. It holds the item's classification and context, which include the worker's TypeSafe requests, and its agent runs;
  • each agent run, named <agent> agent run, a child run of the work item. It holds the agent run's context loads and step runs, and its output is the agent run's steps, how it ended and its cost. After the session continues as new partway through an agent run, the rest of its step runs are under a second <agent> agent run with the same agent_run_id;
  • each step run, named <agent>/<step> step run, a child run of the agent run. It lasts from the sandbox's start to its stop. Its input is the work item's id, and its output is the agent loop's final text;
  • the agent loop, named <agent> agent loop, which is a child run of the step run. Its input and output are the loop's messages;
  • each model call, a child run of the agent loop;
  • the sandbox spans, when the worker exports them to LangSmith.

The work item's run has the metadata keys session, thread_id, subject, work_item, event and action. Each step run has these metadata keys:

  • session, the session's workflow id;
  • subject, repository, agent, step, agent_run_id, event and action, with the step run record's values;
  • dry_run.

An agent run has the metadata keys session, thread_id, subject, agent, agent_run_id and work_item. Filter runs on them in LangSmith. The workflow generates the step run's run id and records it as the step run's langsmith_run_id, and generates the agent run's first run id as the agent run's id.

LangSmithPlugin gives a version 4 id to each run that the workflow creates without a run id. The plugin draws those ids from a generator with the same seed as workflow.uuid4(), so it generates the same sequence of ids that workflow.uuid4() returns. The workflow therefore derives the step run's id from a workflow.uuid4() value as a version 5 UUID. No id that the plugin generates can equal a version 5 UUID.

The work items of one session or triage workflow form one LangSmith thread. Its id is derived from the workflow id, so a continued run stays in it. When the workflow starts, it writes a link to the thread at the top of its current details in the Temporal UI, and the status query returns it as langsmith_thread. A workflow that cannot reach LangSmith as it starts runs without the link. The thread's traces include LangSmith's cost for each model call.

A Lambda worker evicts the workflow from its cache when its invocation ends, and the next worker replays the workflow's history, agent loop included. workflow.unsafe.is_replaying_history_events() also returns True while a worker evicts the workflow. The agent loop's runs are therefore created by auto_swe.harness.deep_agents.tracer:

  • ReplaySafeTracer records the LangChain runs of the loop, such as LangGraph and each node.
  • Each of those runs, and each run of a @traceable function inside one, is a ReplaySafeRun. Its id is derived from its parent's id and its position among the parent's child runs. Its start time is the workflow's time. A replay therefore creates each run again with the same id and start time.
  • A ReplaySafeRun sends nothing to LangSmith while the workflow replays its history.

A run that one worker starts and another worker ends is one run in LangSmith. The agent loop has one LangGraph run. A replay records no error and no second llm run for a model call. LangSmithPlugin gives the same guarantee for the runs that the workflow creates with @traceable, such as the step run and the agent loop.

For a bedrock: model, the activity's run has the model's Bedrock ID as its model name. For a Claude model, the run is ChatAnthropicBedrock, with the provider anthropic-bedrock. For any other bedrock: model, the run is ChatBedrockConverse, with the provider amazon_bedrock. If LangSmith has no price for that provider and name, the run has no cost. To give it one, add a model price in the LangSmith workspace's settings that matches them.

TypeSafe requests​

auto_swe.credentials.typesafe.system_one records each of the worker's TypeSafe requests as a LangSmith llm run:

RunRequest
Jev: comment intentThe classification of what a comment that mentions the App asks for
Jev: failure causeThe triage context's classification of a Dagster failure's cause
Jev: diagnosis gradedagster_submit_diagnosis's grade of a diagnosis
Jev: rubric <step>A rubric step's questions, one Noul per criterion

The run's input is the request's state and questions. Its output is the response. Its metadata has these keys:

  • ls_provider, set to typesafe;
  • ls_model_name, set to the Jev model;
  • usage_metadata, set to the request's tokens.

A classification is a child run of its work item's run, and a grade is a child run of the step run whose tool call asked for it. An eval's request has no parent run.

LangSmith computes each run's cost from its built-in typesafe_jev model price. The price matches the provider typesafe and every Jev model's name. It charges $0.042 per million input tokens and nothing for output tokens.

Feedback​

The session records people's feedback on the agents' work as LangSmith feedback on the trace of the step run it is about. Each record's id is derived from the reaction, reply or command it records, so LangSmith stores a record sent twice only once. The feedback counter counts the records sent.

KeyFeedbackScoreRecorded
reaction👍 or 👎 on an agent's inline comment, thread reply or comment in the pull request's conversation1 for 👍, 0 for 👎When the session ends: the pull request closes, the session is idle for idle_timeout, or it gets end_session
replyA reply in the thread of an agent's comment, with or without a mentionnoneWhen the reply arrives
user_feedback@ai-auto-swe feedback <text>, from someone with write accessnoneWhen the comment arrives, on the session's latest step run

Each record's comment is the person's text, or for a reaction the title of the finding or consideration. A record of a reaction on a thread reply or on a comment in the pull request's conversation has no comment. Each record's source_info has:

  • the person's login and the comment's URL
  • for a reaction, the comment's kind, which is finding, consideration, reply or answer. An answer is a comment in the pull request's conversation.
  • for a reaction on a finding, the finding's severity and category

GitHub sends no webhook for a reaction, so the session reads the reactions through the context type's collect_ratings_activity before it deletes its sandboxes. The App's own 👍 and 👎, which it adds to each comment it posts, are left out. A comment that someone deleted before the session ends has no recorded reactions.

In LangSmith, filter an agent's runs by the agent metadata key and open the Feedback column, or filter by feedback key.