Skip to main content

The triage agent

The triage agent finds the root cause of failed production Dagster runs and failed asset checks, and files each cause as a Linear issue in the Data Platform team's Triage state. It files one issue per root cause, not one per failure: a cause that fails twenty assets has one issue that lists them. A failure it cannot attribute to a cause goes to an issue for the asset, check, step or job that failed. It never changes code or data.

A link on a triage issue's attachment opens this page. The attachment's URL names what it records:

  • ?cause=<category>:<subject> names the cause the issue is for;
  • ?unattributed=<subject> names what failed when triage could not attribute a failure to a cause. The subject is an asset, check, step or run, or, for failures that arrived over the run budget, a code location and a day;
  • ?symptom=<fingerprint> names a failure recorded on the issue.

Triage does not write to an issue that has a ?standing=needs-human attachment.

How a failure reaches it​

The TriageSensorsComponent in projects/ai_org/src/ai_org/defs/triage/defs.yaml defines two sensors that watch every code location of the deployment:

  • triage_run_failure_sensor sends each failed run, with each failed step, its asset keys and its root-cause exception. A step that failed on a blocking asset check is left to the check sensor.
  • triage_asset_check_sensor sends each run's failed asset checks.

Each sensor sends a failure only when:

  • the run was launched by one of the component's run_origins, or the code location sets AUTO_SWE_TRIAGE_ALL_ORIGINS to true;
  • Dagster will not retry the run (dagster/will_retry is not true);
  • for a check, its severity is one of check_severities.

Set AUTO_SWE_TRIAGE_ALL_ORIGINS on the ai_org code location in Dagster+, then redeploy the code location. The sensors read the new value from their next tick.

A branch deployment gets no triage sensors.

AttributeDefaultMeaning
run_origins[schedule, automation_condition]The launchers whose failed runs are triaged: schedule, automation_condition, sensor, backfill, manual, or all
check_severities[ERROR]The check severities triaged; add WARN to triage warnings too
minimum_interval_seconds600How often each sensor checks for new failures, in seconds. A tick of the run failure sensor reads at most 5 failed runs, which is Dagster's default limit for a run status sensor. Failed runs that the sensor does not triage count toward that limit. A tick of the check sensor reads at most 100 check evaluations
dry_runtrueGrade and report what would be filed, and write nothing to Linear
runs_per_hour, runs_per_day12, 60The run budget's run limits; see below
usd_per_hour, usd_per_day20, 100The run budget's dollar limits; see below

The triage workflow​

Each sensor starts the deployment's DagsterTriage workflow, dagster-triage/prod, if none is running, and signals it the failure as a dagster.failure work item: one failed run, or one run's failed checks. The workflow is defined in auto_swe/triage/workflow.py, and it runs on the session queue. It is not a session. It takes only dagster.failure work items, and runs the triage agent's diagnose step on each one. The triage agent is off by default, so a session that receives a dagster.failure work item runs no agent on it.

For each failure, the workflow:

  1. checks the run budget's dollar limits;
  2. runs the prematch, which may record the failure without a step run;
  3. checks the run budget;
  4. loads the failure's context with dagster_failure_context;
  5. runs the step that the triage agent's matching trigger names, diagnose, with the StepRunner the sessions use. It runs that one step, and does not follow the step's on;
  6. schedules the failure to run again when the agent marked it blocked; see Retries of a blocked failure.

It handles one failure at a time, so each diagnosis sees the issues that earlier diagnoses filed. The workflow:

  • ignores a failure it has seen;
  • ends after 24 hours with no failure;
  • continues as new when its history grows long.

Its status query lists its counters, lifecycle events, recent step runs, the failure it is handling and its scheduled retries.

The run budget​

TriageConfig.run_budget limits the step runs the workflow starts, and their cost, in a rolling hour and a rolling day. Its limits, and how a run's cost counts toward them, are in The run budget.

The workflow keeps the runs it counts in its own state, and passes them on when it continues as new. Without an end_triage signal, it ends only after 24 hours with no failure. By then, every run it counted is more than a day old, so the next workflow has no earlier run to count. A workflow that an end_triage signal ends sooner does not pass its runs on.

A failure that arrives over any limit runs no agent and loads no context. For that failure, the workflow:

  • emits an over_budget lifecycle event;
  • counts the failure in counters.over_budget;
  • runs dagster_triage_overflow, which records the failure on the issue for its code location and the day, in UTC, as Failures that triage cannot attribute describes.

The dollar limits count the workflow's model requests outside its step runs as well as the step runs. These requests are:

  • the prematch's questions;
  • the context's classification of the cause;
  • the grade of each diagnosis.

Each activity reports the tokens of its requests by model, and the workflow prices them at model_prices.json's prices when it records them. The prematch is not a step run, so only the dollar limits apply to it.

Retries of a blocked failure​

The agent marks a group blocked when one of its tools fails, or its access to Nessie, Linear or GitHub fails. A work item has one member, so it has one group, and that group is the whole work item. For a blocked group, dagster_submit_diagnosis files nothing and returns the step's blocked outcome, with the agent's text as its reason. The step's on maps blocked to end, as the loader requires of every outcome a step's tool returns.

DagsterTriageWorkflow reads the outcome of the step run it started. When the outcome is blocked and the work item has had fewer retries than TriageConfig.blocked_retries, the workflow:

  • counts the retry in counters.blocked_retries;
  • emits a blocked_retry lifecycle event, whose detail names the retry, when it is due and the reason;
  • logs a warning;
  • starts a workflow timer of TriageConfig.blocked_retry_delay.

When the timer fires, the workflow puts the work item, with only the failures the prematch did not record, at the end of its queue. The queue ignores a work item it has received, so the workflow adds the retried work item to the queue directly. The retry then goes through every step of the list above, starting with the check of the dollar limits. So the prematch can record the retried failures on an issue that was filed after the first step run.

When the agent marks the work item blocked in the step run of its last retry, dagster_triage_blocked records its failures, with the agent's reason, as Failures that triage cannot attribute describes. A workflow that ends while a retry is scheduled cancels the retry's timer and records the work item the same way. A workflow that continues as new passes its scheduled retries on to the next run, which starts a timer for each of them. It also passes on the retry count of each retried work item that is in its queue.

SettingDefaultMeaning
blocked_retry_delay1 hourHow long after a blocked step run the workflow runs the step again
blocked_retries2How many times the workflow runs the step again for a work item that the agent marked blocked

The defaults assume that the agent is blocked by a service that it or its tools read, such as Nessie, Linear, GitHub or Dagster+, and that the service is down or refuses its requests. The delay gives such an outage an hour to end before each retry. A failure that is still blocked after the last retry is filed about two hours after its first step run. Two retries cost at most two more step runs for a work item, and each of them counts toward the run budget. The triage sensors send neither setting, so changing them is a change to the workflow's code.

The prematch​

After the check of the dollar limits, dagster_triage_prematch looks for the issue whose cause explains each distinct error of the work item's failures. The failures of an error with such an issue are recorded on it with no step run. A failure recorded this way may be:

  • the same exception on the same asset in another partition;
  • a check that fails again;
  • the same cause failing an asset the issue does not list.

The agent gets the work item with only the failures the prematch did not record. When the prematch recorded them all, no step runs.

The prematch groups the failures by kind, check, exception class and message. Each group is one distinct error, and the prematch handles each group on its own. A group may be failed steps, failed checks, or a run that failed with no step failure. It handles at most TriageConfig.prematch_max_errors groups, 10 by default, because each search counts toward Linear's limit of 30 searches a minute. The failures of any later group go to the agent.

  1. Linear's semantic search finds the issues for the error. The query is the one-line description of the group's first failure, as the triage prompt lists it. The search returns at most TriageConfig.prematch_candidates, 5 by default, of the issues with a cause attachment that are:

    • open;
    • canceled within reopen_within_days, 14 days;
    • completed within reopen_within_days in the team that dagster_submit_diagnosis reopens issues in.

    The prematch replaces a duplicate with the issue it duplicates. It keeps that issue only when the issue is open or was closed within reopen_within_days, and when dagster_submit_diagnosis would record a failure on it. When Linear refuses the search because it is over its limit of 30 searches a minute, the prematch searches again after 5, 10, 20 and 30 seconds. When Linear still refuses it, refuses it for another reason, or finds no issue, the group goes to the agent.

  2. Jev answers two questions about the issues in one request. One is its probability that one of the issues has the failures' root cause. The other, asked when there is more than one issue, is which issue has it, with a probability for each.

  3. The group's failures are recorded on the issue Jev chose when:

    • the first probability reaches TriageConfig.prematch_threshold, 0.7 by default;
    • Jev's probability of the issue it chose reaches TriageConfig.prematch_choice_threshold, 0.7 by default.

    Otherwise the group goes to the agent. When the request to Jev fails, only that group goes to the agent.

Linear's search reads an issue's title and description, not its attachments, so it matches a failure to the Cause section and the table of failures the issue was filed with.

For each issue, Jev reads:

  • the issue's title and Cause section;
  • the errors recorded on the issue for the group's symptoms, or, for an issue that holds none of those symptoms, the errors recorded on its other symptoms.

Each option of the question about which issue has the cause is one of these issues, with its title, Cause section and errors.

Jev also reads the group's failures. For failed steps, it reads the errors of the group's steps in the run's log, with their innermost frames. For a failed run, it reads the errors of the run as a whole. The activity reads the run's log only when a group of failed steps or a failed run has issues to ask about.

The activity records the failures as a filing records failures on the issue for their cause. It writes the symptom and run attachments, and a comment when an asset or check the issue did not list has failed. It reopens a Done issue, with a comment that the cause failed again. Errors recorded on the same issue share one set of writes. When Linear refuses an issue's writes, the failures of the errors recorded on it go to the agent.

The activity has the context step's timeout, 10 minutes, because it reads the same run log. When it fails, every failure continues to the agent. The activity reads no repository settings, so the reopen window and team are the defaults of TriageSettings.

The worker logs one line per prematch. The line names the work item and its symptoms. It then says, for each distinct error, how the prematch routed its failures, with each issue Jev was asked about and Jev's probability of it, or why Jev was not asked. When Jev was asked about any error, the workflow also emits a prematch lifecycle event with the same routing. It counts the distinct errors:

  • that the prematch recorded, in counters.prematched;
  • whose probability that an issue has their root cause was below prematch_threshold, in counters.prematch_refused;
  • whose probability reached prematch_threshold while Jev's probability of the issue it chose was below prematch_choice_threshold, in counters.prematch_ambiguous;
  • whose search Linear refused, in counters.prematch_search_refused;
  • for which the request to Jev failed, in counters.prematch_unanswered;
  • whose writes Linear refused, in counters.prematch_not_written.

The prematch runs for every failure. The workflow's own config, TriageConfig, sets:

  • prematch_threshold;
  • prematch_choice_threshold;
  • prematch_candidates;
  • prematch_max_errors;
  • prematch_model, the Jev model the prematch asks, which needs a price in model_prices.json.

The triage sensors send none of them, so changing them is a change to the workflow's code. In a dry run, the prematch asks Jev and writes nothing.

What it reads​

Before the agent starts, the worker reads, for the failed run:

  • the run's step failures, with the deepest exception and its innermost frames outside Dagster's own code;
  • the issues each failure's symptom was attributed to before;
  • the open triage issues, those created in the last day, and those closed in the last 30 days, each with its cause key.

Jev then rates how likely each category of cause below is, from the failures and the run's errors, in one Choice question. The agent starts from the categories Jev rates highest, so it needs fewer turns to reach the cause.

The agent then investigates in a sandbox with the repository checked out at the run's commit. It works through a fixed list of hypotheses, from a recent deploy to an out-of-memory task, and records the evidence that confirms or rules out each.

What the sandbox reaches​

The credentials and sandbox.egress of agents/triage/agent.yaml give the sandbox access to:

  • Nessie, read-only, which serves the catalog and each table's metadata;
  • Linear, read-only;
  • GitHub, read-only, for the repository's commits;
  • GitHub's release downloads, for the setup script;
  • PyPI, for uv run.

By design, it cannot reach:

  • Dagster+, except through the agent's tools. The sandbox has no Dagster+ token.
  • The third-party sources the assets fetch from. The egress proxy refuses them with CONNECT tunnel failed, response 403.
  • AWS APIs, such as the ECS API. The sandbox has no AWS credentials.
  • Object storage. The proxy refuses it with 403, so a table scan fails. The table's metadata still comes from Nessie.

The instructions tell the agent to diagnose from what its tools, the checkout, Nessie and Linear show, and to let Jev grade that evidence. The agent marks a group blocked only when one of its tools fails, or its access to Nessie, Linear or GitHub fails. It never marks a group blocked because one of the limits above stopped a check.

The repository's CLAUDE.md tells agents to run dg commands through the dagster-cli skill, whose commands read Dagster+ with a Dagster+ token. The instructions tell the triage agent that this does not apply to it.

The agent's repo_context.skills.only names the only skills it gets. The agent does not get a skill added to the repository later. It gets these skills:

  • content-placement and writing-style, from .claude/skills/, for the text of its diagnosis;
  • dagster-expert, from the dagster plugin, for how Dagster's assets, sensors, automation conditions and components work. Its pages on dg api do not apply, since the instructions forbid dg;
  • typesafe-ai, from the typesafe plugin, for the code that asks Jev.

The plugins are the ones .claude/settings.json enables.

The Dagster+ tools​

The agent reads Dagster+ with the tools below, which the worker runs with the read token against the failure's deployment:

ToolReturns
dagster_run_eventsA run's step and run failures, with their exceptions and frames, and each asset check evaluation it recorded, failed ones first, with the evaluation's metadata. step_key limits it to one step
dagster_step_logsOne step's log, oldest first: each line the step's code logged through context.log and each event Dagster recorded for the step, with its level and event type. It lists WARNING and above unless level names a lower level
dagster_check_historyAn asset check's latest executions, newest first: partition, status, run, description and metadata. partition limits it to one partition, and limit, at most 30, sets how many

dagster_step_logs reads the run's event log, which dagster_run_events also reads. It lists the warnings a step logged before it failed, such as a source's redirects. A line that a Python logger writes without a Dagster handler is in the step's captured stderr only. No tool reads the captured stderr.

Each tool truncates a metadata value or a log message to 1,500 characters and its whole result to 30,000. The tools act only on dagster.failure work items, so only the triage agent can name them.

How a diagnosis is graded​

The agent submits its diagnosis of the run with one call of dagster_submit_diagnosis, as one group. It gives a cause key, <category>:<subject>, the cause, its evidence and what it ruled out.

The categories follow what a failure's error shows, because Jev classifies the failure before anyone investigates it. An error does not tell a table in a bad state from a mistake in the code that reads it, so both are data-contract, and the subject names the side that has to change. code-defect is left to logic or configuration that is wrong whatever the data.

CategorySubject
external-sourceThe key of the asset that reads the third-party source
credentialsThe key of the asset whose source refused the credential, or the name of the service that refused it
data-contractThe key of the asset that has to change: the upstream whose output no longer fits its readers, or the reader that reads it wrongly
code-defectThe dotted module path of the defect: logic or configuration that is wrong whatever the data
data-validationThe key of the asset whose data broke a rule and was refused
platformThe name of the service that failed, in lowercase
resourcesThe code location whose runs lacked memory, time or the compute to start

Jev, TypeSafe's System One model, grades each group in one request:

  • Conviction, a Score on five levels: speculation, pattern match, supported by context, validated by system state, validated with alternatives ruled out.
  • Contradicted: whether any evidence contradicts the cause.
  • Alternative: whether another cause, not ruled out, fits as well.
  • Same cause: when the agent matched an existing issue, whether it has the same root cause and not only a similar symptom.

Code, not the model, decides the outcome. A group is identified when its conviction is at least min_conviction (supported by context), contradiction is at most max_contradicted, and the alternative is at most max_alternative. Any other group is inconclusive.

Where a group goes​

The groupGoes to
Identified, and Jev finds the matched issue has the same cause (same_cause)The matched issue
Identified, and an issue already holds its cause keyThat issue
Identified, and that issue is openRecorded on it. A comment is posted only when the cause fails an asset or check the issue did not list
Identified, and that issue was canceledRecorded on it, without reopening
Identified, and that issue was Done within reopen_within_daysReopened to Triage, with a comment
Identified, and that issue was Done earlierA new issue, related to it
Identified, with no issue for its causeA new issue, related to any issue Jev found related but not the same
InconclusiveThe issue for each of its unattributed: keys; see Failures that triage cannot attribute
Blocked, because one of the agent's tools or its own access failedNot filed. The workflow runs the step again. After the last retry, the workflow files the group's failures on the issue for each of their unattributed: keys. See Retries of a blocked failure

A new issue goes to the team's Triage state, unassigned. Its priority is the one the agent gave the group. A group without one gets:

  • High when a step or run failed;
  • Medium when an ERROR check failed;
  • Low otherwise.

Its labels are labels, the category and the code locations.

Its description has these sections:

SectionHolds
CauseThe cause, with the agent's optional data sample as a table and its optional Mermaid diagram
FixWhat has to change, and where. Left out when the agent gave no fix
Next stepOne action that confirms the problem or the fix, with its command
ImpactThe failures the issue was filed for: a table of the failed assets, steps and runs, and a table of the failed checks, each with its code location, run and partition. A table with no rows is left out
EvidenceJev's grade and each item of evidence, with path:line@commit sources linked to GitHub. The section is collapsed
Ruled outEach hypothesis and the evidence against it. The section is collapsed

The triage tools write the description once, when they create the issue, and never change it; people may edit it. Every later failure of the cause goes to the issue's attachments, which Linear keys by URL, so recording a failure again updates the same attachment:

  • the cause attachment, whose metadata records the cause key and when a failure of the cause was first and last recorded. Its subtitle shows how long ago the last failure was recorded;
  • one attachment per symptom. Its title is the name of each asset and the check or error, followed by the rest of the asset key. Linear cuts a long title at its end, so the names come first, and the rest of the key keeps the titles of one issue's symptoms distinct. Its subtitle shows the code location, its count of failures and how long ago the last one was recorded. Its metadata records the failure and its last 10 runs, at most one failure per run. The metadata also holds the attributes and messages that Linear shows when a person opens the attachment:
    • the asset, check, severity, error, counts and partitions;
    • one message per run, with an excerpt of its error and its link;
  • one attachment per failed Dagster run, at the run's URL, with no limit. Linear's attachmentsForURL finds a run's issue from that URL.

The times in these attachments are when the triage tools recorded each failure, which follows the failure itself by however long the triage took.

Failures that triage cannot attribute​

Triage cannot attribute the failures of a group when:

  • Jev's grade of its diagnosis is inconclusive;
  • the agent marked it blocked in the step run of its last retry;
  • it arrived over the run budget, so no agent ran.

Each failure of such a group goes to the issue for its unattributed: key. In an inconclusive or blocked group, the subject of a failure's key names what failed and does not depend on the error, so the next failure of the same asset, check, step or run gets the same key. In a group that arrived over the run budget, every failure gets one key for its code location and the day, in UTC, whatever failed. The failures of a code location that arrive over the budget on one day therefore go to one issue, and not to one issue for each asset. The keys are:

The failureIts key
A step that wrote assetsunattributed:<asset key>, with the keys joined by commas when the step wrote more than one
A checkunattributed:<asset key>:<check name>
A step that wrote no assetunattributed:<code location>/<step key>
A run that failed with no failed stepunattributed:<asset keys>, the run's planned assets sorted and joined by commas. A run with no planned asset, or one whose events Dagster+ cannot return, gets unattributed:<code location>/<job name>
Any failure that arrived over the run budgetunattributed:overflow/<code location>/<YYYY-MM-DD>

The planned assets of a run come from its AssetMaterializationPlannedEvents, which the triage tools read from Dagster+ when they file the failure. A run's job is almost always __ASSET_JOB, so a key from the job name would put every run of a code location that failed with no failed step on one issue.

A subject longer than 100 characters is cut to its first 91 characters, followed by ~ and the first 8 hex digits of the SHA-256 of the whole subject. The cut keeps the attachment URL short, and the hash keeps two long subjects with the same first 91 characters apart. A group whose failures have several keys goes to several issues.

The issue for a key is the one with the attachment ?unattributed=<subject>. Triage finds it the way it finds the issue for a cause key. It takes the oldest open issue with the attachment, or else the one closed last. Then:

  • an open issue is recorded on, with a comment;
  • an issue in the team that was closed, Done or Canceled, within reopen_within_days is reopened to Triage, with a comment;
  • any other closed issue gets a new issue related to it.

A canceled issue for a cause is recorded on without reopening. A Canceled issue for an unattributed: key is reopened, because canceling it does not say why its failures happen. The comment holds:

  • why triage could not attribute the failures;
  • the failures;
  • the agent's proposed cause, proposed fix and evidence, when the agent gave them.

Several keys of a group can have one issue, such as after a person marks one key's issue a duplicate of another's. The group's failures with those keys then go to that issue together, with one comment and at most one reopening.

A new issue goes to the team's Triage state. Its priority is the one a group gets when the agent gave it none. Its labels are labels, unattributed_label and the code locations. Its title is Failures of <subject> that triage could not attribute. For failures that arrived over the run budget, its title is Failures in <code location> on <day> that arrived over the triage run budget. A subject that would make the first title longer than 120 characters is cut and ends with an ellipsis. 120 characters is also the longest title the agent may give an issue for a cause. The description has these sections:

SectionHolds
Why triage could not attribute itJev's reasons for an inconclusive grade, the agent's blocked text, or the run budget's limit that was spent
Proposed causeThe agent's cause key and cause. Left out when the agent gave no diagnosis
Proposed fixThe agent's fix. Left out when it gave none
Next stepFor an inconclusive group, an instruction to confirm or rule out the proposed cause, followed by the agent's next step. Otherwise, an instruction to investigate the failures and to rerun the failed run if its error looks transient
ImpactThe failures, as on an issue for a cause
Evidence and Ruled outAs on an issue for a cause. Left out when the agent gave no diagnosis

The issue has no Cause section and no cause attachment, because it holds no cause.

unattributed is a prefix of the key and not a CauseCategory. The agent picks its category from CauseCategory. The tool's arguments and the prompt's Subjects section allow only its members, so the agent cannot give an unattributed: key. The URL of the attachment for the issue's key starts with ?unattributed= and not with ?cause=, so every query of the issues with a cause attachment leaves the issue out:

  • the prematch never asks about one, so the agent diagnoses the failure again;
  • the prompt's Issues list leaves it out;
  • the prompt's Symptom history names it as an issue for failures that triage could not attribute. The agent can find its identifier there, and a match the agent gives to one is ignored.

When a later diagnosis of a failure with the same unattributed: key identifies its cause, the group goes to the issue for that cause key, as Where a group goes describes. The issue for the unattributed: key keeps the failures recorded on it, and nothing merges the two issues. A person who finds both marks the issue for the unattributed: key a duplicate of the cause's issue, or closes it. For a duplicate, Linear moves its ?unattributed= attachment to the cause's issue, so later failures with that key are recorded there, with a comment. The cause's issue keeps its one cause attachment, so it stays the issue for its cause.

Correcting it​

  • Mark a duplicate. Linear moves the duplicate's attachments to the issue you choose, so the next failure with that cause or symptom is recorded there.
  • Write .auto-swe/triage.md on main for standing guidance, such as known noise or the subject to use for a source. The agent reads it with its instructions.
  • Override a setting in .auto-swe/config.yaml on main, under triage.
SettingDefaultMeaning
linear_teamDPFThe key of the team new issues go to
labels[auto-triage]Labels on every new issue
reopen_within_days14How long after a Done issue, or a Canceled issue for an unattributed: key, closes a recurrence reopens it
unattributed_labelneeds-investigationThe label on every new issue for failures that triage could not attribute, besides labels
gradingsee agents/triage/agent.yaml; the model defaults to JEV_MODELJev's model and the thresholds above

dagster_triage_overflow, which records a work item over the run budget, does not read these settings, so an issue it creates goes to DPF with the default labels.

Evaluating it​

workflows/auto_swe/evals/triage/cases.yaml is a pydantic-evals dataset. Each case is a set of failed production runs whose causes a merged fix confirmed. python -m evals.triage.run starts a Temporal dev server and runs the session worker in the eval's own process. It sends each case's runs to a separate dry-run triage workflow for that case. In a dry run the prematch still asks Jev, and a run it records on an open issue reaches no agent. The eval scores the groups the agent filed, and counts each run the prematch recorded as filed under the cause key of that issue:

  • Reached: causes whose failures were all diagnosed, not marked blocked;
  • KeyMatches: causes filed under the confirmed cause key;
  • FalseMerges: cause keys that failures of more than one cause were filed under, so that they share an issue;
  • FalseSplits: causes whose failures were filed under more than one cause key, so that they are on more than one issue.

A false merge hides a second cause behind the first, so it costs more than a false split, which a person fixes by marking a duplicate.

The eval ends a case's workflow once the workflow has handled the case's runs. So a run that the agent marked blocked gets no retry, and its cause does not count toward Reached. The report also gives each completed case's estimated cost, token counts and prematched, the number of its runs the prematch recorded. Its totals of estimated cost and of tokens by model include the cases that failed or timed out. Run the eval before changing the instructions, the model or the grading thresholds:

set -a; . ~/.config/auto-swe-poc/secrets.env; set +a
uv run --directory workflows/auto_swe python -m evals.triage.run

The process needs:

  • the worker's credentials;
  • workflows/auto_swe/.env, for the payload directory;
  • a dg plus login.

It runs the dev server with the temporal CLI on PATH. Without one on PATH, the Temporal SDK downloads the CLI. Other options:

  • --case <name>: run only that case. Repeatable.
  • --repeat <n>: run each case n times. Each repeat of a case costs as much as a production triage of its runs.
  • --max-concurrency <n>: run up to n cases at once. The default is 1.
  • --existing-worker: use the server that TEMPORAL_ADDRESS names and the worker polling it, as workflows/auto_swe/scripts/README.md starts them.

The eval exits with status 1 when a case, an evaluator or a report evaluator fails.

workflows/auto_swe/evals/cause/cases.yaml is a pydantic-evals dataset of failed production runs, one per kind of error. Each case has the cause category that a merged fix, the run's own events or a ruling on that kind of failure established. Each case records the text the triage context gives Jev for its run. The cases share no error with the examples in CAUSE_CRITERIA. python -m evals.cause.run asks Jev to classify each case twice in one request, with those examples and without them. For each of the two questions, the report gives:

  • correct: whether Jev answered the expected category or an accepted one;
  • p(right): Jev's probability for those categories;
  • the number of right answers and the mean p(right), by expected category and over every case;
  • the requests' tokens by model, and their estimated cost.

The eval asks only TypeSafe, so it needs TYPESAFE_API_KEY and no Temporal server. Run it before changing the categories, their examples or JEV_MODEL, and add a case when triage misclassifies a new kind of failure:

uv run --directory workflows/auto_swe python -m evals.cause.run

--case <name>, --repeat <n> and --max-concurrency <n> work as they do for the triage eval. By default, every case runs at once. The eval exits with status 1 under the same conditions as the triage eval. python -m evals.cause.capture reads production Dagster+ and records the text of each case that has no recorded text. With --all, it records the text again for every case.

workflows/auto_swe/evals/search/cases.yaml holds one case for each symptom record on the issues for a diagnosed cause. A case is the failure the record describes, the time its latest run was recorded, and the issue that holds the record. python -m evals.search.run runs the prematch's semantic search for each failure as of that time, with the prematch's own query, filter and number of candidates. For each case, the report gives:

  • found: whether the issue that holds the record is among the candidates, with its rank, or the candidates when it is not;
  • first: whether that issue is the nearest candidate.

It then counts both separately for two sets of failures:

  • the failures each issue was filed from, which its description was written from;
  • the failures recorded on the issue later.

Linear's search reads an issue's title and description, not its symptom records. The issue that holds a failure's record can therefore rank below other issues.

The eval reads Linear with LINEAR_API_KEY. Each case makes one semantic search, and Linear allows 30 a minute, so --max-concurrency is 4 by default. --candidates <n> sets the number of candidates the search returns. Run it before changing the prematch's query or filter, and as issues accumulate, since more issues compete for the same candidates:

LINEAR_API_KEY=... uv run --directory workflows/auto_swe python -m evals.search.run

python -m evals.search.capture writes cases.yaml again from Linear's current issues. A symptom record without a kind field, written before symptom records had one, has no case.

Turning it on​

  1. Fill these secrets; see Secrets:
    • auto-swe/linear/write-oauth-client. The agent files its issues as the Linear write app.
    • auto-swe/typesafe/api-key
    • auto-swe/dagster/read-token
  2. The secret dagster/temporal/api-key holds the API key of a Temporal Cloud service account. In Temporal Cloud, give that service account write access to the auto-swe.obsii namespace. projects/ai_org/container_context.yaml sets AUTO_SWE_TEMPORAL_API_KEY_SECRET_ID to dagster/temporal/api-key.
  3. Watch the workflow in dry run. Each step run's dagster_submit_diagnosis result says what it would have filed, and the workflow's history in the Temporal UI shows that result. temporal workflow query --namespace auto-swe.obsii --workflow-id dagster-triage/prod --name status lists the step runs.
  4. Set dry_run: false in the component's defs.yaml. The sensors send their settings with each failure, so the running workflow writes to Linear from the next failure on.

The triage workflow runs the triage agent by name. The agent is off by default, and a session runs no agent on a dagster.failure work item, so no session runs it.