The triage agent
The triage agent finds the root cause of failed production Dagster runs and failed asset checks, and files each cause as a Linear issue in the Data Platform team's Triage state. It files one issue per root cause, not one per failure: a cause that fails twenty assets has one issue that lists them. A failure it cannot attribute to a cause goes to an issue for the asset, check, step or job that failed. It never changes code or data.
A link on a triage issue's attachment opens this page. The attachment's URL names what it records:
?cause=<category>:<subject>names the cause the issue is for;?unattributed=<subject>names what failed when triage could not attribute a failure to a cause. The subject is an asset, check, step or run, or, for failures that arrived over the run budget, a code location and a day;?symptom=<fingerprint>names a failure recorded on the issue.
Triage does not write to an issue that has a ?standing=needs-human attachment.
How a failure reaches it
The TriageSensorsComponent in projects/ai_org/src/ai_org/defs/triage/defs.yaml
defines two sensors that watch every code location of the deployment:
triage_run_failure_sensorsends each failed run, with each failed step, its asset keys and its root-cause exception. A step that failed on a blocking asset check is left to the check sensor.triage_asset_check_sensorsends each run's failed asset checks.
Each sensor sends a failure only when:
- the run was launched by one of the component's
run_origins, or the code location setsAUTO_SWE_TRIAGE_ALL_ORIGINStotrue; - Dagster will not retry the run (
dagster/will_retryis nottrue); - for a check, its severity is one of
check_severities.
Set AUTO_SWE_TRIAGE_ALL_ORIGINS on the ai_org code location in Dagster+, then
redeploy the code location. The sensors read the new value from their next tick.
A branch deployment gets no triage sensors.
| Attribute | Default | Meaning |
|---|---|---|
run_origins | [schedule, automation_condition] | The launchers whose failed runs are triaged: schedule, automation_condition, sensor, backfill, manual, or all |
check_severities | [ERROR] | The check severities triaged; add WARN to triage warnings too |
minimum_interval_seconds | 600 | How often each sensor checks for new failures, in seconds. A tick of the run failure sensor reads at most 5 failed runs, which is Dagster's default limit for a run status sensor. Failed runs that the sensor does not triage count toward that limit. A tick of the check sensor reads at most 100 check evaluations |
dry_run | true | Grade and report what would be filed, and write nothing to Linear |
runs_per_hour, runs_per_day | 12, 60 | The run budget's run limits; see below |
usd_per_hour, usd_per_day | 20, 100 | The run budget's dollar limits; see below |
The triage workflow
Each sensor starts the deployment's DagsterTriage workflow, dagster-triage/prod, if
none is running, and signals it the failure as a dagster.failure work item: one failed
run, or one run's failed checks. The workflow is defined in
auto_swe/triage/workflow.py, and it runs on the session queue. It is not a session. It
takes only dagster.failure work items, and runs the triage agent's diagnose step on
each one. The triage agent is off by default, so a session that receives a
dagster.failure work item runs no agent on it.
For each failure, the workflow:
- checks the run budget's dollar limits;
- runs the prematch, which may record the failure without a step run;
- checks the run budget;
- loads the failure's context with
dagster_failure_context; - runs the step that the
triageagent's matching trigger names,diagnose, with theStepRunnerthe sessions use. It runs that one step, and does not follow the step'son; - schedules the failure to run again when the agent marked it blocked; see Retries of a blocked failure.
It handles one failure at a time, so each diagnosis sees the issues that earlier diagnoses filed. The workflow:
- ignores a failure it has seen;
- ends after 24 hours with no failure;
- continues as new when its history grows long.
Its status query lists its counters, lifecycle events, recent step runs, the failure
it is handling and its scheduled retries.
The run budget
TriageConfig.run_budget limits the step runs the workflow starts, and their cost, in
a rolling hour and a rolling day. Its limits, and how a run's cost counts toward them,
are in The run budget.
The workflow keeps the runs it counts in its own state, and passes them on when it
continues as new. Without an end_triage signal, it ends only after 24 hours with no
failure. By then, every run it counted is more than a day old, so the next workflow
has no earlier run to count. A workflow that an end_triage signal ends sooner does
not pass its runs on.
A failure that arrives over any limit runs no agent and loads no context. For that failure, the workflow:
- emits an
over_budgetlifecycle event; - counts the failure in
counters.over_budget; - runs
dagster_triage_overflow, which records the failure on the issue for its code location and the day, in UTC, as Failures that triage cannot attribute describes.
The dollar limits count the workflow's model requests outside its step runs as well as the step runs. These requests are:
- the prematch's questions;
- the context's classification of the cause;
- the grade of each diagnosis.
Each activity reports the tokens of its requests by model, and the workflow prices
them at model_prices.json's prices when it records them. The prematch is not a step run, so only the dollar limits apply to it.
Retries of a blocked failure
The agent marks a group blocked when one of its tools fails, or its access to
Nessie, Linear or GitHub fails. A work item has one member, so it has one group, and
that group is the whole work item. For a blocked group, dagster_submit_diagnosis
files nothing and returns the step's blocked outcome, with the agent's text as its
reason. The step's on maps blocked to end, as the loader requires of every
outcome a step's tool returns.
DagsterTriageWorkflow reads the outcome of the step run it started. When the outcome
is blocked and the work item has had fewer retries than
TriageConfig.blocked_retries, the workflow:
- counts the retry in
counters.blocked_retries; - emits a
blocked_retrylifecycle event, whose detail names the retry, when it is due and the reason; - logs a warning;
- starts a workflow timer of
TriageConfig.blocked_retry_delay.
When the timer fires, the workflow puts the work item, with only the failures the prematch did not record, at the end of its queue. The queue ignores a work item it has received, so the workflow adds the retried work item to the queue directly. The retry then goes through every step of the list above, starting with the check of the dollar limits. So the prematch can record the retried failures on an issue that was filed after the first step run.
When the agent marks the work item blocked in the step run of its last retry,
dagster_triage_blocked records its failures, with the agent's reason, as
Failures that triage cannot attribute
describes. A workflow that ends while a retry is scheduled cancels the retry's timer
and records the work item the same way. A workflow that continues as new passes its
scheduled retries on to the next run, which starts a timer for each of them. It also
passes on the retry count of each retried work item that is in its queue.
| Setting | Default | Meaning |
|---|---|---|
blocked_retry_delay | 1 hour | How long after a blocked step run the workflow runs the step again |
blocked_retries | 2 | How many times the workflow runs the step again for a work item that the agent marked blocked |
The defaults assume that the agent is blocked by a service that it or its tools read, such as Nessie, Linear, GitHub or Dagster+, and that the service is down or refuses its requests. The delay gives such an outage an hour to end before each retry. A failure that is still blocked after the last retry is filed about two hours after its first step run. Two retries cost at most two more step runs for a work item, and each of them counts toward the run budget. The triage sensors send neither setting, so changing them is a change to the workflow's code.
The prematch
After the check of the dollar limits, dagster_triage_prematch looks for the issue
whose cause explains each distinct error of the work item's failures. The failures of
an error with such an issue are recorded on it with no step run. A failure recorded
this way may be:
- the same exception on the same asset in another partition;
- a check that fails again;
- the same cause failing an asset the issue does not list.
The agent gets the work item with only the failures the prematch did not record. When the prematch recorded them all, no step runs.
The prematch groups the failures by kind, check, exception class and message. Each
group is one distinct error, and the prematch handles each group on its own. A group
may be failed steps, failed checks, or a run that failed with no step failure. It
handles at most TriageConfig.prematch_max_errors groups, 10 by default, because each
search counts toward Linear's limit of 30 searches a minute. The failures of any later
group go to the agent.
-
Linear's semantic search finds the issues for the error. The query is the one-line description of the group's first failure, as the triage prompt lists it. The search returns at most
TriageConfig.prematch_candidates, 5 by default, of the issues with a cause attachment that are:- open;
- canceled within
reopen_within_days, 14 days; - completed within
reopen_within_daysin the team thatdagster_submit_diagnosisreopens issues in.
The prematch replaces a duplicate with the issue it duplicates. It keeps that issue only when the issue is open or was closed within
reopen_within_days, and whendagster_submit_diagnosiswould record a failure on it. When Linear refuses the search because it is over its limit of 30 searches a minute, the prematch searches again after 5, 10, 20 and 30 seconds. When Linear still refuses it, refuses it for another reason, or finds no issue, the group goes to the agent. -
Jev answers two questions about the issues in one request. One is its probability that one of the issues has the failures' root cause. The other, asked when there is more than one issue, is which issue has it, with a probability for each.
-
The group's failures are recorded on the issue Jev chose when:
- the first probability reaches
TriageConfig.prematch_threshold, 0.7 by default; - Jev's probability of the issue it chose reaches
TriageConfig.prematch_choice_threshold, 0.7 by default.
Otherwise the group goes to the agent. When the request to Jev fails, only that group goes to the agent.
- the first probability reaches
Linear's search reads an issue's title and description, not its attachments, so it matches a failure to the Cause section and the table of failures the issue was filed with.
For each issue, Jev reads:
- the issue's title and Cause section;
- the errors recorded on the issue for the group's symptoms, or, for an issue that holds none of those symptoms, the errors recorded on its other symptoms.
Each option of the question about which issue has the cause is one of these issues, with its title, Cause section and errors.
Jev also reads the group's failures. For failed steps, it reads the errors of the group's steps in the run's log, with their innermost frames. For a failed run, it reads the errors of the run as a whole. The activity reads the run's log only when a group of failed steps or a failed run has issues to ask about.
The activity records the failures as a filing records failures on the issue for their cause. It writes the symptom and run attachments, and a comment when an asset or check the issue did not list has failed. It reopens a Done issue, with a comment that the cause failed again. Errors recorded on the same issue share one set of writes. When Linear refuses an issue's writes, the failures of the errors recorded on it go to the agent.
The activity has the context step's timeout, 10 minutes, because it reads the same run
log. When it fails, every failure continues to the agent. The activity reads no
repository settings, so the reopen window and team are the defaults of
TriageSettings.
The worker logs one line per prematch. The line names the work item and its symptoms.
It then says, for each distinct error, how the prematch routed its failures, with each
issue Jev was asked about and Jev's probability of it, or why Jev was not asked. When
Jev was asked about any error, the workflow also emits a prematch lifecycle event
with the same routing. It counts the distinct errors:
- that the prematch recorded, in
counters.prematched; - whose probability that an issue has their root cause was below
prematch_threshold, incounters.prematch_refused; - whose probability reached
prematch_thresholdwhile Jev's probability of the issue it chose was belowprematch_choice_threshold, incounters.prematch_ambiguous; - whose search Linear refused, in
counters.prematch_search_refused; - for which the request to Jev failed, in
counters.prematch_unanswered; - whose writes Linear refused, in
counters.prematch_not_written.
The prematch runs for every failure. The workflow's own config, TriageConfig, sets:
prematch_threshold;prematch_choice_threshold;prematch_candidates;prematch_max_errors;prematch_model, the Jev model the prematch asks, which needs a price inmodel_prices.json.
The triage sensors send none of them, so changing them is a change to the workflow's code. In a dry run, the prematch asks Jev and writes nothing.
What it reads
Before the agent starts, the worker reads, for the failed run:
- the run's step failures, with the deepest exception and its innermost frames outside Dagster's own code;
- the issues each failure's symptom was attributed to before;
- the open triage issues, those created in the last day, and those closed in the last 30 days, each with its cause key.
Jev then rates how likely each category of cause below is, from the failures and the run's errors, in one Choice question. The agent starts from the categories Jev rates highest, so it needs fewer turns to reach the cause.
The agent then investigates in a sandbox with the repository checked out at the run's commit. It works through a fixed list of hypotheses, from a recent deploy to an out-of-memory task, and records the evidence that confirms or rules out each.
What the sandbox reaches
The credentials and sandbox.egress of agents/triage/agent.yaml give the sandbox
access to:
- Nessie, read-only, which serves the catalog and each table's metadata;
- Linear, read-only;
- GitHub, read-only, for the repository's commits;
- GitHub's release downloads, for the setup script;
- PyPI, for
uv run.
By design, it cannot reach:
- Dagster+, except through the agent's tools. The sandbox has no Dagster+ token.
- The third-party sources the assets fetch from. The egress proxy refuses them with
CONNECT tunnel failed, response 403. - AWS APIs, such as the ECS API. The sandbox has no AWS credentials.
- Object storage. The proxy refuses it with 403, so a table scan fails. The table's metadata still comes from Nessie.
The instructions tell the agent to diagnose from what its tools, the checkout, Nessie
and Linear show, and to let Jev grade that evidence. The agent marks a group blocked
only when one of its tools fails, or its access to Nessie, Linear or GitHub fails. It
never marks a group blocked because one of the limits above stopped a check.
The repository's CLAUDE.md tells agents to run dg commands through the dagster-cli
skill, whose commands read Dagster+ with a Dagster+ token. The instructions tell the
triage agent that this does not apply to it.
The agent's repo_context.skills.only names the only skills it gets. The agent does
not get a skill added to the repository later. It gets these skills:
content-placementandwriting-style, from.claude/skills/, for the text of its diagnosis;dagster-expert, from thedagsterplugin, for how Dagster's assets, sensors, automation conditions and components work. Its pages ondg apido not apply, since the instructions forbiddg;typesafe-ai, from thetypesafeplugin, for the code that asks Jev.
The plugins are the ones .claude/settings.json enables.
The Dagster+ tools
The agent reads Dagster+ with the tools below, which the worker runs with the read token against the failure's deployment:
| Tool | Returns |
|---|---|
dagster_run_events | A run's step and run failures, with their exceptions and frames, and each asset check evaluation it recorded, failed ones first, with the evaluation's metadata. step_key limits it to one step |
dagster_step_logs | One step's log, oldest first: each line the step's code logged through context.log and each event Dagster recorded for the step, with its level and event type. It lists WARNING and above unless level names a lower level |
dagster_check_history | An asset check's latest executions, newest first: partition, status, run, description and metadata. partition limits it to one partition, and limit, at most 30, sets how many |
dagster_step_logs reads the run's event log, which dagster_run_events also reads.
It lists the warnings a step logged before it failed, such as a source's redirects. A
line that a Python logger writes without a Dagster handler is in the step's captured
stderr only. No tool reads the captured stderr.
Each tool truncates a metadata value or a log message to 1,500 characters and its
whole result to 30,000. The tools act only on dagster.failure work items, so only
the triage agent can name them.
How a diagnosis is graded
The agent submits its diagnosis of the run with one call of
dagster_submit_diagnosis, as one group. It gives a
cause key, <category>:<subject>, the cause, its evidence and what it ruled out.
The categories follow what a failure's error shows, because Jev classifies the failure
before anyone investigates it. An error does not tell a table in a bad state from a
mistake in the code that reads it, so both are data-contract, and the subject names
the side that has to change. code-defect is left to logic or configuration that is
wrong whatever the data.
| Category | Subject |
|---|---|
external-source | The key of the asset that reads the third-party source |
credentials | The key of the asset whose source refused the credential, or the name of the service that refused it |
data-contract | The key of the asset that has to change: the upstream whose output no longer fits its readers, or the reader that reads it wrongly |
code-defect | The dotted module path of the defect: logic or configuration that is wrong whatever the data |
data-validation | The key of the asset whose data broke a rule and was refused |
platform | The name of the service that failed, in lowercase |
resources | The code location whose runs lacked memory, time or the compute to start |
Jev, TypeSafe's System One model, grades each group in one request:
- Conviction, a Score on five levels: speculation, pattern match, supported by context, validated by system state, validated with alternatives ruled out.
- Contradicted: whether any evidence contradicts the cause.
- Alternative: whether another cause, not ruled out, fits as well.
- Same cause: when the agent matched an existing issue, whether it has the same root cause and not only a similar symptom.
Code, not the model, decides the outcome. A group is identified when its conviction is
at least min_conviction (supported by context), contradiction is at most
max_contradicted, and the alternative is at most max_alternative. Any other group is
inconclusive.
Where a group goes
| The group | Goes to |
|---|---|
Identified, and Jev finds the matched issue has the same cause (same_cause) | The matched issue |
| Identified, and an issue already holds its cause key | That issue |
| Identified, and that issue is open | Recorded on it. A comment is posted only when the cause fails an asset or check the issue did not list |
| Identified, and that issue was canceled | Recorded on it, without reopening |
Identified, and that issue was Done within reopen_within_days | Reopened to Triage, with a comment |
| Identified, and that issue was Done earlier | A new issue, related to it |
| Identified, with no issue for its cause | A new issue, related to any issue Jev found related but not the same |
| Inconclusive | The issue for each of its unattributed: keys; see Failures that triage cannot attribute |
| Blocked, because one of the agent's tools or its own access failed | Not filed. The workflow runs the step again. After the last retry, the workflow files the group's failures on the issue for each of their unattributed: keys. See Retries of a blocked failure |
A new issue goes to the team's Triage state, unassigned. Its priority is the one the agent gave the group. A group without one gets:
- High when a step or run failed;
- Medium when an ERROR check failed;
- Low otherwise.
Its labels are labels, the category and the code locations.
Its description has these sections:
| Section | Holds |
|---|---|
| Cause | The cause, with the agent's optional data sample as a table and its optional Mermaid diagram |
| Fix | What has to change, and where. Left out when the agent gave no fix |
| Next step | One action that confirms the problem or the fix, with its command |
| Impact | The failures the issue was filed for: a table of the failed assets, steps and runs, and a table of the failed checks, each with its code location, run and partition. A table with no rows is left out |
| Evidence | Jev's grade and each item of evidence, with path:line@commit sources linked to GitHub. The section is collapsed |
| Ruled out | Each hypothesis and the evidence against it. The section is collapsed |
The triage tools write the description once, when they create the issue, and never change it; people may edit it. Every later failure of the cause goes to the issue's attachments, which Linear keys by URL, so recording a failure again updates the same attachment:
- the cause attachment, whose metadata records the cause key and when a failure of the cause was first and last recorded. Its subtitle shows how long ago the last failure was recorded;
- one attachment per symptom. Its title is the name of each asset and the check or
error, followed by the rest of the asset key. Linear cuts a long title at its end, so
the names come first, and the rest of the key keeps the titles of one issue's
symptoms distinct. Its subtitle shows the code location, its count of failures and
how long ago the last one was recorded. Its metadata records the failure and its last
10 runs, at most one failure per run. The metadata also holds the
attributesandmessagesthat Linear shows when a person opens the attachment:- the asset, check, severity, error, counts and partitions;
- one message per run, with an excerpt of its error and its link;
- one attachment per failed Dagster run, at the run's URL, with no limit. Linear's
attachmentsForURLfinds a run's issue from that URL.
The times in these attachments are when the triage tools recorded each failure, which follows the failure itself by however long the triage took.
Failures that triage cannot attribute
Triage cannot attribute the failures of a group when:
- Jev's grade of its diagnosis is inconclusive;
- the agent marked it
blockedin the step run of its last retry; - it arrived over the run budget, so no agent ran.
Each failure of such a group goes to the issue for its unattributed: key. In an
inconclusive or blocked group, the subject of a failure's key names what failed and does
not depend on the error, so the next failure of the same asset, check, step or run gets
the same key. In a group that arrived over the run budget, every failure gets one key
for its code location and the day, in UTC, whatever failed. The failures of a code
location that arrive over the budget on one day therefore go to one issue, and not to
one issue for each asset. The keys are:
| The failure | Its key |
|---|---|
| A step that wrote assets | unattributed:<asset key>, with the keys joined by commas when the step wrote more than one |
| A check | unattributed:<asset key>:<check name> |
| A step that wrote no asset | unattributed:<code location>/<step key> |
| A run that failed with no failed step | unattributed:<asset keys>, the run's planned assets sorted and joined by commas. A run with no planned asset, or one whose events Dagster+ cannot return, gets unattributed:<code location>/<job name> |
| Any failure that arrived over the run budget | unattributed:overflow/<code location>/<YYYY-MM-DD> |
The planned assets of a run come from its AssetMaterializationPlannedEvents, which
the triage tools read from Dagster+ when they file the failure. A run's job is almost
always __ASSET_JOB, so a key from the job name would put every run of a code location
that failed with no failed step on one issue.
A subject longer than 100 characters is cut to its first 91 characters, followed by ~
and the first 8 hex digits of the SHA-256 of the whole subject. The cut keeps the
attachment URL short, and the hash keeps two long subjects with the same first 91
characters apart. A group whose failures have several keys goes to several issues.
The issue for a key is the one with the attachment ?unattributed=<subject>. Triage
finds it the way it finds the issue for a cause key. It takes the oldest open issue with
the attachment, or else the one closed last. Then:
- an open issue is recorded on, with a comment;
- an issue in the team that was closed, Done or Canceled, within
reopen_within_daysis reopened to Triage, with a comment; - any other closed issue gets a new issue related to it.
A canceled issue for a cause is recorded on without reopening. A Canceled issue for an
unattributed: key is reopened, because canceling it does not say why its failures
happen. The comment holds:
- why triage could not attribute the failures;
- the failures;
- the agent's proposed cause, proposed fix and evidence, when the agent gave them.
Several keys of a group can have one issue, such as after a person marks one key's issue a duplicate of another's. The group's failures with those keys then go to that issue together, with one comment and at most one reopening.
A new issue goes to the team's Triage state. Its priority is the one a group gets when
the agent gave it none. Its labels are labels, unattributed_label and the code
locations. Its title is Failures of <subject> that triage could not attribute. For
failures that arrived over the run budget, its title is
Failures in <code location> on <day> that arrived over the triage run budget. A
subject that would make the first title longer than 120 characters is cut and ends with
an ellipsis. 120 characters is also the longest title the agent may give an issue for a
cause. The description has these sections:
| Section | Holds |
|---|---|
| Why triage could not attribute it | Jev's reasons for an inconclusive grade, the agent's blocked text, or the run budget's limit that was spent |
| Proposed cause | The agent's cause key and cause. Left out when the agent gave no diagnosis |
| Proposed fix | The agent's fix. Left out when it gave none |
| Next step | For an inconclusive group, an instruction to confirm or rule out the proposed cause, followed by the agent's next step. Otherwise, an instruction to investigate the failures and to rerun the failed run if its error looks transient |
| Impact | The failures, as on an issue for a cause |
| Evidence and Ruled out | As on an issue for a cause. Left out when the agent gave no diagnosis |
The issue has no Cause section and no cause attachment, because it holds no cause.
unattributed is a prefix of the key and not a CauseCategory. The agent picks its
category from CauseCategory. The tool's arguments and the prompt's Subjects section
allow only its members, so the agent cannot give an unattributed: key. The URL of
the attachment for the issue's key starts with ?unattributed= and not with ?cause=,
so every query of the issues with a cause attachment leaves the issue out:
- the prematch never asks about one, so the agent diagnoses the failure again;
- the prompt's Issues list leaves it out;
- the prompt's Symptom history names it as an issue for failures that triage could not
attribute. The agent can find its identifier there, and a
matchthe agent gives to one is ignored.
When a later diagnosis of a failure with the same unattributed: key identifies its
cause, the group goes to the issue for that cause key, as
Where a group goes describes. The issue for the unattributed:
key keeps the failures recorded on it, and nothing merges the two issues. A person who
finds both marks the issue for the unattributed: key a duplicate of the cause's issue,
or closes it. For a duplicate, Linear moves its ?unattributed= attachment to the
cause's issue, so later failures with that key are recorded there, with a comment. The
cause's issue keeps its one cause attachment, so it stays the issue for its cause.
Correcting it
- Mark a duplicate. Linear moves the duplicate's attachments to the issue you choose, so the next failure with that cause or symptom is recorded there.
- Write
.auto-swe/triage.mdonmainfor standing guidance, such as known noise or the subject to use for a source. The agent reads it with its instructions. - Override a setting in
.auto-swe/config.yamlonmain, undertriage.
| Setting | Default | Meaning |
|---|---|---|
linear_team | DPF | The key of the team new issues go to |
labels | [auto-triage] | Labels on every new issue |
reopen_within_days | 14 | How long after a Done issue, or a Canceled issue for an unattributed: key, closes a recurrence reopens it |
unattributed_label | needs-investigation | The label on every new issue for failures that triage could not attribute, besides labels |
grading | see agents/triage/agent.yaml; the model defaults to JEV_MODEL | Jev's model and the thresholds above |
dagster_triage_overflow, which records a work item over the run budget, does not read
these settings, so an issue it creates goes to DPF with the default labels.
Evaluating it
workflows/auto_swe/evals/triage/cases.yaml is a pydantic-evals dataset. Each case is
a set of failed production runs whose causes a merged fix confirmed.
python -m evals.triage.run starts a Temporal dev server and runs the session worker in
the eval's own process. It sends each case's runs to a separate dry-run triage workflow
for that case. In a dry run the prematch still asks Jev, and a run it records on an open
issue reaches no agent. The eval scores the groups the agent filed, and counts each run
the prematch recorded as filed under the cause key of that issue:
Reached: causes whose failures were all diagnosed, not marked blocked;KeyMatches: causes filed under the confirmed cause key;FalseMerges: cause keys that failures of more than one cause were filed under, so that they share an issue;FalseSplits: causes whose failures were filed under more than one cause key, so that they are on more than one issue.
A false merge hides a second cause behind the first, so it costs more than a false split, which a person fixes by marking a duplicate.
The eval ends a case's workflow once the workflow has handled the case's runs. So a
run that the agent marked blocked gets no retry, and its cause does not count toward
Reached. The report also gives each completed case's estimated cost, token counts and prematched, the number of its runs the
prematch recorded. Its totals of estimated cost and of tokens by model include the cases
that failed or timed out. Run the eval before changing the instructions, the model or
the grading thresholds:
set -a; . ~/.config/auto-swe-poc/secrets.env; set +a
uv run --directory workflows/auto_swe python -m evals.triage.run
The process needs:
- the worker's credentials;
workflows/auto_swe/.env, for the payload directory;- a
dg plus login.
It runs the dev server with the temporal CLI on PATH. Without one on PATH, the
Temporal SDK downloads the CLI. Other options:
--case <name>: run only that case. Repeatable.--repeat <n>: run each casentimes. Each repeat of a case costs as much as a production triage of its runs.--max-concurrency <n>: run up toncases at once. The default is 1.--existing-worker: use the server thatTEMPORAL_ADDRESSnames and the worker polling it, asworkflows/auto_swe/scripts/README.mdstarts them.
The eval exits with status 1 when a case, an evaluator or a report evaluator fails.
workflows/auto_swe/evals/cause/cases.yaml is a pydantic-evals dataset of failed
production runs, one per kind of error. Each case has the cause category that a merged
fix, the run's own events or a ruling on that kind of failure established. Each case
records the text the triage context gives Jev for its run. The cases share no error
with the examples in CAUSE_CRITERIA. python -m evals.cause.run asks Jev to classify
each case twice in one request, with those examples and without them. For each of the
two questions, the report gives:
correct: whether Jev answered the expected category or an accepted one;p(right): Jev's probability for those categories;- the number of right answers and the mean
p(right), by expected category and over every case; - the requests' tokens by model, and their estimated cost.
The eval asks only TypeSafe, so it needs TYPESAFE_API_KEY and no Temporal server. Run
it before changing the categories, their examples or JEV_MODEL, and add a case when
triage misclassifies a new kind of failure:
uv run --directory workflows/auto_swe python -m evals.cause.run
--case <name>, --repeat <n> and --max-concurrency <n> work as they do for the
triage eval. By default, every case runs at once. The eval exits with status 1 under the
same conditions as the triage eval. python -m evals.cause.capture reads production
Dagster+ and records the text of each case that has no recorded text. With --all, it
records the text again for every case.
workflows/auto_swe/evals/search/cases.yaml holds one case for each symptom record on
the issues for a diagnosed cause. A case is the failure the record describes, the
time its latest run was recorded, and the issue that holds the record. python -m evals.search.run runs
the prematch's semantic search for each failure as of that time, with the prematch's
own query, filter and number of candidates. For each case, the report gives:
found: whether the issue that holds the record is among the candidates, with its rank, or the candidates when it is not;first: whether that issue is the nearest candidate.
It then counts both separately for two sets of failures:
- the failures each issue was filed from, which its description was written from;
- the failures recorded on the issue later.
Linear's search reads an issue's title and description, not its symptom records. The issue that holds a failure's record can therefore rank below other issues.
The eval reads Linear with LINEAR_API_KEY. Each case makes one semantic search, and
Linear allows 30 a minute, so --max-concurrency is 4 by default. --candidates <n>
sets the number of candidates the search returns. Run it before changing the
prematch's query or filter, and as issues accumulate, since more issues compete for
the same candidates:
LINEAR_API_KEY=... uv run --directory workflows/auto_swe python -m evals.search.run
python -m evals.search.capture writes cases.yaml again from Linear's current
issues. A symptom record without a kind field, written before symptom records had
one, has no case.
Turning it on
- Fill these secrets; see Secrets:
auto-swe/linear/write-oauth-client. The agent files its issues as the Linear write app.auto-swe/typesafe/api-keyauto-swe/dagster/read-token
- The secret
dagster/temporal/api-keyholds the API key of a Temporal Cloud service account. In Temporal Cloud, give that service account write access to theauto-swe.obsiinamespace.projects/ai_org/container_context.yamlsetsAUTO_SWE_TEMPORAL_API_KEY_SECRET_IDtodagster/temporal/api-key. - Watch the workflow in dry run. Each step run's
dagster_submit_diagnosisresult says what it would have filed, and the workflow's history in the Temporal UI shows that result.temporal workflow query --namespace auto-swe.obsii --workflow-id dagster-triage/prod --name statuslists the step runs. - Set
dry_run: falsein the component'sdefs.yaml. The sensors send their settings with each failure, so the running workflow writes to Linear from the next failure on.
The triage workflow runs the triage agent by name. The agent is off by
default, and a session runs no agent on a dagster.failure work item, so no session
runs it.