ECS Task Usage
When a step that runs in its own ECS task finishes, a success hook or a failure hook reads the
memory and vCPU that the task requested and used. It reads them from inside the task, through the
ECS task metadata endpoint v4. It then passes a StepTaskUsage record to an emitter.
with_task_usage_hooks(definitions, emit) in ai_dagster.task_usage attaches the hooks to
every executable assets definition and asset check definition. The emitter is the function
passed as emit. It decides where the records go.
Each record describes one step. A step is the unit that runs in one ECS task. The ecs/memory
and ecs/cpu tags of the step's definition set that task's size.
- The record of a
multi_assetstep lists all the step's assets. - The record of an asset check step lists the step's checks.
The record
| Field | Source | Meaning |
|---|---|---|
deployment, code_location | DAGSTER_CLOUD_DEPLOYMENT_NAME, DAGSTER_CLOUD_LOCATION_NAME | Where the step ran. A branch deployment's name is its hash |
run_id, job_name, step_key | Dagster | The run and step that ran in the task |
succeeded | Which hook ran | |
asset_keys, check_keys | The definition the hooks are attached to | Every asset and check of the step's definition, whether or not the run selected it |
partition_key | The run's dagster/partition tag | |
step_finished_at | The hook's start | |
memory_rss_peak_mib | /task/stats → memory_stats.stats.total_active_anon plus total_inactive_anon, summed over containers and sampled every 2 seconds from when the step's definitions load | The largest anonymous memory, shared memory included, of every process in the task. It leaves out page cache. It misses a peak shorter than 2 seconds and the time of a failed request. Records written before the field existed, or whose stats the sampler could not parse, have none |
usage.task_arn | /task → TaskARN | The step's task |
usage.memory_requested_mib | /task → Limits.Memory | The task's memory, with the definition's ecs/memory tag or the executor's memory config applied |
usage.memory_peak_mib | /task/stats → memory_stats.max_usage, summed over containers | The peak memory usage that the kernel recorded for the task's containers, page cache included |
usage.vcpu_requested | /task → Limits.CPU | The task's vCPU |
usage.cpu_seconds | /task/stats → cpu_stats.cpu_usage.total_usage, summed over containers | CPU time used since the task's first container started |
usage.elapsed_seconds | container StartedAt to sampled_at | |
usage.sampled_at | /task/stats → read | When the ECS container agent took the stats sample |
TaskUsage also derives these properties:
memory_utilizationis peak ÷ requested memory.vcpu_averageiscpu_seconds÷elapsed_seconds.cpu_utilizationis average ÷ requested vCPU.
A memory_utilization near 1.0 means the task's peak, page cache included, came close to its
memory limit. The kernel reclaims page cache before it kills the task, so a step that reads or
writes large files can reach 1.0 without running out of memory.
Where the records go
Each project's definitions.py applies with_task_usage_logging() from
ai_dagster.task_usage_log. Its emitter does the following for each record, in this order:
-
It logs a warning in the Dagster run in each of these cases:
- The task's peak memory usage was above 90% of its memory.
- The step succeeded and the task used under 30% of its memory, and Fargate allows a smaller memory size for the task's vCPU that is at least 1.5 times the peak. The warning names the smallest such size.
- The step succeeded and the task used under 30% of its vCPU, and Fargate allows a smaller vCPU size with a memory size at least 1.5 times the peak.
A failed step may have stopped before its largest allocation, so only the first case applies to it.
-
When it logs no warning, it logs the task's peak memory and average vCPU, each with its share of the requested amount, at debug level.
-
It writes the record as one JSON object to the CloudWatch log group
/ppm/ecs-task-usage, in the log stream<code_location>. The object also has"event": "ecs_task_usage", and itsusageobject also has the properties thatTaskUsagederives. The group's retention period is 365 days. The group holds the records of every deployment, branch deployments included.
deployments/aws/cloudformation/ecs-agent-vpc-private.yaml creates the group as
EcsTaskUsageLogGroup. The template's ecs-task-usage-logs policy lets
UserCodeExecutionRole write to the group.
The emitter calls the CloudWatch Logs API rather than writing to the task's stdout. The log driver often drops the last seconds of a step task's stdout, and the hook runs in those last seconds.
When a PutLogEvents request fails because the code location's stream does not exist, the
emitter creates the stream and sends the request again.
A failed CloudWatch write fails the hook and leaves the step's result unchanged. The Dagster
run still has the warnings or the debug line. CloudWatch rejects an event more than 14 days
old or more than 2 hours in the future without failing the PutLogEvents request. The emitter
fails the hook when CloudWatch rejects the event.
To list one task's records in CloudWatch Logs Insights, filter on the task's ARN:
filter usage.task_arn like /<task ID>/.
This CloudWatch Logs Insights query lists the prod records, lowest memory utilization first:
fields @timestamp, deployment, code_location, step_key, asset_keys.0,
usage.memory_peak_mib, usage.memory_requested_mib, usage.memory_utilization,
usage.vcpu_requested, usage.cpu_utilization
| filter deployment = "prod"
| sort usage.memory_utilization asc
Weekly right-sizing
ecs_right_sizing_job in the ai_org code location runs on ecs_right_sizing_schedule, on
Mondays at 06:00 UTC. On branch deployments, the schedule's default status is stopped.
The job produces right-sizings. A right-sizing is a RightSizing record that recommends a
different task size for one step:
- An upsize gives more memory to a step whose RSS came close to its memory limit.
- A downsize gives less memory to a step that uses little of its memory, or gives the next smaller Fargate vCPU size to a step that uses little of its vCPU. A step that uses little of both gets both.
A step that runs out of memory writes no record, and its failure reaches triage. Triage raises the step's memory and runs the step again, and the successful run's record shows the memory the step needs. The job reads only the records of successful runs, because a failed step may have stopped before its largest allocation. An upsize gives a step more memory before it runs out. The job:
-
Runs a Logs Insights query over the last 35 days of
prodrecords of successful steps in/ppm/ecs-task-usage, with one row per step and task size. The window is longer than the longest month, so it includes the peak of a step whose input is largest on one day of each month. -
Takes the step's current size from the task size in the step's latest record.
-
Recommends upsizing a step whose largest
memory_rss_peak_mibat its current size is at least 90% of its current memory, when the window has at least 5 of the step's records with an RSS at that size. Records at an earlier size do not count. The RSS leaves out page cache. The kernel reclaims page cache before it kills a process for running out of memory, so a step that reads or writes large files gets no upsize for its page cache. The recommended size is the smallest Fargate size with at least the step's vCPU and at least 1.5 times that RSS. It keeps the step's vCPU unless that vCPU allows no memory size that large. When no Fargate size is that large, the job recommends Fargate's largest size. The job recommends no upsize for a step already at Fargate's largest size, or whose records at its current size have no RSS. -
Recommends less memory for a step whose largest peak at any size in the window is under 50% of its current memory, when the window has at least 5 of the step's records. The peak is
usage.memory_peak_mib, which includes page cache. -
Recommends the next smaller Fargate vCPU size for a step when all of these hold:
- the step's average CPU utilization at its current vCPU is under 25%
- the window has at least 5 of the step's records at that vCPU
- the step's largest RSS at its current size is under 90% of its memory
A step whose records at its current size have no RSS gets no vCPU downsize. A step given fewer vCPU can run longer.
-
Chooses the size of a downsize. When the job recommends less memory for the step, the recommended memory is the smallest Fargate memory size that is at least 1.5 times the largest peak, or at least the step's current memory when that is less. Otherwise, it is the smallest Fargate memory size that is at least the step's current memory. The job looks for that memory size at the step's current vCPU. When the job recommends fewer vCPU for the step, it first looks at the next smaller Fargate vCPU size. A downsize never gives a step more memory, and it must make the vCPU or the memory smaller.
-
Records these values in the op's output metadata:
- the number of steps
- the number of right-sizings
- a table of the right-sizings
-
Starts auto_swe's
EcsRightSizingworkflow with the right-sizings, when there are any and the job runs outside a branch deployment.
The job produces no right-sizing for any other step.
The attributes of EcsRightSizingComponent in
projects/ai_org/src/ai_org/defs/platform/ecs_right_sizing/defs.yaml set the deployment, the
window, the schedule and the memory and CPU thresholds. low_memory_utilization must be below
high_memory_utilization.
A right-sizing includes:
- whether it is an upsize or a downsize
- the step's code location, step key, asset keys and check keys
- the step's largest peak and its share of the current memory
- the largest RSS in the step's records, when the records have it
- the step's current and recommended vCPU and memory
- the step's average CPU utilization at its current vCPU
To apply a right-sizing, give the step's definition the recommended vCPU and memory. A
component's ecs_vcpu and ecs_memory_gb attributes set the size of every step the component
defines. Lower them only when each of those steps can run at the smaller size.
The right-sizing issue
The EcsRightSizing workflow in workflows/auto_swe files the right-sizings of one job run as
one Linear issue in the triage team's Triage state. The issue has the label ecs-right-sizing.
It has medium priority when it has an upsize, and low priority otherwise. It has:
- an "Upsizes" section and a "Downsizes" section. The workflow leaves out a section that has no right-sizings. Each section has a table of its right-sizings and each step's asset and check keys.
- an attachment that links to the job's Dagster run, when the job has the run's URL
The job starts the workflow on the auto-swe task queue, in the Temporal namespace that
projects/ai_org/container_context.yaml configures with the AUTO_SWE_TEMPORAL_* environment
variables. The workflow ID is ecs-right-sizing/<run ID>, so each job run files at most one
issue. The workflow chooses the issue's ID before its activity runs. Linear refuses a second
issue with that ID, so a retried activity files no second issue. Without DAGSTER_CLOUD_URL,
the job has no URL for its run, and the issue has no link to the run.
workflows/auto_swe/tests/data/ecs_right_sizing_input.json is a sample of the workflow's
input. The tests of ai_org check that the job builds it, and the tests of auto_swe check
that the workflow accepts it.
Sampling
The /task/stats endpoint returns the latest stats sample that the ECS container agent took,
which can be older than the request. The hook polls it once a second until every container has
a sample taken after step_finished_at, for at most 20 polls.
The polls keep the step's task running. They do not delay the run. Dagster launches downstream
steps and finishes the run on the step's STEP_SUCCESS event, which comes before the hook.
If no poll returns a sample taken after step_finished_at, the record has the last sample
returned. A sampled_at earlier than step_finished_at means the usage leaves out the end of
the step.
What is not recorded
- A task killed for running out of memory. The step's process ends before any hook runs.
The step fails in Dagster, and ECS records
stoppedReason: OutOfMemoryErroron the task. - Peak CPU. The endpoint reports cumulative CPU time, so one reading gives the average.
- A step that ran in the run's own task. The task's usage covers the whole run, so the hook
records nothing when the task is the one in the run's
ecs/task_arntag. - A run without an
ecs/task_arntag. Only the ECS run launcher sets the tag. A non-isolated run executes in the code server's task, whose usage covers everything the server has done since it started. - An attempt that a retry policy retries. Dagster runs no success or failure hook for it, so only the final attempt's task is recorded.
- A run that belongs to a backfill. A backfill run, whether it covers one partition or a
range, has the
dagster/backfilltag. Its task size and usage can differ from a scheduled run's, so the hook records nothing for it. - A run outside ECS. Without
ECS_CONTAINER_METADATA_URI_V4the hook records nothing.
Docker reports memory_stats.max_usage only under cgroup v1. Without it, the hook fails with a
validation error. Dagster logs the error as a HOOK_ERRORED event and does not change the
step's result.