Deploys from main
A CI run on main checks and tests every project, service and workflow application. It
builds, pushes and deploys only the components affected since the last deployed commit.
The detect-python-impacts job of ci.yml chooses them.
The last deployed commit
The last deployed commit is the newest commit, other than the run's own, that is an
ancestor of the run's commit and whose main CI run succeeded.
scripts/last_deployed_commit.py chooses it from the 100 newest finished main runs.
The run passes the commit to pants --changed-since --changed-dependents=transitive,
and scripts/pants_affected_components.py --main maps the changed targets to components.
A run that was cancelled is never the last deployed commit, so the next successful run also deploys its changes. A failed run is never the last deployed commit either, but it can have deployed part of its changes before it failed. A later commit, such as a revert, can undo those changes, so that nothing changed between the last deployed commit and HEAD. A run therefore deploys everything when a run between the last deployed commit and its own commit failed.
When a run deploys everything
A run deploys every component when:
- it finds no last deployed commit, or the GitHub API call fails;
- it is a
workflow_dispatchrun; - the run of a commit between the last deployed commit and its own commit ended in
failureortimed_out; - a file in
DEPLOY_ALL_FILES, or under a directory inDEPLOY_ALL_PREFIXES, ofscripts/last_deployed_commit.pychanged since the last deployed commit.
Those sets hold three kinds of file:
- The files that decide what a run deploys and how:
.github/workflows/ci.yml,pants.ci.toml,scripts/deploy_serverless_worker.sh,scripts/last_deployed_commit.pyandscripts/pants_affected_components.py. A fix to one of them must reach the components that earlier runs left out. - The root files that change an image or what Dagster+ registers:
pants.toml,dg.toml,dagster_cloud.yaml,dagster_cloud_settings.yamland.dockerignore. Thedeployjob appliesdagster_cloud_settings.yaml, and it runs only when a code location deploys.scripts/tests/test_last_deployed_commit.pychecks that every file of the rootbuild_metadatatarget other thanmypy.ini,pytest.iniandruff.tomlis inDEPLOY_ALL_FILES. pants-plugins/. Theshared_source_targetsbackend and the BUILD prelude decide which sources and requirements each resolve's PEX holds, andpants --changed-sincemaps a change to them to no component.
How the selectors differ on main
On a pull request, a root build_metadata target or a lockfile of a package that every
resolve consumes usually comes from scaffolding a component. The selectors then choose
the components that appear beside it, and every component of a kind when none appears.
On main, the range spans every merge since the last deployed commit, and the root files
that change an image already deployed everything. With --main, those targets select
nothing, and each component is selected only through its own changed targets. A change
to ruff.toml or mypy.ini alone deploys nothing. A dependency bump deploys the
components whose own lockfile changed.
On main and on a pull request, a change under a component's tests/, evals/ or
scripts/ directory deploys nothing, because every image is built from the
component's src/ PEX.
pants --changed-since lists a changed file only when a Pants target owns it. A file that
an image build reads, such as a Dockerfile or build.yaml, belongs in the component's
deploy_metadata files target. A change to a file outside it deploys nothing.
Re-running a main run
Re-run only the newest main run. A run of an older commit checks the finished runs of later commits first:
- When one of them succeeded, the run deploys nothing, because that run already deployed the older commit's changes.
- Otherwise the run fails, because deploying the older commit would replace a newer build.
The check runs in the detect-python-impacts job. "Re-run failed jobs" on an older run
reuses that job's first result and skips the check.
A failed deploy fails the run
A run counts as deployed when it succeeds, so each deploy step fails the run when its component did not deploy:
dg plus deploy finishwaits for each code location to load in Dagster+, and fails on a load error.- The ECS roll waits for the deployment it started to complete. It fails when the circuit breaker rolls the deployment back, when another deployment replaces it, or after 20 minutes.
scripts/deploy_serverless_worker.shwaits for each Lambda function update.
The next run on main deploys everything, because the failed run lies between its last deployed commit and its own commit.
A run cancelled while it deploys can also leave part of its changes deployed. The next
run deploys only what changed since the last deployed commit; to repair a cancelled
deploy, run the CI workflow on main with workflow_dispatch.
Redeploying a component
A component whose image depends on a base image, such as python:3.14-slim or an apt
layer, is rebuilt only when its own code changes.
To redeploy every component, run the CI workflow on main with workflow_dispatch.