Skip to main content

Deploys from main

A CI run on main checks and tests every project, service and workflow application. It builds, pushes and deploys only the components affected since the last deployed commit. The detect-python-impacts job of ci.yml chooses them.

The last deployed commit​

The last deployed commit is the newest commit, other than the run's own, that is an ancestor of the run's commit and whose main CI run succeeded. scripts/last_deployed_commit.py chooses it from the 100 newest finished main runs.

The run passes the commit to pants --changed-since --changed-dependents=transitive, and scripts/pants_affected_components.py --main maps the changed targets to components.

A run that was cancelled is never the last deployed commit, so the next successful run also deploys its changes. A failed run is never the last deployed commit either, but it can have deployed part of its changes before it failed. A later commit, such as a revert, can undo those changes, so that nothing changed between the last deployed commit and HEAD. A run therefore deploys everything when a run between the last deployed commit and its own commit failed.

When a run deploys everything​

A run deploys every component when:

  • it finds no last deployed commit, or the GitHub API call fails;
  • it is a workflow_dispatch run;
  • the run of a commit between the last deployed commit and its own commit ended in failure or timed_out;
  • a file in DEPLOY_ALL_FILES, or under a directory in DEPLOY_ALL_PREFIXES, of scripts/last_deployed_commit.py changed since the last deployed commit.

Those sets hold three kinds of file:

  • The files that decide what a run deploys and how: .github/workflows/ci.yml, pants.ci.toml, scripts/deploy_serverless_worker.sh, scripts/last_deployed_commit.py and scripts/pants_affected_components.py. A fix to one of them must reach the components that earlier runs left out.
  • The root files that change an image or what Dagster+ registers: pants.toml, dg.toml, dagster_cloud.yaml, dagster_cloud_settings.yaml and .dockerignore. The deploy job applies dagster_cloud_settings.yaml, and it runs only when a code location deploys. scripts/tests/test_last_deployed_commit.py checks that every file of the root build_metadata target other than mypy.ini, pytest.ini and ruff.toml is in DEPLOY_ALL_FILES.
  • pants-plugins/. The shared_source_targets backend and the BUILD prelude decide which sources and requirements each resolve's PEX holds, and pants --changed-since maps a change to them to no component.

How the selectors differ on main​

On a pull request, a root build_metadata target or a lockfile of a package that every resolve consumes usually comes from scaffolding a component. The selectors then choose the components that appear beside it, and every component of a kind when none appears.

On main, the range spans every merge since the last deployed commit, and the root files that change an image already deployed everything. With --main, those targets select nothing, and each component is selected only through its own changed targets. A change to ruff.toml or mypy.ini alone deploys nothing. A dependency bump deploys the components whose own lockfile changed.

On main and on a pull request, a change under a component's tests/, evals/ or scripts/ directory deploys nothing, because every image is built from the component's src/ PEX.

pants --changed-since lists a changed file only when a Pants target owns it. A file that an image build reads, such as a Dockerfile or build.yaml, belongs in the component's deploy_metadata files target. A change to a file outside it deploys nothing.

Re-running a main run​

Re-run only the newest main run. A run of an older commit checks the finished runs of later commits first:

  • When one of them succeeded, the run deploys nothing, because that run already deployed the older commit's changes.
  • Otherwise the run fails, because deploying the older commit would replace a newer build.

The check runs in the detect-python-impacts job. "Re-run failed jobs" on an older run reuses that job's first result and skips the check.

A failed deploy fails the run​

A run counts as deployed when it succeeds, so each deploy step fails the run when its component did not deploy:

  • dg plus deploy finish waits for each code location to load in Dagster+, and fails on a load error.
  • The ECS roll waits for the deployment it started to complete. It fails when the circuit breaker rolls the deployment back, when another deployment replaces it, or after 20 minutes.
  • scripts/deploy_serverless_worker.sh waits for each Lambda function update.

The next run on main deploys everything, because the failed run lies between its last deployed commit and its own commit.

A run cancelled while it deploys can also leave part of its changes deployed. The next run deploys only what changed since the last deployed commit; to repair a cancelled deploy, run the CI workflow on main with workflow_dispatch.

Redeploying a component​

A component whose image depends on a base image, such as python:3.14-slim or an apt layer, is rebuilt only when its own code changes.

To redeploy every component, run the CI workflow on main with workflow_dispatch.