CI credentials
The credentials that CI uses to push images and deploy are secrets of two GitHub environments. GitHub gives an environment's secrets only to a job that references the environment, and only after the job passes the environment's deployment protection rules.
aws-prodholds the prod credentials. Onlymaincan deploy to it, so a workflow that someone pushes to another branch cannot read them, even when it referencesaws-prod.aws-branch-deployholds the branch-deploy credentials. Any branch can deploy to it, with no approval. Any workflow on any branch can therefore read them. Each of these credentials can do only what a branch deploy needs.
Never store a credential that can change production as a repository or organization secret. Every workflow on every branch can read those.
Environments
| Environment | Protection rule | Secret | What it can do |
|---|---|---|---|
aws-prod | Only main can deploy | AWS_AI_PROD_KEY, AWS_AI_PROD_SECRET | ECR, Lambda, ECS, Secrets Manager, and the resources that terraform.yml applies |
aws-prod | Only main can deploy | DAGSTER_AGENT_TOKEN | Deploy code locations to prod; list and delete branch deployments |
aws-branch-deploy | Any branch can deploy | AWS_BRANCH_DEPLOY_KEY, AWS_BRANCH_DEPLOY_SECRET | Push to the ai-<oem>-branch ECR repositories; read dagster/nessie/m2m-writer-token |
aws-branch-deploy | Any branch can deploy | DAGSTER_BRANCH_DEPLOY_TOKEN | Create, update and delete Dagster+ branch deployments |
The AWS key of aws-branch-deploy belongs to the IAM user
gha-ppm-data-platform-branch-deploy, defined in
deployments/aws/terraform/solutions/dagster-agent/ci_branch_deploy_iam.tf. Its access
key is created by hand, so the secret never enters the Terraform state.
DAGSTER_BRANCH_DEPLOY_TOKEN is the user token of the Dagster+ service user
github-ci-branch. The user has the Editor role on all branch deployments and no access
to prod. Editor is the lowest role that can register code locations. It can also read
and edit every Dagster+ environment variable scoped to branch deployments, so none of
those variables may hold a credential that can change production.
The Nessie writer role can create, write and delete every branch except main. It can
read main, so CI can create a branch deploy's Nessie branch from it.
A job that references aws-prod runs only in a run whose ref is main. These runs have
that ref:
- a push to
main; - a
workflow_dispatchstarted frommain; - a scheduled run;
- a
pull_request_targetrun of a pull request intomain.
A workflow_dispatch of terraform.yml started from another branch fails.
Jobs
| Workflow | Job | Environment |
|---|---|---|
ci.yml | build_image | aws-prod on main; aws-branch-deploy on a pull request with the branch-deploy label, in a run a person started; none otherwise |
ci.yml | build_service_image, deploy_workflow_lambda | aws-prod on main; none otherwise |
ci.yml | deploy | aws-prod on main; aws-branch-deploy on a pull request |
terraform.yml | every job | aws-prod |
stale-branch-deployment-cleanup.yml | cleanup | aws-prod |
branch-deployment-cleanup.yml | teardown-deployment | aws-branch-deploy |
branch-deployment-cleanup.yml | cleanup | aws-prod for a pull request merged into main, whose Nessie branch the job merges into main; aws-branch-deploy for a pull request that was closed without merging or lost the label, and the job deletes the Nessie branch |
stale-branch-deployment-cleanup.yml has only a schedule trigger. GitHub runs it from
the latest commit on main. Nobody can start a new run by hand. Keep the schedule as its
only trigger, so that no person can start a run that deletes branch deployments and Nessie
branches with the aws-prod credentials.
A pull request's run builds service images but does not push them, because a branch deployment runs no service.
The cleanup job of branch-deployment-cleanup.yml does not run for a pull request
merged into a branch other than main. That pull request's Nessie branch is neither
merged nor deleted.
What a branch deploy can still reach
A branch deployment runs the pull request's code in the prod ECS cluster with the Dagster
user-code task role. That role reads the service Nessie token and the other secrets that
user code reads, such as dagster/mongo/ppm-uri and the Temporal API key. It writes the
warehouse buckets and the DynamoDB tables that user code uses. Anyone who can push a
branch and add the branch-deploy label can therefore run code with that access,
whatever the CI credentials allow.