Skip to main content

CI credentials

The credentials that CI uses to push images and deploy are secrets of two GitHub environments. GitHub gives an environment's secrets only to a job that references the environment, and only after the job passes the environment's deployment protection rules.

  • aws-prod holds the prod credentials. Only main can deploy to it, so a workflow that someone pushes to another branch cannot read them, even when it references aws-prod.
  • aws-branch-deploy holds the branch-deploy credentials. Any branch can deploy to it, with no approval. Any workflow on any branch can therefore read them. Each of these credentials can do only what a branch deploy needs.

Never store a credential that can change production as a repository or organization secret. Every workflow on every branch can read those.

Environments​

EnvironmentProtection ruleSecretWhat it can do
aws-prodOnly main can deployAWS_AI_PROD_KEY, AWS_AI_PROD_SECRETECR, Lambda, ECS, Secrets Manager, and the resources that terraform.yml applies
aws-prodOnly main can deployDAGSTER_AGENT_TOKENDeploy code locations to prod; list and delete branch deployments
aws-branch-deployAny branch can deployAWS_BRANCH_DEPLOY_KEY, AWS_BRANCH_DEPLOY_SECRETPush to the ai-<oem>-branch ECR repositories; read dagster/nessie/m2m-writer-token
aws-branch-deployAny branch can deployDAGSTER_BRANCH_DEPLOY_TOKENCreate, update and delete Dagster+ branch deployments

The AWS key of aws-branch-deploy belongs to the IAM user gha-ppm-data-platform-branch-deploy, defined in deployments/aws/terraform/solutions/dagster-agent/ci_branch_deploy_iam.tf. Its access key is created by hand, so the secret never enters the Terraform state.

DAGSTER_BRANCH_DEPLOY_TOKEN is the user token of the Dagster+ service user github-ci-branch. The user has the Editor role on all branch deployments and no access to prod. Editor is the lowest role that can register code locations. It can also read and edit every Dagster+ environment variable scoped to branch deployments, so none of those variables may hold a credential that can change production.

The Nessie writer role can create, write and delete every branch except main. It can read main, so CI can create a branch deploy's Nessie branch from it.

A job that references aws-prod runs only in a run whose ref is main. These runs have that ref:

  • a push to main;
  • a workflow_dispatch started from main;
  • a scheduled run;
  • a pull_request_target run of a pull request into main.

A workflow_dispatch of terraform.yml started from another branch fails.

Jobs​

WorkflowJobEnvironment
ci.ymlbuild_imageaws-prod on main; aws-branch-deploy on a pull request with the branch-deploy label, in a run a person started; none otherwise
ci.ymlbuild_service_image, deploy_workflow_lambdaaws-prod on main; none otherwise
ci.ymldeployaws-prod on main; aws-branch-deploy on a pull request
terraform.ymlevery jobaws-prod
stale-branch-deployment-cleanup.ymlcleanupaws-prod
branch-deployment-cleanup.ymlteardown-deploymentaws-branch-deploy
branch-deployment-cleanup.ymlcleanupaws-prod for a pull request merged into main, whose Nessie branch the job merges into main; aws-branch-deploy for a pull request that was closed without merging or lost the label, and the job deletes the Nessie branch

stale-branch-deployment-cleanup.yml has only a schedule trigger. GitHub runs it from the latest commit on main. Nobody can start a new run by hand. Keep the schedule as its only trigger, so that no person can start a run that deletes branch deployments and Nessie branches with the aws-prod credentials.

A pull request's run builds service images but does not push them, because a branch deployment runs no service.

The cleanup job of branch-deployment-cleanup.yml does not run for a pull request merged into a branch other than main. That pull request's Nessie branch is neither merged nor deleted.

What a branch deploy can still reach​

A branch deployment runs the pull request's code in the prod ECS cluster with the Dagster user-code task role. That role reads the service Nessie token and the other secrets that user code reads, such as dagster/mongo/ppm-uri and the Temporal API key. It writes the warehouse buckets and the DynamoDB tables that user code uses. Anyone who can push a branch and add the branch-deploy label can therefore run code with that access, whatever the CI credentials allow.