Skip to main content

GCP Access

Dagster user code reads BigQuery and Cloud Storage in the OEM GCP projects through Workload Identity Federation. No service-account key exists. The ECS task's AWS role is the identity, and GCP grants the BigQuery and Cloud Storage roles to that role directly. No service account is impersonated.

Trust path​

  1. Every code server, run task and step task runs as the Dagster agent stack's UserCodeExecutionRole.
  2. The dagster provider of the ppm-data-platform-aws workload identity pool, in ai-app-386318, accepts a GetCallerIdentity request that this role signs. Its attribute condition rejects every other role.
  3. GCP's Security Token Service exchanges the verified request for a federated access token. The token's principal is the pool's attribute.aws_identity principal set for <account>:<role name>.
  4. That principal set holds the BigQuery roles in each OEM project, and roles/storage.objectViewer in ai-app-stellantis.

The GCP iam Terraform solution defines the provider and the grants in deployments/gcp/terraform/solutions/iam/dagster_federation.tf. Its dagster_workload_identity_audience output is the value of the GCP_WORKLOAD_IDENTITY_AUDIENCE variable that each BigQuery code location sets in its container_context.yaml.

Cloud Storage accepts a federated principal only on buckets with uniform bucket-level access. A federated principal cannot create signed URLs.

Credentials in code​

ai_dagster.resources.gcp_credentials() returns the credentials every GCP client in user code uses:

  • With GCP_WORKLOAD_IDENTITY_AUDIENCE set, it returns federated credentials for that pool provider.
  • Without it, it returns Application Default Credentials. Local development uses this path: run gcloud auth application-default login.

It builds the credentials once per process, so every client shares one access token, which google-auth refreshes before it expires.

google-auth reads AWS credentials from the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables or from the EC2 metadata server. An ECS task has neither: its credentials come from the container credentials endpoint. The federated credentials therefore take a BotoAwsCredentialsSupplier, which reads them through boto3's credential chain. The supplier is a constructor argument, so a credential configuration file cannot express it.

ai_dagster.components.BigQueryResourceComponent binds a BigQueryClientResource to a resource key. The resource opens a BigQuery client with gcp_credentials(). Code that opens a BigQuery Storage BigQueryReadClient passes it the same credentials, because the read client otherwise authenticates with Application Default Credentials.

Adding a code location that reads GCP​

  1. Declare an ai_dagster.components.BigQueryResourceComponent in the location's defs/resources/defs.yaml.
  2. Add GCP_WORKLOAD_IDENTITY_AUDIENCE to the env_vars of its container_context.yaml.
  3. If the location reads a GCP project that has no grant yet, add the project to dagster_project_roles in dagster_federation.tf.