GCP Access
Dagster user code reads BigQuery and Cloud Storage in the OEM GCP projects through Workload Identity Federation. No service-account key exists. The ECS task's AWS role is the identity, and GCP grants the BigQuery and Cloud Storage roles to that role directly. No service account is impersonated.
Trust path
- Every code server, run task and step task runs as the Dagster agent stack's
UserCodeExecutionRole. - The
dagsterprovider of theppm-data-platform-awsworkload identity pool, inai-app-386318, accepts aGetCallerIdentityrequest that this role signs. Its attribute condition rejects every other role. - GCP's Security Token Service exchanges the verified request for a federated access
token. The token's principal is the pool's
attribute.aws_identityprincipal set for<account>:<role name>. - That principal set holds the BigQuery roles in each OEM project, and
roles/storage.objectViewerinai-app-stellantis.
The GCP iam Terraform solution defines the provider and the grants in
deployments/gcp/terraform/solutions/iam/dagster_federation.tf. Its
dagster_workload_identity_audience output is the value of the
GCP_WORKLOAD_IDENTITY_AUDIENCE variable that each BigQuery code location sets in its
container_context.yaml.
Cloud Storage accepts a federated principal only on buckets with uniform bucket-level access. A federated principal cannot create signed URLs.
Credentials in code
ai_dagster.resources.gcp_credentials() returns the credentials every GCP client in user
code uses:
- With
GCP_WORKLOAD_IDENTITY_AUDIENCEset, it returns federated credentials for that pool provider. - Without it, it returns Application Default Credentials. Local development uses this
path: run
gcloud auth application-default login.
It builds the credentials once per process, so every client shares one access token, which google-auth refreshes before it expires.
google-auth reads AWS credentials from the AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables or from the EC2
metadata server. An ECS task has neither: its credentials come
from the container credentials endpoint. The federated credentials therefore take a
BotoAwsCredentialsSupplier, which reads them through boto3's credential chain. The
supplier is a constructor argument, so a credential configuration file cannot express it.
ai_dagster.components.BigQueryResourceComponent binds a BigQueryClientResource to a
resource key. The resource opens a BigQuery client with gcp_credentials(). Code that
opens a BigQuery Storage BigQueryReadClient passes it the same credentials, because the
read client otherwise authenticates with Application Default Credentials.
Adding a code location that reads GCP
- Declare an
ai_dagster.components.BigQueryResourceComponentin the location'sdefs/resources/defs.yaml. - Add
GCP_WORKLOAD_IDENTITY_AUDIENCEto theenv_varsof itscontainer_context.yaml. - If the location reads a GCP project that has no grant yet, add the project to
dagster_project_rolesindagster_federation.tf.