02d8d8367a
* Add initial dataflow template code + TF infra * Refactor the datapipeline DAG to use flex template operator, cleanup code * Remove unneeded bash scripts, update README with manual examples * Refactor datapipeline_dc_tags.py and include new Flex template * Update docs to reflect changes * Remove sub-dependencies and keep apache beam * Add missing license headers and update tests * Set resouces to 291 in tests * Update outputs via tfdoc * Update with outputs order and tfdoc * Correct number of resources * Fix to add region into command from var * Enable service account impersonation for running builds * Update example dataflow run command to use orchestrator SA * Remove hard coded values in example * Keep original airflow files, add new which use Flex template as example * Update tests and doc * Fix number of resources in plan * Run tfdoc remove files section in README * Fix number of modules in tfdoc * Update number of resources * Add missin service account * Update DF demo README * Quick rename --------- Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
16 KiB
16 KiB
IAM bindings reference
Legend: + additive, • conditional.
Project cmn
| members | roles |
|---|---|
| gcp-data-analysts group |
roles/datacatalog.viewer |
| gcp-data-engineers group |
roles/dlp.estimatesAdmin roles/dlp.reader roles/dlp.user |
| gcp-data-security group |
roles/datacatalog.admin roles/dlp.admin |
| load-df-0 serviceAccount |
roles/datacatalog.viewer roles/dlp.user |
| trf-bq-0 serviceAccount |
roles/datacatalog.categoryFineGrainedReader roles/datacatalog.viewer |
| trf-df-0 serviceAccount |
roles/datacatalog.categoryFineGrainedReader roles/datacatalog.viewer roles/dlp.user |
Project drp
| members | roles |
|---|---|
| gcp-data-engineers group |
roles/bigquery.dataEditor roles/pubsub.editor roles/storage.admin |
| drp-bq-0 serviceAccount |
roles/bigquery.dataEditor |
| drp-cs-0 serviceAccount |
roles/storage.objectCreator |
| drp-ps-0 serviceAccount |
roles/pubsub.publisher |
| load-df-0 serviceAccount |
roles/bigquery.user roles/pubsub.subscriber roles/storage.admin roles/storage.objectAdmin |
| orc-cmp-0 serviceAccount |
roles/pubsub.subscriber roles/storage.objectViewer |
Project dwh-conf
| members | roles |
|---|---|
| gcp-data-analysts group |
roles/bigquery.dataViewer roles/bigquery.jobUser roles/bigquery.metadataViewer roles/bigquery.user roles/datacatalog.tagTemplateViewer roles/datacatalog.viewer roles/storage.objectViewer |
| gcp-data-engineers group |
roles/bigquery.dataEditor roles/storage.admin |
| SERVICE_IDENTITY_service-networking serviceAccount |
roles/servicenetworking.serviceAgent + |
| load-df-0 serviceAccount |
roles/datacatalog.categoryAdmin |
| trf-bq-0 serviceAccount |
roles/bigquery.dataOwner roles/bigquery.jobUser |
| trf-df-0 serviceAccount |
roles/bigquery.dataOwner roles/storage.objectCreator roles/storage.objectViewer |
Project dwh-cur
| members | roles |
|---|---|
| gcp-data-analysts group |
roles/bigquery.dataViewer roles/bigquery.jobUser roles/bigquery.metadataViewer roles/bigquery.user roles/datacatalog.tagTemplateViewer roles/datacatalog.viewer roles/storage.objectViewer |
| gcp-data-engineers group |
roles/bigquery.dataEditor roles/storage.admin |
| SERVICE_IDENTITY_service-networking serviceAccount |
roles/servicenetworking.serviceAgent + |
| load-df-0 serviceAccount |
roles/datacatalog.categoryAdmin |
| trf-bq-0 serviceAccount |
roles/bigquery.dataOwner roles/bigquery.jobUser |
| trf-df-0 serviceAccount |
roles/bigquery.dataOwner roles/storage.objectCreator roles/storage.objectViewer |
Project dwh-lnd
| members | roles |
|---|---|
| gcp-data-analysts group |
roles/bigquery.dataViewer roles/bigquery.jobUser roles/bigquery.metadataViewer roles/bigquery.user roles/datacatalog.tagTemplateViewer roles/datacatalog.viewer roles/storage.objectViewer |
| gcp-data-engineers group |
roles/bigquery.dataEditor roles/storage.admin |
| SERVICE_IDENTITY_service-networking serviceAccount |
roles/servicenetworking.serviceAgent + |
| load-df-0 serviceAccount |
roles/bigquery.dataOwner roles/bigquery.jobUser roles/storage.objectCreator |
| trf-bq-0 serviceAccount |
roles/bigquery.dataOwner roles/datacatalog.categoryAdmin |
| trf-df-0 serviceAccount |
roles/bigquery.dataOwner |
Project lod
| members | roles |
|---|---|
| gcp-data-engineers group |
roles/compute.viewer roles/dataflow.admin roles/dataflow.developer roles/viewer |
| SERVICE_IDENTITY_dataflow-service-producer-prod serviceAccount |
roles/storage.objectAdmin |
| SERVICE_IDENTITY_service-networking serviceAccount |
roles/servicenetworking.serviceAgent + |
| load-df-0 serviceAccount |
roles/bigquery.jobUser roles/dataflow.admin roles/dataflow.worker roles/storage.objectAdmin |
| orc-cmp-0 serviceAccount |
roles/dataflow.admin |
Project orc
Project trf
| members | roles |
|---|---|
| gcp-data-engineers group |
roles/bigquery.jobUser roles/dataflow.admin |
| SERVICE_IDENTITY_dataflow-service-producer-prod serviceAccount |
roles/storage.objectAdmin |
| SERVICE_IDENTITY_service-networking serviceAccount |
roles/servicenetworking.serviceAgent + |
| orc-cmp-0 serviceAccount |
roles/dataflow.admin |
| trf-bq-0 serviceAccount |
roles/bigquery.jobUser |
| trf-df-0 serviceAccount |
roles/dataflow.worker roles/storage.objectAdmin |