Add project-factory based data platform dataset to FAST project factory stage (#3957)

* dp rewrite stage 0, projects

* remove plan files

* generalize handling of basepath for projects in project-factory module

* central-0 ---> core-0

* add schemas, validate YAMLs, tags

* aspect types

* data catalog policy tag factory

* add support for data catalog taxonomy to project factory

* complete retrofit of old stage configuration, except networking

* shared vpc networking

* networking

* data platform as pf dataset

* docs

* test

* remove legacy dp stage, fix tests and links

* boilerplate

* tfdoc

* fix unrelated tfdoc

* schemas

* fix errors

* schema

* duplicate schemas

* yamllint

* Fix module naming convention for aspect-types

* Fix factories_config in vpcs.tf for net-vpc-factory compatibility

* Update schema documentation based on schema changes

* Fix false rename conflict in .config.yaml files

* Sync schemas and update documentation

* Fix path expansion for aspect-types and revert projects_input to master

* Restore path expansion for org_policies in projects-iam call

* Fix trailing newlines in schema duplicates to satisfy duplicate-diff

* Fix path expansion for data_catalog_taxonomy in taxonomies.tf

* Update inventory for data-platform test and clean up debug prints

* Add full values to data-platform inventory

* Align Stage 2 VPC Factory integration with Stage 0 and fix tests

TAG=agy

* Fix project factory context resolution and data platform datasets

- Update tag context keys in project factory to use file key without 'projects/' prefix.
- Fix tag reference in product-0.yaml.
- Fix shared_vpc_service_config in shared-0.yaml by moving service account to network_users.
- Set parent for domain-0 folder to data-platform.
- Mock net-dev-0 project ID in tests.
- Update inventories.

TAG=agy
CONV=4b37fa5b-bf59-4604-9e8f-b55353d967a0

* Fix project-level tag keys context resolution in project factory

* Fix commented out tag reference in domain-0 .config.yaml

* Fix merge() calls with empty arguments in project-factory and data-catalog-policy-tag

* Update Data Platform dataset README with prerequisites and customization guide

* Add Table of Contents to Data Platform dataset README

* docs: update Data Platform README with project templates tip

* Document data platform output files and linking sequence in README

* Update data platform README with VPC-SC and delegated IAM details

* Refactor data platform dataset and align stage defaults

* Update test inventory and variables for data platform with new prefix
This commit is contained in:
Ludovico Magnocavallo
2026-05-12 16:44:32 +02:00
committed by GitHub
parent 3b830dd3e4
commit 981e4581ee
61 changed files with 3471 additions and 40 deletions

View File

@@ -766,6 +766,8 @@ iam:
- $iam_principals:service_accounts/dev-tb-app0-0/automation/rw
"roles/viewer":
- $iam_principals:service_accounts/dev-tb-app0-0/automation/ro
factories_config:
data_catalog_taxonomy: data/taxonomies/sample.yaml
shared_vpc_host_config:
enabled: true
service_accounts:
@@ -871,6 +873,7 @@ compute.disableSerialPortAccess:
| name | description | modules | resources |
|---|---|---|---|
| [aspect-types.tf](./aspect-types.tf) | Aspect types resources. | <code>dataplex-aspect-types</code> | |
| [automation.tf](./automation.tf) | None | <code>gcs</code> · <code>iam-service-account</code> | |
| [budgets.tf](./budgets.tf) | Billing budget factory locals. | <code>billing-account</code> | |
| [folders.tf](./folders.tf) | Folder hierarchy factory resources. | <code>folder</code> | |
@@ -885,6 +888,7 @@ compute.disableSerialPortAccess:
| [projects-pubsub.tf](./projects-pubsub.tf) | None | <code>pubsub</code> | |
| [projects-service-accounts.tf](./projects-service-accounts.tf) | None | <code>iam-service-account</code> | |
| [projects.tf](./projects.tf) | None | <code>project</code> | <code>terraform_data</code> |
| [taxonomies.tf](./taxonomies.tf) | Taxonomy resources. | <code>data-catalog-policy-tag</code> | |
| [variables-billing.tf](./variables-billing.tf) | None | | |
| [variables-folders.tf](./variables-folders.tf) | None | | |
| [variables-projects.tf](./variables-projects.tf) | None | | |
@@ -907,20 +911,20 @@ compute.disableSerialPortAccess:
| name | description | sensitive |
|---|---|:---:|
| [folder_ids](outputs.tf#L102) | Folder ids. | |
| [iam_principals](outputs.tf#L107) | IAM principals mappings. | |
| [kms_keys](outputs.tf#L112) | KMS key ids. | |
| [log_buckets](outputs.tf#L117) | Log bucket ids. | |
| [project_ids](outputs.tf#L124) | Project ids. | |
| [project_numbers](outputs.tf#L129) | Project numbers. | |
| [projects](outputs.tf#L136) | Project attributes. | |
| [pubsub_topics](outputs.tf#L141) | PubSub topic ids. | |
| [service_account_emails](outputs.tf#L148) | Service account emails. | |
| [service_account_iam_emails](outputs.tf#L155) | Service account IAM-format emails. | |
| [service_account_ids](outputs.tf#L162) | Service account IDs. | |
| [service_accounts](outputs.tf#L169) | Service account emails. | |
| [service_agents](outputs.tf#L174) | Service agent emails. | |
| [storage_buckets](outputs.tf#L185) | Bucket names. | |
| [folder_ids](outputs.tf#L107) | Folder ids. | |
| [iam_principals](outputs.tf#L112) | IAM principals mappings. | |
| [kms_keys](outputs.tf#L117) | KMS key ids. | |
| [log_buckets](outputs.tf#L122) | Log bucket ids. | |
| [project_ids](outputs.tf#L129) | Project ids. | |
| [project_numbers](outputs.tf#L134) | Project numbers. | |
| [projects](outputs.tf#L141) | Project attributes. | |
| [pubsub_topics](outputs.tf#L146) | PubSub topic ids. | |
| [service_account_emails](outputs.tf#L153) | Service account emails. | |
| [service_account_iam_emails](outputs.tf#L160) | Service account IAM-format emails. | |
| [service_account_ids](outputs.tf#L167) | Service account IDs. | |
| [service_accounts](outputs.tf#L174) | Service account emails. | |
| [service_agents](outputs.tf#L179) | Service agent emails. | |
| [storage_buckets](outputs.tf#L190) | Bucket names. | |
<!-- END TFDOC -->
## Tests

View File

@@ -0,0 +1,44 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
# tfdoc:file:description Aspect types resources.
module "aspect-types" {
source = "../dataplex-aspect-types"
for_each = {
for k, v in local.projects_input : k => v
if try(v.factories_config.aspect_types, null) != null
}
project_id = module.projects[each.key].project_id
factories_config = {
aspect_types = lookup(each.value.factories_config, "aspect_types", null) == null ? null : try(pathexpand(
var.factories_config.basepath == null || startswith(each.value.factories_config.aspect_types, "/") || startswith(each.value.factories_config.aspect_types, ".")
? each.value.factories_config.aspect_types :
"${var.factories_config.basepath}/${each.value.factories_config.aspect_types}"
), null)
}
context = merge(local.ctx, {
iam_principals = merge(
local.ctx_iam_principals,
lookup(local.self_sas_iam_emails, each.key, {}),
local.projects_service_agents
)
project_ids = merge(
local.ctx.project_ids,
{ for k, v in module.projects : k => v.project_id }
)
})
}

View File

@@ -0,0 +1,26 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
description: taxonomy description
activated_policy_types:
- FINE_GRAINED_ACCESS_CONTROL
iam:
roles/viewer:
- $iam_principals:gcp-devops
tags:
tag-a:
description: tag a description
iam:
roles/viewer:
- $iam_principals:gcp-devops

View File

@@ -23,6 +23,11 @@ locals {
}
outputs_projects = {
for k, v in local.projects_input : k => {
aspect_types = (
v.factories_config.aspect_types == null
? {}
: module.aspect-types[k].ids
)
automation = {
bucket = try(
module.automation-bucket[local._outputs_automation_buckets[k]].name,

View File

@@ -51,7 +51,9 @@ locals {
local.data_defaults.defaults.contacts
)
factories_config = {
aspect_types = try(v.factories_config.aspect_types, null)
custom_roles = try(v.factories_config.custom_roles, null)
data_catalog_taxonomy = try(v.factories_config.data_catalog_taxonomy, null)
observability = try(v.factories_config.observability, null)
org_policies = try(v.factories_config.org_policies, null)
pam_entitlements = try(v.factories_config.pam_entitlements, null)

View File

@@ -53,18 +53,18 @@ locals {
ctx_project_numbers = merge(local.ctx.project_numbers, local.project_numbers)
# cross-project tag contexts, keyed on project name
ctx_tag_keys = merge(local.ctx.tag_keys, {
for k, v in merge([
for k, v in merge({}, [
for pk, pv in local.projects_input : {
for tk, tv in module.projects[pk].tag_keys :
"${pv.name}/${tk}" => tv.id
"${pk}/${tk}" => tv.id
}
]...) : k => v
})
ctx_tag_values = merge(local.ctx.tag_values, {
for k, v in merge([
for k, v in merge({}, [
for pk, pv in local.projects_input : {
for tk, tv in module.projects[pk].tag_values :
"${pv.name}/${tk}" => tv.id
"${pk}/${tk}" => tv.id
}
]...) : k => v
})
@@ -140,13 +140,16 @@ module "projects" {
folder_ids = local.ctx_folder_ids
})
default_service_account = try(each.value.default_service_account, "keep")
# Exclude factories that are either:
# a) Handled in parallel by calling specific modules (e.g., aspect_types, data_catalog_taxonomy)
# b) Handled in the projects-iam call to leverage expanded context (e.g., org_policies)
factories_config = {
for k, v in each.value.factories_config : k => try(pathexpand(
var.factories_config.basepath == null || startswith(v, "/") || startswith(v, ".")
? v :
"${var.factories_config.basepath}/${v}"
), null)
if k != "org_policies"
if !contains(["aspect_types", "data_catalog_taxonomy", "org_policies"], k)
}
kms_autokeys = try(each.value.kms.autokeys, {})
labels = merge(

View File

@@ -0,0 +1,125 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Dataplex Aspect Type",
"type": "object",
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"display_name": {
"type": "string"
},
"labels": {
"type": "object"
},
"metadata_template": {
"type": "string"
},
"iam": {
"$ref": "#/$defs/iam"
},
"iam_bindings": {
"$ref": "#/$defs/iam_bindings"
},
"iam_bindings_additive": {
"$ref": "#/$defs/iam_bindings_additive"
}
},
"$defs": {
"iam": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^(?:roles/|\\$custom_roles:|organizations/[0-9]+/roles/|([a-z0-9.]+:)?projects/[a-z0-9-]+/roles/)": {
"type": "array",
"items": {
"type": "string",
"pattern": "^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:||\\$iam_principals:[a-z0-9_-]+)"
}
}
}
},
"iam_bindings": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^[a-z0-9_-]+$": {
"type": "object",
"additionalProperties": false,
"properties": {
"members": {
"type": "array",
"items": {
"type": "string",
"pattern": "^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\\$iam_principals:[a-z0-9_-]+)"
}
},
"role": {
"type": "string",
"pattern": "^(?:roles/|\\$custom_roles:|organizations/[0-9]+/roles/|([a-z0-9.]+:)?projects/[a-z0-9-]+/roles/)"
},
"condition": {
"type": "object",
"additionalProperties": false,
"required": [
"expression",
"title"
],
"properties": {
"expression": {
"type": "string"
},
"title": {
"type": "string"
},
"description": {
"type": "string"
}
}
}
}
}
}
},
"iam_bindings_additive": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^[a-z0-9_-]+$": {
"type": "object",
"additionalProperties": false,
"properties": {
"member": {
"type": "string",
"pattern": "^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\\$iam_principals:[a-z0-9_-]+)"
},
"role": {
"type": "string",
"pattern": "^(?:roles/|\\$custom_roles:|organizations/[0-9]+/roles/|([a-z0-9.]+:)?projects/[a-z0-9-]+/roles/)"
},
"condition": {
"type": "object",
"additionalProperties": false,
"required": [
"expression",
"title"
],
"properties": {
"expression": {
"type": "string"
},
"title": {
"type": "string"
},
"description": {
"type": "string"
}
}
}
}
}
}
}
}
}

View File

@@ -0,0 +1,50 @@
# Dataplex Aspect Type
<!-- markdownlint-disable MD036 -->
## Properties
*additional properties: false*
- **description**: *string*
- **display_name**: *string*
- **labels**: *object*
- **metadata_template**: *string*
- **iam**: *reference([iam](#refs-iam))*
- **iam_bindings**: *reference([iam_bindings](#refs-iam_bindings))*
- **iam_bindings_additive**: *reference([iam_bindings_additive](#refs-iam_bindings_additive))*
## Definitions
- **iam**<a name="refs-iam"></a>: *object*
<br>*additional properties: false*
- **`^(?:roles/|\$custom_roles:|organizations/[0-9]+/roles/|([a-z0-9.]+:)?projects/[a-z0-9-]+/roles/)`**: *array*
- items: *string*
<br>*pattern: ^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:||\$iam_principals:[a-z0-9_-]+)*
- **iam_bindings**<a name="refs-iam_bindings"></a>: *object*
<br>*additional properties: false*
- **`^[a-z0-9_-]+$`**: *object*
<br>*additional properties: false*
- **members**: *array*
- items: *string*
<br>*pattern: ^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\$iam_principals:[a-z0-9_-]+)*
- **role**: *string*
<br>*pattern: ^(?:roles/|\$custom_roles:|organizations/[0-9]+/roles/|([a-z0-9.]+:)?projects/[a-z0-9-]+/roles/)*
- **condition**: *object*
<br>*additional properties: false*
- ⁺**expression**: *string*
- ⁺**title**: *string*
- **description**: *string*
- **iam_bindings_additive**<a name="refs-iam_bindings_additive"></a>: *object*
<br>*additional properties: false*
- **`^[a-z0-9_-]+$`**: *object*
<br>*additional properties: false*
- **member**: *string*
<br>*pattern: ^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\$iam_principals:[a-z0-9_-]+)*
- **role**: *string*
<br>*pattern: ^(?:roles/|\$custom_roles:|organizations/[0-9]+/roles/|([a-z0-9.]+:)?projects/[a-z0-9-]+/roles/)*
- **condition**: *object*
<br>*additional properties: false*
- ⁺**expression**: *string*
- ⁺**title**: *string*
- **description**: *string*

View File

@@ -111,6 +111,9 @@
"type": "object",
"additionalProperties": false,
"properties": {
"display_name": {
"type": "string"
},
"description": {
"type": "string"
},
@@ -297,9 +300,15 @@
"type": "object",
"additionalProperties": false,
"properties": {
"aspect_types": {
"type": "string"
},
"custom_roles": {
"type": "string"
},
"data_catalog_taxonomy": {
"type": "string"
},
"observability": {
"type": "string"
},

View File

@@ -37,6 +37,7 @@
<br>*additional properties: false*
- **`^[a-z0-9-]+$`**: *object*
<br>*additional properties: false*
- **display_name**: *string*
- **description**: *string*
- **prefix**: *string*
- **iam**: *reference([iam](#refs-iam))*
@@ -95,7 +96,9 @@
<br>*enum: ['PREVENT', 'DELETE', 'ABANDON']*
- **factories_config**: *object*
<br>*additional properties: false*
- **aspect_types**: *string*
- **custom_roles**: *string*
- **data_catalog_taxonomy**: *string*
- **observability**: *string*
- **org_policies**: *string*
- **quotas**: *string*

View File

@@ -0,0 +1,153 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": false,
"properties": {
"activated_policy_types": {
"type": "array",
"items": {
"type": "string"
}
},
"description": {
"type": "string"
},
"iam": {
"$ref": "#/$defs/iam"
},
"iam_bindings": {
"$ref": "#/$defs/iam_bindings"
},
"iam_bindings_additive": {
"$ref": "#/$defs/iam_bindings_additive"
},
"tags": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^.+$": {
"type": "object",
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"iam": {
"$ref": "#/$defs/iam"
},
"iam_bindings": {
"$ref": "#/$defs/iam_bindings"
},
"iam_bindings_additive": {
"$ref": "#/$defs/iam_bindings_additive"
}
}
}
}
}
},
"$defs": {
"iam": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^(?:roles/|\\$custom_roles:)": {
"type": "array",
"items": {
"type": "string",
"pattern": "^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\\$iam_principals:[a-z0-9_-]+)"
}
}
}
},
"iam_bindings": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^[a-z0-9_-]+$": {
"type": "object",
"additionalProperties": false,
"required": [
"members",
"role"
],
"properties": {
"members": {
"type": "array",
"items": {
"type": "string",
"pattern": "^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\\$iam_principals:[a-z0-9_-]+)"
}
},
"role": {
"type": "string",
"pattern": "^(?:roles/|\\$custom_roles:)"
},
"condition": {
"type": "object",
"additionalProperties": false,
"required": [
"expression",
"title"
],
"properties": {
"expression": {
"type": "string"
},
"title": {
"type": "string"
},
"description": {
"type": "string"
}
}
}
}
}
}
},
"iam_bindings_additive": {
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^[a-z0-9_-]+$": {
"type": "object",
"additionalProperties": false,
"required": [
"member",
"role"
],
"properties": {
"member": {
"type": "string",
"pattern": "^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\\$iam_principals:[a-z0-9_-]+)"
},
"role": {
"type": "string",
"pattern": "^(?:roles/|\\$custom_roles:)"
},
"condition": {
"type": "object",
"additionalProperties": false,
"required": [
"expression",
"title"
],
"properties": {
"expression": {
"type": "string"
},
"title": {
"type": "string"
},
"description": {
"type": "string"
}
}
}
}
}
}
}
}
}

View File

@@ -0,0 +1,57 @@
# None
<!-- markdownlint-disable MD036 -->
## Properties
*additional properties: false*
- **activated_policy_types**: *array*
- items: *string*
- **description**: *string*
- **iam**: *reference([iam](#refs-iam))*
- **iam_bindings**: *reference([iam_bindings](#refs-iam_bindings))*
- **iam_bindings_additive**: *reference([iam_bindings_additive](#refs-iam_bindings_additive))*
- **tags**: *object*
<br>*additional properties: false*
- **`^.+$`**: *object*
<br>*additional properties: false*
- **description**: *string*
- **iam**: *reference([iam](#refs-iam))*
- **iam_bindings**: *reference([iam_bindings](#refs-iam_bindings))*
- **iam_bindings_additive**: *reference([iam_bindings_additive](#refs-iam_bindings_additive))*
## Definitions
- **iam**<a name="refs-iam"></a>: *object*
<br>*additional properties: false*
- **`^(?:roles/|\$custom_roles:)`**: *array*
- items: *string*
<br>*pattern: ^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\$iam_principals:[a-z0-9_-]+)*
- **iam_bindings**<a name="refs-iam_bindings"></a>: *object*
<br>*additional properties: false*
- **`^[a-z0-9_-]+$`**: *object*
<br>*additional properties: false*
- ⁺**members**: *array*
- items: *string*
<br>*pattern: ^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\$iam_principals:[a-z0-9_-]+)*
- ⁺**role**: *string*
<br>*pattern: ^(?:roles/|\$custom_roles:)*
- **condition**: *object*
<br>*additional properties: false*
- ⁺**expression**: *string*
- ⁺**title**: *string*
- **description**: *string*
- **iam_bindings_additive**<a name="refs-iam_bindings_additive"></a>: *object*
<br>*additional properties: false*
- **`^[a-z0-9_-]+$`**: *object*
<br>*additional properties: false*
- ⁺**member**: *string*
<br>*pattern: ^(?:domain:|group:|serviceAccount:|user:|principal:|principalSet:|\$iam_principals:[a-z0-9_-]+)*
- ⁺**role**: *string*
<br>*pattern: ^(?:roles/|\$custom_roles:)*
- **condition**: *object*
<br>*additional properties: false*
- ⁺**expression**: *string*
- ⁺**title**: *string*
- **description**: *string*

View File

@@ -0,0 +1,46 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
# tfdoc:file:description Taxonomy resources.
module "taxonomies" {
source = "../data-catalog-policy-tag"
for_each = {
for k, v in local.projects_input : k => v
if try(v.factories_config.data_catalog_taxonomy, null) != null
}
project_id = module.projects[each.key].project_id
factories_config = {
taxonomy = lookup(each.value.factories_config, "data_catalog_taxonomy", null) == null ? null : try(pathexpand(
var.factories_config.basepath == null || startswith(each.value.factories_config.data_catalog_taxonomy, "/") || startswith(each.value.factories_config.data_catalog_taxonomy, ".")
? each.value.factories_config.data_catalog_taxonomy :
"${var.factories_config.basepath}/${each.value.factories_config.data_catalog_taxonomy}"
), null)
}
name = "taxonomy"
location = try(each.value.locations.storage, "europe-west1")
context = merge(local.ctx, {
iam_principals = merge(
local.ctx_iam_principals,
lookup(local.self_sas_iam_emails, each.key, {}),
local.projects_service_agents
)
project_ids = merge(
local.ctx.project_ids,
{ for k, v in module.projects : k => v.project_id }
)
})
}

View File

@@ -240,7 +240,9 @@ variable "projects" {
threat_detector_provider = optional(string)
}), {})
factories_config = optional(object({
aspect_types = optional(string)
custom_roles = optional(string)
data_catalog_taxonomy = optional(string)
observability = optional(string)
org_policies = optional(string)
pam_entitlements = optional(string)