981e4581ee
* dp rewrite stage 0, projects * remove plan files * generalize handling of basepath for projects in project-factory module * central-0 ---> core-0 * add schemas, validate YAMLs, tags * aspect types * data catalog policy tag factory * add support for data catalog taxonomy to project factory * complete retrofit of old stage configuration, except networking * shared vpc networking * networking * data platform as pf dataset * docs * test * remove legacy dp stage, fix tests and links * boilerplate * tfdoc * fix unrelated tfdoc * schemas * fix errors * schema * duplicate schemas * yamllint * Fix module naming convention for aspect-types * Fix factories_config in vpcs.tf for net-vpc-factory compatibility * Update schema documentation based on schema changes * Fix false rename conflict in .config.yaml files * Sync schemas and update documentation * Fix path expansion for aspect-types and revert projects_input to master * Restore path expansion for org_policies in projects-iam call * Fix trailing newlines in schema duplicates to satisfy duplicate-diff * Fix path expansion for data_catalog_taxonomy in taxonomies.tf * Update inventory for data-platform test and clean up debug prints * Add full values to data-platform inventory * Align Stage 2 VPC Factory integration with Stage 0 and fix tests TAG=agy * Fix project factory context resolution and data platform datasets - Update tag context keys in project factory to use file key without 'projects/' prefix. - Fix tag reference in product-0.yaml. - Fix shared_vpc_service_config in shared-0.yaml by moving service account to network_users. - Set parent for domain-0 folder to data-platform. - Mock net-dev-0 project ID in tests. - Update inventories. TAG=agy CONV=4b37fa5b-bf59-4604-9e8f-b55353d967a0 * Fix project-level tag keys context resolution in project factory * Fix commented out tag reference in domain-0 .config.yaml * Fix merge() calls with empty arguments in project-factory and data-catalog-policy-tag * Update Data Platform dataset README with prerequisites and customization guide * Add Table of Contents to Data Platform dataset README * docs: update Data Platform README with project templates tip * Document data platform output files and linking sequence in README * Update data platform README with VPC-SC and delegated IAM details * Refactor data platform dataset and align stage defaults * Update test inventory and variables for data platform with new prefix
FAST add-ons
Each of the folders contained here is a separate "add-on" that can be used to add extra functionality to a specific stage.
Add-ons can be thought of as additional thin layers on top of a stage, that reuse its IaC resources and leverage the same IAM configuration: the same service accounts are used to run the add-on, and state configuration is stored in the same bucket as their "parent stage" under a different prefix.
Add-on configuration
To configure an add-on, once you have identified its "parent stage" (networking, security, etc.) you have to configure the org setup stage so that its providers file, and associated GCS resources to host the state file, are created. The following example configures resources for the NGFW Networking add-on.
First, the new provider file is declared in the defaults.yaml file.
# defaults.yaml (snippet)
output_files:
# ...
providers:
# ...
2-networking-ngfw:
bucket: $storage_buckets:iac-0/iac-stage-state
prefix: 2-networking-ngfw
service_account: $iam_principals:service_accounts/iac-0/iac-networking-rw
Then, the GCS folder (shown here) or bucket for the Terraform state is defined in the IaC project.
# projects/iac-0.yaml
buckets:
# ...
iac-stage-state:
description: Terraform state for stage automation.
managed_folders:
# ...
2-networking-ngfw:
iam:
roles/storage.admin:
- $iam_principals:service_accounts/iac-0/iac-networking-rw
$custom_roles:storage_viewer:
- $iam_principals:service_accounts/iac-0/iac-networking-ro
Once the org setup stage is ready and applied, the add-on can be run using the generated provider files. The fast-links.sh can be used to link or copy the relevant files to the add-on, in the same way it's used for regular stages.