2eaa0d5e27
* Allow creation of dynamic tags * Extend project factory and related modules to support dynamic values * Extend folder and organization modules * project and organization readme * Simplify dynamic tag support and remove unnecessary restrictions • Schemas & Validations: Removed the restriction that forbade combining IAM fields with allowed_values_regex on tags. Updated validations in project and organization modules, and simplified all relevant JSON schemas. • Module Tag Bindings: Simplified the tag_value assignment in folder , project , gcs , bigquery-dataset , and kms modules by removing the defensive can(regex(...)) check and calling templatestring directly. • Outputs: Removed the tags_dynamic output from project and organization modules, as the same information is now available in tag_keys . • Project Factory: Updated tag_vars_projects in projects.tf to use the native namespaced_name attribute and filtered manually for dynamic tags. * fix(organization, project): fix linting and tests for dynamic tag support - Align allowed_values_regex and description extraction in _tags_merged locals to use lookup() for consistency with other fields. - Fix spacing in project context variable (alphabetical ordering). - Update organization tags test to include the new cost_center tag key with allowed_values_regex. - Update project tags test to include the new cost_center tag key and reflect the resolved allowed_values_regex on environment. * refactor(gcs): refine tag bindings and fix context test - Add _tag_bindings local to pre-resolve context references, enabling templatestring to receive a direct map reference (required by Terraform). - Use var.context.tag_vars instead of the non-existent local.ctx.tag_vars. - Fix HCL syntax in context.tfvars (escaped inner quotes). - Update context test inventory to reflect 3 tag bindings including a dynamic value resolved via templatestring. * refactor: align modules with tag binding context pattern - Add _tag_bindings local + templatestring dance to cloud-run-v2, compute-vm, folder, kms modules (bigquery-dataset already had it) - Exclude tag_vars from local.ctx in cloud-run-v2, compute-vm, folder, kms, project modules (bigquery-dataset already had it) - Add tag_vars to context variable in cloud-run-v2, compute-vm modules (others already had it) - Update all context tests with dynamic tag binding values using var.context.tag_vars * docs: add module-level tftest.yaml test instructions to GEMINI.md * docs: regenerate READMEs after tag-regex alignment - Regenerate variable tables in 7 module READMEs to reflect line number shifts from prior tag-regex changes - Add tag_vars exclusion to gcs ctx local - Fix whitespace alignment in iam-service-account and project-factory tag_vars blocks - Update tftest resource counts for organization and project - Remove tags_dynamic from organization/project output tables * fix(project-factory): update test inventory for tag_bindings module split - Move tag binding address from folder-2 to folder-2-iam in test inventory (tag_bindings moved from creation to IAM modules) - Update module instance count from 34 to 35 - Regenerate README tables after terraform fmt line shifts - Apply terraform fmt to variables.tf * refactor(project-factory): remove unnecessary depends_on from folder-iam modules Folder IAM modules depend on their own folder creation modules, not on module.projects. The explicit depends_on was leftover from an earlier design. * FAST stages * Address review comments. - FAST Stages: - Added tag_keys to output-files.tf in 0-org-setup to pass org tags via tfvars. - Sorted tag_keys and tag_values in output-files.tf. - Updated project-factory, networking, and security stages to use tag_keys. - Filtered tag_keys for dynamic tags only. - Modules: - Excluded tag_vars from local.ctx in iam-service-account and organization. - Simplified tag_value in iam-service-account. - Tests: - Updated test inventories for 0-org-setup and project-factory. * Fix tf format * Fix tfdoc * docs: add ADR for templatestring vars convention and update status of base path ADR * More tfdoc * Update schemas * Use endswith in context loop * Address review * Update FAST readmes * Update last modules * Terraform fmt * Revert alloydb * Fix whitespace --------- Co-authored-by: Ludovico Magnocavallo <ludo@qix.it>
FAST stages
Each of the folders contained here is a separate "stage", or Terraform root module.
Each stage can be run in isolation (for example to only bring up a hub and spoke VPC in an existing environment), but when combined together they form a modular setup that allows top-down configuration of a whole GCP organization.
When deploying as part of a whole organization setup, each stage provides information on its resources to the following stages via predefined contracts, and each stage can pick and choose what to leverage from the preceding ones.
This has two important consequences:
- any stage can be swapped out and replaced by different code as long as it respects the contract, by providing a predefined set of outputs and optionally accepting a predefined set of variables
- data flow between stages can be partially automated (see stage 0 documentation on output files), reducing the effort and pain required to compile variables by hand
One important assumption is that the flow of data is always forward looking (or sideways for optional components), so no stage needs to depend on outputs generated further down the chain. This greatly simplifies both the logic and the implementation, and allows stages to be effectively independent.
To achieve this, we rely on specific GCP functionality like delegated role grants to allow controlled delegation of responsibilities, and conditional access via tags to constrain scope for organization-level roles or when specific resources are managed lower in the chain than IAM bindings.
Refer to each stage's documentation for a detailed description of its purpose, the architectural choices made in its design, and how it can be configured and wired together to terraform a whole GCP organization. The following is a brief overview of each stage.
Stages encapsulate core designs and functionality that is common in most type of GCP organization set-ups. Specialized designs or additional configurations that add specific functionality on top of stages to meet very specific use cases are defined via add-ons.
To destroy a previous FAST deployment follow the instructions detailed in cleanup.
Organization (0)
- Organization Setup This stage combines the legacy bootstrap and resource management stages described below, allowing easy configuration of all related resources via factories. Its flexibility supports any type of organizational design, while still supporting traditional FAST stages like VPC Service Controls, security, networking, and any stage 3.
VPC Service Controls (1)
- VPC Service Controls Optionally configures VPC Service Controls protection for the organization.
Shared resources (2)
- Security
Manages centralized security configurations in a separate stage, and is typically owned by the security team. This stage implements VPC Security Controls via separate perimeters for environments and central services, and creates projects to host centralized KMS keys used by the whole organization. It's meant to be easily extended to include other security-related resources which are required, like Secret Manager.
Exports: KMS key ids, CA ids - Networking (Networking factory) Manages centralized network resources in a separate stage, and is typically owned by the networking team. This stage provides several different design as YaML datasets, including hub-and-spoke with VPC Peerings, VPNs, NVAs and NCC. Exports: host project ids and numbers, vpc self links
- Project Factory YAML-based factory to create and configure application or team-level projects. Configuration includes VPC-level settings for Shared VPC, service-level configuration for CMEK encryption via centralized keys, and service account creation for workloads and applications. This stage can be cloned if an org-wide or dedicated per-environment factories are needed.
Importing existing setup into FAST
For brownfield implementations you may need to import existing setting in the organization, folders, etc. These snippets can help you add existing settings into the YAML file
Scripts below require yq in at least version 4. It was tested using yq v4.47.2.
IAM bindings
To create iam: part of the factory YAML file, you can use following snippet:
gcloud <resource> get-iam-policy <resource name> | yq '.bindings | map({"key": .role, "value": .members}) | from_entries'
For example use following code, to get IAM bindings on organization level to be used in 0-org-setup/dataset/.../organization/.config.yaml
gcloud organizations get-iam-policy 12345 | yq '.bindings | map({"key": .role, "value": .members}) | from_entries'
To create iam_by_principals: part of the factory YAML file, you can use following snippet:
gcloud <resource> get-iam-policy <resource name> | yq '
[.bindings | .[] | .members[] as $member | { "member": $member, "role": .role}] |
group_by(.member) | sort_by(.[0].member) | .[] | { .[0].member: map(.role)}
'