Commit Graph

57 Commits

Author SHA1 Message Date
Julio Castillo
008a3719ad Support service_agents_config.skip_iam in project-factory and fast stages (#4007)
* Support service_agents_config.skip_iam in project-factory and fast stages

* Fix inventories

* Change service-agent creation/iam order
2026-06-01 10:04:54 +00:00
Ludovico Magnocavallo
981e4581ee Add project-factory based data platform dataset to FAST project factory stage (#3957)
* dp rewrite stage 0, projects

* remove plan files

* generalize handling of basepath for projects in project-factory module

* central-0 ---> core-0

* add schemas, validate YAMLs, tags

* aspect types

* data catalog policy tag factory

* add support for data catalog taxonomy to project factory

* complete retrofit of old stage configuration, except networking

* shared vpc networking

* networking

* data platform as pf dataset

* docs

* test

* remove legacy dp stage, fix tests and links

* boilerplate

* tfdoc

* fix unrelated tfdoc

* schemas

* fix errors

* schema

* duplicate schemas

* yamllint

* Fix module naming convention for aspect-types

* Fix factories_config in vpcs.tf for net-vpc-factory compatibility

* Update schema documentation based on schema changes

* Fix false rename conflict in .config.yaml files

* Sync schemas and update documentation

* Fix path expansion for aspect-types and revert projects_input to master

* Restore path expansion for org_policies in projects-iam call

* Fix trailing newlines in schema duplicates to satisfy duplicate-diff

* Fix path expansion for data_catalog_taxonomy in taxonomies.tf

* Update inventory for data-platform test and clean up debug prints

* Add full values to data-platform inventory

* Align Stage 2 VPC Factory integration with Stage 0 and fix tests

TAG=agy

* Fix project factory context resolution and data platform datasets

- Update tag context keys in project factory to use file key without 'projects/' prefix.
- Fix tag reference in product-0.yaml.
- Fix shared_vpc_service_config in shared-0.yaml by moving service account to network_users.
- Set parent for domain-0 folder to data-platform.
- Mock net-dev-0 project ID in tests.
- Update inventories.

TAG=agy
CONV=4b37fa5b-bf59-4604-9e8f-b55353d967a0

* Fix project-level tag keys context resolution in project factory

* Fix commented out tag reference in domain-0 .config.yaml

* Fix merge() calls with empty arguments in project-factory and data-catalog-policy-tag

* Update Data Platform dataset README with prerequisites and customization guide

* Add Table of Contents to Data Platform dataset README

* docs: update Data Platform README with project templates tip

* Document data platform output files and linking sequence in README

* Update data platform README with VPC-SC and delegated IAM details

* Refactor data platform dataset and align stage defaults

* Update test inventory and variables for data platform with new prefix
2026-05-12 16:44:32 +02:00
Luca Prete
04e64c4ae2 Move org policies lower in the project factory dependency chain to support extended context (#3937)
* module project-factory: include project in conditional_var context for org policies

* module project-factory: include project and folders in conditional_var context for org policies

- Move project org policies (explicit and factory) to projects-iam invocation.
- Move folder org policies (explicit and factory) to folder-X-iam invocations (levels 1-4).
- Inject folder_ids into projects-iam condition_vars and pass resolved folders.
- Update and regenerate test inventories (example.yaml, simple.yaml, hardened.yaml).

TAG=agy
CONV=e0f45850-ab01-4600-a2b6-4de62465c204

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2026-05-06 14:48:08 +02:00
Simon Roberts
26dbaa2d6e Enable terraform_naming_convention in tflint (#3930)
* Draft terraform_naming_convention

* Two fast/stages fixes for terraform_naming_convention

* Disable terraform_naming_convention for resources for now

* module fixes for terraform_naming_convention

* tfdoc

* Remove "moved" from recipe and needs-fixing

* Fix moved for spoke_ra

* fix tests

* Use default (snake_case) for resources

* factory.terraform_data.project-preconditions

* First-pass migration of resources + tests

* Fix tests/modules/organization

* Require snake_case for variables; Add annotations for _testing

* permit _fast_debug variable

* Fix net_vpc_factory and net_vpc_firewall tests

* tfdoc addons and recipe

* Fix more tests

* Fix some net-global -> net_global tests

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2026-05-06 06:06:26 +00:00
Julio Castillo
2eaa0d5e27 Add support for dynamic tags (#3897)
* Allow creation of dynamic tags

* Extend project factory and related modules to support dynamic values

* Extend folder and organization modules

* project and organization readme

* Simplify dynamic tag support and remove unnecessary restrictions

  • Schemas & Validations: Removed the restriction that forbade combining IAM fields with  allowed_values_regex  on tags. Updated validations in  project  and  organization  modules, and
  simplified all relevant JSON schemas.
  • Module Tag Bindings: Simplified the  tag_value  assignment in  folder ,  project ,  gcs ,  bigquery-dataset , and  kms  modules by removing the defensive  can(regex(...))  check and
  calling  templatestring  directly.
  • Outputs: Removed the  tags_dynamic  output from  project  and  organization  modules, as the same information is now available in  tag_keys .
  • Project Factory: Updated  tag_vars_projects  in  projects.tf  to use the native  namespaced_name  attribute and filtered manually for dynamic tags.

* fix(organization, project): fix linting and tests for dynamic tag support

- Align allowed_values_regex and description extraction in _tags_merged
  locals to use lookup() for consistency with other fields.
- Fix spacing in project context variable (alphabetical ordering).
- Update organization tags test to include the new cost_center tag key
  with allowed_values_regex.
- Update project tags test to include the new cost_center tag key and
  reflect the resolved allowed_values_regex on environment.

* refactor(gcs): refine tag bindings and fix context test

- Add _tag_bindings local to pre-resolve context references, enabling
  templatestring to receive a direct map reference (required by Terraform).
- Use var.context.tag_vars instead of the non-existent local.ctx.tag_vars.
- Fix HCL syntax in context.tfvars (escaped inner quotes).
- Update context test inventory to reflect 3 tag bindings including a
  dynamic value resolved via templatestring.

* refactor: align modules with tag binding context pattern

- Add _tag_bindings local + templatestring dance to cloud-run-v2,
  compute-vm, folder, kms modules (bigquery-dataset already had it)
- Exclude tag_vars from local.ctx in cloud-run-v2, compute-vm, folder,
  kms, project modules (bigquery-dataset already had it)
- Add tag_vars to context variable in cloud-run-v2, compute-vm modules
  (others already had it)
- Update all context tests with dynamic tag binding values using
  var.context.tag_vars

* docs: add module-level tftest.yaml test instructions to GEMINI.md

* docs: regenerate READMEs after tag-regex alignment

- Regenerate variable tables in 7 module READMEs to reflect
  line number shifts from prior tag-regex changes
- Add tag_vars exclusion to gcs ctx local
- Fix whitespace alignment in iam-service-account and
  project-factory tag_vars blocks
- Update tftest resource counts for organization and project
- Remove tags_dynamic from organization/project output tables

* fix(project-factory): update test inventory for tag_bindings module split

- Move tag binding address from folder-2 to folder-2-iam in test
  inventory (tag_bindings moved from creation to IAM modules)
- Update module instance count from 34 to 35
- Regenerate README tables after terraform fmt line shifts
- Apply terraform fmt to variables.tf

* refactor(project-factory): remove unnecessary depends_on from folder-iam modules

Folder IAM modules depend on their own folder creation modules, not
on module.projects. The explicit depends_on was leftover from an
earlier design.

* FAST stages

* Address review comments.

- FAST Stages:
  - Added tag_keys to output-files.tf in 0-org-setup to pass org tags via tfvars.
  - Sorted tag_keys and tag_values in output-files.tf.
  - Updated project-factory, networking, and security stages to use tag_keys.
  - Filtered tag_keys for dynamic tags only.
- Modules:
  - Excluded tag_vars from local.ctx in iam-service-account and organization.
  - Simplified tag_value in iam-service-account.
- Tests:
  - Updated test inventories for 0-org-setup and project-factory.

* Fix tf format

* Fix tfdoc

* docs: add ADR for templatestring vars convention and update status of base path ADR

* More tfdoc

* Update schemas

* Use endswith in context loop

* Address review

* Update FAST readmes

* Update last modules

* Terraform fmt

* Revert alloydb

* Fix whitespace

---------

Co-authored-by: Ludovico Magnocavallo <ludo@qix.it>
2026-04-24 20:45:45 +00:00
Liam Nesteroff
f5ee82ab3a Add DNS Armor support (#3874)
* add dns armor module

* add dns armor to pf

* added missing/optional attributes

* Update project schemas

* Set version file copyright year to 2025

* replace module with single resource

* moved into it's own file

* Added tests and defaulting enabled to false

* Add optional name parameter and updated schemas

* make dns_threat_detector.enabled optional in project schemas

---------

Co-authored-by: Luca Prete <preteluca@gmail.com>
2026-04-24 08:58:48 +02:00
Wiktor Niesiobędzki
6bfa7077a2 Recreate subscription of topic replacement, fix tests 2026-04-13 14:47:37 +02:00
Ludovico Magnocavallo
534fd4faf0 add support for service agents to project factory service accounts IAM (#3830) 2026-04-02 08:31:39 +00:00
Ludovico Magnocavallo
67b1543e90 Support additional attributes for buckets/datasets in project factory module (#3755)
* extend attributes for project factory secondary resources

* remove extra files

* complete

* tf fmt

* tfdoc

* schemas

* fix tests

* tfdoc
2026-02-20 11:57:59 +00:00
Ludovico Magnocavallo
bcca9e44ac Support project-level tag key/value contexts in project factory (#3714)
* cross-project tag context

* improve regression test

* add tag contexts to README contexts table
2026-02-09 08:54:29 +00:00
Ludovico Magnocavallo
fda89827a2 revert #3704 (#3713) 2026-02-07 11:08:25 +01:00
Vannick Trinquier
d499dc6928 Add support for bucket logging configuration in module gcs and project-factory (#3699) 2026-02-06 14:14:46 +07:00
lopezvit
97297d6065 fix(project-factory): Correctly interpolate IAM principals in tags (#3704)
* fix(project-factory): Correctly interpolate IAM principals in tags

Moves the processing of `tags` and `tag_bindings` from the `projects` module instance to the `projects-iam` instance.

This fixes a bug where IAM principals for automation service accounts, referenced via `$iam_principals:service_accounts/...`, were not being interpolated within `tags` IAM definitions. The `projects` module was called before the automation service account context was available, leading to the literal string being used instead of the service account email. Processing tags in the `projects-iam` module ensures the full context is available for interpolation.

Adds new tests for both the `project` and `project-factory` modules to validate the fix.

* fix(project-factory): Tag creation is now done in 2 steps.

1st step(projects): Creation of the tags without IAM bindings
2nd step(projects-iam): IAM bindings without creating the tags again
That way we are more backwards compatible as tags and tags values are back to be under  module.project-factory.module.projects["*"].google_tags_tag_*

* fix(modules/project-factory): introduce fix suggested by @ludoo, fix logs

* fix(modules/project-factory): fix linting

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2026-02-05 15:50:43 +00:00
Ludovico Magnocavallo
8e7253ba11 add missing IAM interface attributes to service account module (#3700) 2026-02-04 12:07:06 +01:00
Julio Castillo
d46b39b717 Add missing context interpolations (#3659) 2026-01-20 20:21:56 +01:00
Julio Castillo
d9e1b924a1 Add asset_feeds to resman modules (#3658)
* Add asset_feeds to resman modules

* Add examples and update readmes

* Extend pubsub_topic context to project and folder modules

* Use pubsub_topic context for pubsub_destination

* Update readmes and add project-factory asset_feed example

* Update context tests

* Update schemas
2026-01-20 14:37:35 +00:00
Ludovico Magnocavallo
88306fe99a Adding missing context replacement type to project factory README, add folder_ids to project condition vars (#3642)
* Adding missing context replacement type to project factory README

* add folder ids to project context condition vars
2026-01-12 14:41:07 +01:00
Ludovico Magnocavallo
fedf90d25f Add support for pubsub to project factory (#3608)
* add support for pubsub to project factory

* remove duplicate data access log definitions from folders

* tfdoc

* schemas

* fix example

* add pubsub topics context to org in stage 0
2025-12-23 10:24:33 +00:00
Ludovico Magnocavallo
26d43d8ec5 re-enable project billing association in project factory, extends to folder (#3554) 2025-11-27 20:51:20 +00:00
Ludovico Magnocavallo
897c6ef8c3 Add support for Workload Identity to project module and project factory (#3531)
* module-level support

* fast stage 0

* fix inventory, add outputs/tfvars

* wip

* project factory

* pf outputs

* iam templates will be added where ci/cd configs are managed

* fix merge conflicts
2025-11-17 07:31:21 +00:00
Ludovico Magnocavallo
ba77c6170c Allow configuring data access logs from org/folder/project schemas (#3516)
* modules and FAST support

* module tests

* fast stage 0 dataset

* tfdoc
2025-11-10 10:19:21 +00:00
Ludovico Magnocavallo
f9f015a692 Implement precondition check in project factory to ensure declared templates exist (#3493)
* pf template check

* tfdoc

* test inventories
2025-10-31 15:32:33 +00:00
Ludovico Magnocavallo
c765043c5c add the self project key to service account namespaces (#3490) 2025-10-30 16:42:13 +00:00
Ludovico Magnocavallo
56b213a047 add fourth folder level to project factory module (#3467) 2025-10-26 10:34:02 +01:00
Ludovico Magnocavallo
6fafdc8780 Merge remote-tracking branch 'origin/master' into fast-dev 2025-10-25 08:08:22 +00:00
Wiktor Niesiobędzki
f7c9a341b0 yamlint tests/ 2025-10-24 13:11:17 +02:00
Ludovico Magnocavallo
b0bc896a68 Allow null project id in service account module when reusing service account (#3452)
* allow null project id for service account reuse

* fix pf
2025-10-22 16:51:06 +00:00
Ludovico Magnocavallo
7ea9612b07 Allow skipping data source in service account module (#3450)
* test implementation

* wip

* service account reuse

* fix fast stage test

* revert cicd changes

* remove unused dep

* add comment on extra condition
2025-10-22 13:04:00 +02:00
Julio Castillo
48f6b4cd49 Add PAM support (#3438)
* PAM first pass

* Add factory and extend to organization

* Extend to project, add examples

* Add additionalProperties to all objects

* Fix boilerplate

* Expose pam_entitlements to project-factory

* Fix readme

* Move entitlements to second folder/project pass

* extend tests

* Fix readme

* Remove timeouts from inventories
2025-10-20 12:50:37 +00:00
Ludovico Magnocavallo
4113da128b Support resource-level factories config in project factory module and FAST stages (#3440)
* support resource-level factories in project factory module

* align folder and project schemas across modules and fast

* schema docs
2025-10-18 10:41:19 +00:00
Ludovico Magnocavallo
d0e2a54948 rationalize prefix handling for project factory automation resources (#3345) 2025-09-21 21:07:28 +02:00
Ludovico Magnocavallo
78966f66c3 Add support for project templates to project factory module (#3317)
* add support for project templates to project factory module

* align project factory features in FAST org setup stage
2025-09-11 09:56:01 +02:00
Ludovico Magnocavallo
86d50ffb62 New project factory improvements (#3303)
* Add separate prefix for automation resources in pf

* fix example

* add automation to pf outputs
2025-09-06 10:01:36 +02:00
Ludovico Magnocavallo
bc6950e205 Rename FAST stages preparing for eventual deprecation (#3298)
* renames

* links

* readme

* docs

* update pf modules tests for renames

* condition_vars context in modules

* data platform dataset

* fix links in stage 3 docs

* schema changes

* schema docs

* tfdoc

* update duplicates check

* fast legacy tests

* legacy schema

* fix tests
2025-09-04 08:24:11 +02:00
Ludovico Magnocavallo
3c38bc643e add KMS keys interpolation to project factory (#3145) 2025-06-10 10:24:25 +02:00
Ludovico Magnocavallo
7ce4381ac3 add support for service agent expansion to project factory IAM (#3112) 2025-05-24 12:33:20 +02:00
Luca Prete
13a236bca2 Project Factory: fix reference to automation SAs in IAM block for service accounts (#3100)
* PF: fix reference to automation SAs in IAM block in service accounts for PF

* add test

---------

Co-authored-by: Luca Prete <lucaprete@google.com>
Co-authored-by: Ludo <ludomagno@google.com>
2025-05-20 14:01:50 +02:00
Ludovico Magnocavallo
7cf936dc87 Enable context replacements for IAM principals in project factory module (#3094)
* enable context replacements for iam principals in pf module

* test changes
2025-05-19 11:57:26 +00:00
Ludovico Magnocavallo
717f89dc00 Rationalize project factory context interpolations for automation service accounts (#2959)
* use different keys for automation service accounts

* inventory

* improve error handling on automation buckets
2025-03-16 15:40:47 +00:00
Julio Castillo
50265f2169 Expose tags in project factory (#2929)
* Expose tags in project factory

* Update readme

* Remove default and update schema

* Fix tests
2025-02-24 23:12:17 +01:00
Ludovico Magnocavallo
5312767be4 Add provider output files to project factory stage, single automation bucket in module (#2914)
* single automation bucket, provider output files for project factory

* tfdoc

* fix outputs
2025-02-19 17:45:56 +00:00
Wiktor Niesiobędzki
58051e48e7 Fix tests 2025-02-19 16:47:15 +01:00
Wiktor Niesiobędzki
0a1b6c6e0f Add ability to refer to other project service accounts in Project Factory 2025-02-19 16:47:15 +01:00
Ludovico Magnocavallo
1a4b298cc9 Project factory additions, project module reuse implementation (#2899)
* add support for buckets

* add project-level interpolation for own SAs

* docs

* project reuse changes

* fix example

* tfdoc

* update check documentation tool

* fast tests

* blueprints

* typo
2025-02-15 20:37:45 +01:00
Julio Castillo
d0c3f04c10 Make service agents work in different universes (#2894)
* Make service agents work in different universes

* Use templatestring and two passes for service agent emails

* Fix tests
2025-02-14 12:16:07 +00:00
Julio Castillo
8a2320311d Add bucket_create to modules/gcs (#2827)
* Add `bucket_create` to `modules/gcs`

* Create local with bucket name

* Update variable description

* Fix bucket output

* Fix tests

* Fix tests

* Bump OpenTofu to 1.9.0 (needed for multi-var validations)
2025-01-21 23:48:36 +01:00
Wiktor Niesiobędzki
44a4268afc Fix E2E tests 2024-09-04 12:16:50 +02:00
Julio Castillo
f57635d044 Add managed folders suports to gcs module (#2530)
* Add RPO, make versioning dynamic

* Add manaed folders

* Change autoclass and cors defaults to null

* Update README

* Add iam_by_principals

* Add managed folders var description

* Remove need for managed folders to end in /

* Add inventory to example

* Update readme

* Fix FAST tests
2024-08-28 07:30:52 +00:00
Ludovico Magnocavallo
ad5de9b7ea Refactor FAST project factory and supporting documentation (#2505)
* untested

* teams pattern

* rework doc

* README

* boierplate

* tflint

* Fix tflint for project factory

* Correct path to pf

* resman changes

* fix factory variable default

* fix links

* project factory module substitutions

* tflint

* stage test

* tfdoc

* rename schema, address review comments

* README typos and wording

* tfdoc

* review comments

* remove test from yaml

* revert output workflow changes

* fix sa reference errors

* tfdoc

* pf tag roles

* schema validation

* pf tag roles

* avoid null values in pf context

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
2024-08-20 16:45:42 +00:00
Julio Castillo
0420dec32f Add deletion_policy to project module (#2502)
* Replace skip_delete with deletion_policy

* Fix blueprints

* Update apigee blueprint readme

* Remove skip_delete from test inventories
2024-08-16 18:33:39 +02:00