200 Commits

Author SHA1 Message Date
Julio Castillo
e3e261442f Add skip_iam option to project service agents config (#4005) 2026-06-01 04:36:07 +00:00
Simone Ruffilli
1594a01c6f Cosmetic and linter fixes (#3981) 2026-05-22 08:28:01 +00:00
kovagoadam
1907c38e22 Add IAM deny policies support (#3970)
* Added IAM denial policies

* Moved default to empty, removed trys, added condition vars to expression

* remove redundant null checks

* reduce line length

* boilerplate and principal context expansion

* update readmes

* add explicit validation against null values

* add context tests

* Add missing license headers to examples

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2026-05-21 02:38:06 +00:00
Julio Castillo
2eaa0d5e27 Add support for dynamic tags (#3897)
* Allow creation of dynamic tags

* Extend project factory and related modules to support dynamic values

* Extend folder and organization modules

* project and organization readme

* Simplify dynamic tag support and remove unnecessary restrictions

  • Schemas & Validations: Removed the restriction that forbade combining IAM fields with  allowed_values_regex  on tags. Updated validations in  project  and  organization  modules, and
  simplified all relevant JSON schemas.
  • Module Tag Bindings: Simplified the  tag_value  assignment in  folder ,  project ,  gcs ,  bigquery-dataset , and  kms  modules by removing the defensive  can(regex(...))  check and
  calling  templatestring  directly.
  • Outputs: Removed the  tags_dynamic  output from  project  and  organization  modules, as the same information is now available in  tag_keys .
  • Project Factory: Updated  tag_vars_projects  in  projects.tf  to use the native  namespaced_name  attribute and filtered manually for dynamic tags.

* fix(organization, project): fix linting and tests for dynamic tag support

- Align allowed_values_regex and description extraction in _tags_merged
  locals to use lookup() for consistency with other fields.
- Fix spacing in project context variable (alphabetical ordering).
- Update organization tags test to include the new cost_center tag key
  with allowed_values_regex.
- Update project tags test to include the new cost_center tag key and
  reflect the resolved allowed_values_regex on environment.

* refactor(gcs): refine tag bindings and fix context test

- Add _tag_bindings local to pre-resolve context references, enabling
  templatestring to receive a direct map reference (required by Terraform).
- Use var.context.tag_vars instead of the non-existent local.ctx.tag_vars.
- Fix HCL syntax in context.tfvars (escaped inner quotes).
- Update context test inventory to reflect 3 tag bindings including a
  dynamic value resolved via templatestring.

* refactor: align modules with tag binding context pattern

- Add _tag_bindings local + templatestring dance to cloud-run-v2,
  compute-vm, folder, kms modules (bigquery-dataset already had it)
- Exclude tag_vars from local.ctx in cloud-run-v2, compute-vm, folder,
  kms, project modules (bigquery-dataset already had it)
- Add tag_vars to context variable in cloud-run-v2, compute-vm modules
  (others already had it)
- Update all context tests with dynamic tag binding values using
  var.context.tag_vars

* docs: add module-level tftest.yaml test instructions to GEMINI.md

* docs: regenerate READMEs after tag-regex alignment

- Regenerate variable tables in 7 module READMEs to reflect
  line number shifts from prior tag-regex changes
- Add tag_vars exclusion to gcs ctx local
- Fix whitespace alignment in iam-service-account and
  project-factory tag_vars blocks
- Update tftest resource counts for organization and project
- Remove tags_dynamic from organization/project output tables

* fix(project-factory): update test inventory for tag_bindings module split

- Move tag binding address from folder-2 to folder-2-iam in test
  inventory (tag_bindings moved from creation to IAM modules)
- Update module instance count from 34 to 35
- Regenerate README tables after terraform fmt line shifts
- Apply terraform fmt to variables.tf

* refactor(project-factory): remove unnecessary depends_on from folder-iam modules

Folder IAM modules depend on their own folder creation modules, not
on module.projects. The explicit depends_on was leftover from an
earlier design.

* FAST stages

* Address review comments.

- FAST Stages:
  - Added tag_keys to output-files.tf in 0-org-setup to pass org tags via tfvars.
  - Sorted tag_keys and tag_values in output-files.tf.
  - Updated project-factory, networking, and security stages to use tag_keys.
  - Filtered tag_keys for dynamic tags only.
- Modules:
  - Excluded tag_vars from local.ctx in iam-service-account and organization.
  - Simplified tag_value in iam-service-account.
- Tests:
  - Updated test inventories for 0-org-setup and project-factory.

* Fix tf format

* Fix tfdoc

* docs: add ADR for templatestring vars convention and update status of base path ADR

* More tfdoc

* Update schemas

* Use endswith in context loop

* Address review

* Update FAST readmes

* Update last modules

* Terraform fmt

* Revert alloydb

* Fix whitespace

---------

Co-authored-by: Ludovico Magnocavallo <ludo@qix.it>
2026-04-24 20:45:45 +00:00
Julio Castillo
bc5b203a8f Add toggle to tfdoc to remove type hints (#3842)
* Add toggle to tfdoc to remove type hints

* Update all readmes

* Fix python formatting
2026-04-08 11:08:18 +00:00
Ludovico Magnocavallo
a4eb4d24fd Compute VM module refactor (#3805)
* add ad for compute-vm refactor

* Exclue nic_type from validated fields, add split of main.tf and template.tf

* boot disk

* fix examples and fixtures

* attached disks

* fix further examples and module-level tests

* remove extra file

* fix mig examples

* finish refactoring variables

* align fast and other modules

* refactor(compute-vm): align examples and ADR with the newly implemented interface

This commit addresses the remaining references of the `instance_type` and `confidential_compute` parameters in the testing environment and updates the ADR.

* feat(compute-vm): add network_performance_config to instance and templates

This change implements the usage of the `network_performance_tier` variable we added earlier into the actual Terraform resources.

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
2026-03-26 11:31:40 +00:00
jnahelou
78e00682f8 Ignored condition attribute in IAM tag bindings within the organisation/project modules (#3762)
* fix(modules/organization): conditions ignored in tags

* fix(modules/project): conditions ignored in tags

* fix(modules/project): Tags:1 test skipped due to bad markdown block

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2026-03-18 17:12:46 +00:00
Ludovico Magnocavallo
fb21f6aaf8 Change factories_config type in FAST and project/vpc factory modules, add YAML schema validation (#3728)
* stage 0

* stage 1

* networking

* security

* pf stage

* tfdoc

* align schemas

* inventory

* fix observability

* pf module

* pf module budgets

* align fast stages

* align project subfactories

* tfdoc

* schema validation

* add missing schemas

* Fix observability types

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2026-02-11 15:29:49 +00:00
Julio Castillo
7d33becacf Add asset_search to folder, project, and organization modules. (#3707) 2026-02-05 12:56:27 +01:00
Samuele Perticarari
b2cbd7a3c6 fix: Fixed external broken links (#3703) 2026-02-04 14:35:37 +01:00
Vannick Trinquier
2af44b0651 Add support for security command center mute rules in module organization, folder and project (#3694) 2026-02-04 08:31:05 +07:00
Julio Castillo
8dde9fe773 Fix domains of default service accounts when universe is present (#3670) 2026-01-23 14:11:17 +00:00
Julio Castillo
d9e1b924a1 Add asset_feeds to resman modules (#3658)
* Add asset_feeds to resman modules

* Add examples and update readmes

* Extend pubsub_topic context to project and folder modules

* Use pubsub_topic context for pubsub_destination

* Update readmes and add project-factory asset_feed example

* Update context tests

* Update schemas
2026-01-20 14:37:35 +00:00
Julio Castillo
cff8a25c59 Introduce iam_by_principals_conditional (#3649)
* Introduce iam_by_principals_conditional

* Add iam_by_principals_conditional to project factory

* Update IAM ADR

* Update project factory readme

* Sync FAST schemas

* Update organization schema

* Add resman tests for iam_by_principals_conditional

* Update PF project-defaults.tf

* Update copyright
2026-01-14 11:16:07 +00:00
Eric Zhao
c1248d328a Allow any VPC for (secure) network_tags (#3634)
* feat: allow all for VPC networks

* feat: add examples

* feat: add header

* feat: module test

* fix: update network testing data to pass validation

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2026-01-12 09:34:18 +00:00
Vannick Trinquier
cc24046be8 Add CMEK support to FAST and controls for CMEK encryption (#3556) 2025-12-14 12:14:08 +07:00
Ludovico Magnocavallo
ab0f55216a Add support for descriptive name to projects (#3591)
* add support for descriptive name to projects

* boilerplate

* fmt
2025-12-12 09:06:47 +01:00
Ludovico Magnocavallo
216a12eae5 Fix CI/CD dataset files and provider workflow variable in FAST stage 0 (#3587)
* fix CI/CD dataset files and provider workflow variable

* IAM principals use pool, sts uses provider

* tfdoc

* fix variable description
2025-12-11 15:05:39 +01:00
Ludovico Magnocavallo
a8384b85d1 Auto-grant editor role for cloudservices in project module, expand project ids context in project factory module (#3552)
* service agent editor role

* add internal project ids to context replacement for projects in project factory module
2025-11-27 12:45:52 +00:00
Ludovico Magnocavallo
10e29e1eeb Context improvements: "all service accounts" principal in folder, org, project modules; custom roles in factory condition vars for FAST stage 0 (#3548)
* iam principalsets

* fix folder

* add custom roles to factory condition vars in stage 0

* project shared vpc IAM
2025-11-24 08:28:41 +00:00
Julio Castillo
3959bb3974 Fix aprover -> approver (#3540)
* Fix aprover -> approver

* Once again...
2025-11-19 08:50:23 +01:00
Ludovico Magnocavallo
897c6ef8c3 Add support for Workload Identity to project module and project factory (#3531)
* module-level support

* fast stage 0

* fix inventory, add outputs/tfvars

* wip

* project factory

* pf outputs

* iam templates will be added where ci/cd configs are managed

* fix merge conflicts
2025-11-17 07:31:21 +00:00
Ludovico Magnocavallo
fc7aa71ada Add support for KMS key creation to project factory (#3518)
* initial implementation

* context

* tfdoc

* add support for autokey to projects

* fix typo
2025-11-11 07:23:50 +01:00
Ludovico Magnocavallo
ba77c6170c Allow configuring data access logs from org/folder/project schemas (#3516)
* modules and FAST support

* module tests

* fast stage 0 dataset

* tfdoc
2025-11-10 10:19:21 +00:00
Ludovico Magnocavallo
a4ff84838a Merge remote-tracking branch 'origin/master' into fast-dev 2025-11-05 06:53:32 +00:00
Wiktor Niesiobędzki
d5bc59a238 Fix E2E tests.
* Disable tests for VPC connector and Cloud Functions, CFs are not
  supporrted in the default region
* fix permissions to secrets for Cloud Run
* add permissions admin permissions to any SA within project to
  `var.bucket`
* add permissions to access the secret to any SA within project to
  secrets created by fixture
* disable custom roles in E2E tests, as `var.organization_id` is not the
  same org, within which projects are created in E2E
2025-11-04 10:23:55 +01:00
lcaggio
64632eb957 BigQuery reservation (#3441)
* First Commit.

* Improve

* Fix README

* Fix lint

* fix

* Fix Tests.

* Improve output

* fix yaml lint

* Fix

* Fic test

* Fix

* Fix output
2025-11-03 11:36:42 +01:00
Ludovico Magnocavallo
0a2cc758ac Essential contacts in schemas, and email context substitutions (#3495)
* modules

* fast

* duplicate diff

* fix contacts in FAST stage 0 datasets, update contacts in YAML schemas
2025-11-03 08:53:29 +01:00
Vannick Trinquier
c60ae3652a Adding hardened datasets for preventive and detective Compliance Controls (#3410)
* Adding hardened datasets for preventive and detective Compliance Controls in stage 0 and stage 1 VPC-SC

* Move observability to factory file

* Update documentation

* Update local variable for use

* Update observability factory to use other module

* Add raw diagram file for hardened datasets

* Retrofit change

* Rename log_buckets context variable to be consistent across modules

* Update stage 0 documentation to mention hardened dataset

* Update customer ids list

* Update documentation, path to schema add ID to access level

* Comment organization policy gcp.resourceLocation by default

* Prevent duplicate key error by merging principal roles

* Adding ngfw roles files in hardened datasets

* Update script to validate files differences to support folder and datasets

* Format duplicate-diff python script

* Remove .config.yaml from duplicates

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2025-10-21 10:34:25 +00:00
Ludovico Magnocavallo
ec09414823 Merge remote-tracking branch 'origin/master' into fast-dev 2025-10-20 13:58:44 +00:00
Julio Castillo
48f6b4cd49 Add PAM support (#3438)
* PAM first pass

* Add factory and extend to organization

* Extend to project, add examples

* Add additionalProperties to all objects

* Fix boilerplate

* Expose pam_entitlements to project-factory

* Fix readme

* Move entitlements to second folder/project pass

* extend tests

* Fix readme

* Remove timeouts from inventories
2025-10-20 12:50:37 +00:00
Ludovico Magnocavallo
cc947777ce Merge remote-tracking branch 'origin/master' into fast-dev 2025-10-16 05:57:02 +00:00
Wiktor Niesiobędzki
9f2beb1ee6 Use fixture custom role for modules/project tests 2025-10-13 18:58:52 +02:00
Wiktor Niesiobędzki
849e856437 Use email from variables in tests 2025-10-13 18:58:52 +02:00
Julio Castillo
3498338c87 Support iam_by_principals_additive in 0-org-setup (#3387)
* Support iam_by_principals_additive in 0-org-setup

* Fix typo
2025-10-07 06:56:35 +00:00
Ludovico Magnocavallo
07d7a85da1 Lightly refactor service agents locals in project module (#3380)
* lightly refactor project module service agent locals

* remove debug block

* tfdoc

* revert templatestring to format for universe kms

* reinstate moved block
2025-10-05 10:25:04 +00:00
Ludovico Magnocavallo
4db29a48eb allow forcing jit service agents generation for universe (#3378) 2025-10-05 10:17:57 +02:00
Vannick Trinquier
cfe2e21ce7 feat: add support for SCC Custom Security Health Analytics module in … (#3372)
* feat: add support for SCC Custom Security Health Analytics module in organization, folder and project modules

* fix: update description and docs

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2025-10-03 13:47:50 +02:00
Ludovico Magnocavallo
02b37c5958 Fix alert variable notification prompts type in project module (#3330) 2025-09-17 08:59:18 +02:00
Ludovico Magnocavallo
bc6950e205 Rename FAST stages preparing for eventual deprecation (#3298)
* renames

* links

* readme

* docs

* update pf modules tests for renames

* condition_vars context in modules

* data platform dataset

* fix links in stage 3 docs

* schema changes

* schema docs

* tfdoc

* update duplicates check

* fast legacy tests

* legacy schema

* fix tests
2025-09-04 08:24:11 +02:00
Ludovico Magnocavallo
36648b6b63 FAST light implementation (#3255)
* data wip

* wip data

* update org schema, add note on expansion

* all schemas, workload notes

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* wip

* data wip

* wip

* wip

* wip

* wip

* org module IAM context (using lookup)

* new-style context expansion in project IAM

* remove spurious file

* project module contexts

* finalize context replacement format for project module

* revert org module changes

* fix tag id interpolation in project

* fix tag id interpolation in project

* organization module context

* organization context test

* context expansion for folder tag bindings

* test context expansion for tag bindings

* service account module context

* simplify context local

* context for iam service account

* nuke blueprints

* remove links to blueprints

* vpc sc context in project module

* Add context to GCS module

* Add inline deps to plan_summary script

* Make context a top-level variable for folder, organization, sa

* Add add context top-level to VPC-SC

* move context out of factories_config variable

* tfdoc

* fix merge

* fix merge

* fix examples

* net-vpc module context

* add parent ids to folder context

* rename folder parent context

* fix folder parent check

* new project factory stub

* wip

* wip

* refactor defaults

* project iam

* bueckts and service accounts

* start adding context replacements

* better test data

* automation resources for folders and projects

* automation

* add support for project id interpolation

* first tested apply

* improve IAM description in gcs module

* add context to billing account module

* add notification channels to billing account module context

* add billing budgets to new pf

* schemas and defaults

* bootstrap wip

* bootstrap wip

* bootstrap wip

* pf outputs

* pf fixes

* fix pf sample data

* bootstrap lite fixes

* add locations to organization module contexts

* bootstrap lite fixes

* org fixes, billing accounts

* fix default project parent

* bootstrap lite wip

* add locations to gcs module context

* add context support to logging bucket module

* add context to pubsub module

* split out iam variables in gcs module

* fix logging bucket context test

* bootstrap log sink destinations

* streamline logging-bucket module variables

* fix logging bucket context test

* align logging bucket module interface in fast bootstrap

* add support for project-level log buckets to project factory

* support full context expansion in organization module log sinks

* log buckets in fast-lite bootstrap

* make og sink type optional in organization module

* log sinks in fast-lite bootstrap

* set tag values in factory context

* bootstrap lite data

* output files schema

* billing account schema

* output files

* output providers

* gcs output files

* boilerplate

* tflint

* check documentation

* check docs

* fix project module parent variable validation

* fix log bucket examples

* allow null parent in project module

* silence folder test errors

* fix billing account sink example

* fix project example

* fix billing account module

* fix folder tests

* fix FAST

* fix fast

* tfvars outputs

* wif

* cicd service accounts

* cicd

* allow defaults in context, minimal org policies

* support gcs managed folders in project factory and bootstrap lite

* support prefix in provider output files

* rename bootstrap stage

* gitignore

* gitignore

* security folder, billing IAM

* wip tfvars

* fix typo

* security IAM

* control tag iam/context via variables in organization module

* split tag creation from tag IAM to avoid circular refs

* port organization module tag changes to project module

* implement new-style context expansion in vpc-sc module

* fix fast vpc-sc tests

* boilerplate

* vpc sc stage

* schemas

* fast-lite compatibility for vpc sc stage

* make log project number optional in vpc-sc stage

* networking

* networking

* networking

* networking

* rename and move new stage under fast

* clone pf tests

* use context replacement for internal notification channels in billing account module

* support service agents in project module iam context replacements

* support service agents in project module iam context replacements

* add support for kms keys to project module context

* experimental pf example test and fixes

* fix schemas

* fix tests

* tfdoc

* tfdoc

* pf config

* experimental pf

* remove redundant dot from gcs managed folder IAM keys

* bootstrap experimental test

* project factory exp stage test

* skip tflint for bootstrap experimental test

* tflint

* fix gcs test

* documentation work

* documentation work

* Update README.md

* tfdoc

* tfdoc

* readme

* tfdoc

* readme

* readme

* readme

* readme

* support universe in pf exp projects

* missing universe service agents

* org policies import, non-admin billing IAM

* todo

* fix test

* custom constraints

* fast classic dataset

* fix test data

* context replacements in billing module log sinks

* fix typo

* add support for billing log sinks

* update docs

* readme

* cicd fix and test

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2025-09-02 08:38:57 +02:00
Ludovico Magnocavallo
53f691320e Support IAM tag factory context expansion in organization / project modules and FAST resman stage (#3226)
* tag factory context

* support tag factory contexts in resman

* schemas

* fix schema
2025-07-09 09:04:17 +02:00
Liam Nesteroff
f07e4f64e9 Standardise reuse variable from project module and implement for net-vpc (#3205)
* exposing the network_id from the net-vpc module for use with tag bindings

* convert vpc_create to vpc_reuse

* Changed the reuse vars to standardised attributes

* fixed readme tests and schemas

* modified apigee blueprint to use vpc_reuse with network_id passthrough

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2025-07-03 09:12:05 +10:00
Liam Nesteroff
1fbb2cb330 Added tag factory option for organization module (#3178)
* Added tag factory option for organization module

* added tags-factory tests

* added tag factory for project module

* missing header

* added lookup catch for org tag values and fixed incorrect link in doco

* fixed factory locals from copy/paste

* added full doco/tests for project tags factory. fixed failed test looking for ID in yamls

* added context option for factories_config to configure existing K/Vs

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2025-06-23 16:24:43 +10:00
Ludovico Magnocavallo
fe0a8128dc Allow custom roles in context, add support for shared VPC IAM to project and project factory (#3163)
* add support for custom roles and hos tproject iam to project modules

* align vpc factory
2025-06-15 10:01:22 +02:00
jnahelou
d6004f0cd0 CMEK service agents mapping breaks Composer v2 backward compatibility (#3156)
* allow backward compatibility for unmapped service agents

* docs(modules/project): provide a fallback mechanism in cmek for users to specify additional service agents when needed

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2025-06-12 20:27:41 +02:00
apichick
077d8719dd Added default compute network tier to project module (#3150) 2025-06-10 21:44:38 +00:00
Ludovico Magnocavallo
cb7bed50e3 New FAST data platform (#3066)
* copy from broken dp dev branch

* remove extra excalidraw file

* fix networking yaml

* tfdoc

* tfdoc

* nuke old data platform

* fix tests

* tests

* tflint

* high level diagram

* make location optional in composer schema

* add composer outputs

* docs

* remove schema docs

* tfdoc

* update service agent encryption composer def for composer 3

* encryption keys

* typo

* typo

* fix security IAM

* inventory

* tflint

* Fix roles and diagram.

* Fix tflint

* Fix test DP.

* Fix test

* Diagrams excalidraw gz

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
2025-05-03 21:21:38 +00:00
Wiktor Niesiobędzki
6af479706d Fix failing E2E test 2025-02-28 18:45:14 +01:00
Wiktor Niesiobędzki
0a1b6c6e0f Add ability to refer to other project service accounts in Project Factory 2025-02-19 16:47:15 +01:00