Files
hunfabric/modules/project-factory/main.tf
Ludovico Magnocavallo 36648b6b63 FAST light implementation (#3255)
* data wip

* wip data

* update org schema, add note on expansion

* all schemas, workload notes

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* wip

* data wip

* wip

* wip

* wip

* wip

* org module IAM context (using lookup)

* new-style context expansion in project IAM

* remove spurious file

* project module contexts

* finalize context replacement format for project module

* revert org module changes

* fix tag id interpolation in project

* fix tag id interpolation in project

* organization module context

* organization context test

* context expansion for folder tag bindings

* test context expansion for tag bindings

* service account module context

* simplify context local

* context for iam service account

* nuke blueprints

* remove links to blueprints

* vpc sc context in project module

* Add context to GCS module

* Add inline deps to plan_summary script

* Make context a top-level variable for folder, organization, sa

* Add add context top-level to VPC-SC

* move context out of factories_config variable

* tfdoc

* fix merge

* fix merge

* fix examples

* net-vpc module context

* add parent ids to folder context

* rename folder parent context

* fix folder parent check

* new project factory stub

* wip

* wip

* refactor defaults

* project iam

* bueckts and service accounts

* start adding context replacements

* better test data

* automation resources for folders and projects

* automation

* add support for project id interpolation

* first tested apply

* improve IAM description in gcs module

* add context to billing account module

* add notification channels to billing account module context

* add billing budgets to new pf

* schemas and defaults

* bootstrap wip

* bootstrap wip

* bootstrap wip

* pf outputs

* pf fixes

* fix pf sample data

* bootstrap lite fixes

* add locations to organization module contexts

* bootstrap lite fixes

* org fixes, billing accounts

* fix default project parent

* bootstrap lite wip

* add locations to gcs module context

* add context support to logging bucket module

* add context to pubsub module

* split out iam variables in gcs module

* fix logging bucket context test

* bootstrap log sink destinations

* streamline logging-bucket module variables

* fix logging bucket context test

* align logging bucket module interface in fast bootstrap

* add support for project-level log buckets to project factory

* support full context expansion in organization module log sinks

* log buckets in fast-lite bootstrap

* make og sink type optional in organization module

* log sinks in fast-lite bootstrap

* set tag values in factory context

* bootstrap lite data

* output files schema

* billing account schema

* output files

* output providers

* gcs output files

* boilerplate

* tflint

* check documentation

* check docs

* fix project module parent variable validation

* fix log bucket examples

* allow null parent in project module

* silence folder test errors

* fix billing account sink example

* fix project example

* fix billing account module

* fix folder tests

* fix FAST

* fix fast

* tfvars outputs

* wif

* cicd service accounts

* cicd

* allow defaults in context, minimal org policies

* support gcs managed folders in project factory and bootstrap lite

* support prefix in provider output files

* rename bootstrap stage

* gitignore

* gitignore

* security folder, billing IAM

* wip tfvars

* fix typo

* security IAM

* control tag iam/context via variables in organization module

* split tag creation from tag IAM to avoid circular refs

* port organization module tag changes to project module

* implement new-style context expansion in vpc-sc module

* fix fast vpc-sc tests

* boilerplate

* vpc sc stage

* schemas

* fast-lite compatibility for vpc sc stage

* make log project number optional in vpc-sc stage

* networking

* networking

* networking

* networking

* rename and move new stage under fast

* clone pf tests

* use context replacement for internal notification channels in billing account module

* support service agents in project module iam context replacements

* support service agents in project module iam context replacements

* add support for kms keys to project module context

* experimental pf example test and fixes

* fix schemas

* fix tests

* tfdoc

* tfdoc

* pf config

* experimental pf

* remove redundant dot from gcs managed folder IAM keys

* bootstrap experimental test

* project factory exp stage test

* skip tflint for bootstrap experimental test

* tflint

* fix gcs test

* documentation work

* documentation work

* Update README.md

* tfdoc

* tfdoc

* readme

* tfdoc

* readme

* readme

* readme

* readme

* support universe in pf exp projects

* missing universe service agents

* org policies import, non-admin billing IAM

* todo

* fix test

* custom constraints

* fast classic dataset

* fix test data

* context replacements in billing module log sinks

* fix typo

* add support for billing log sinks

* update docs

* readme

* cicd fix and test

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2025-09-02 08:38:57 +02:00

491 lines
19 KiB
HCL

/**
* Copyright 2025 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
# tfdoc:file:description Projects and billing budgets factory resources.
locals {
_service_agent_emails = flatten([
for k, v in module.projects : [
for kk, vv in v.service_agents : {
key = "${k}/${kk}"
value = "serviceAccount:${vv.email}"
}
]
])
context = {
folder_ids = merge(
var.factories_config.context.folder_ids,
local.hierarchy
)
iam_principals = merge(
var.factories_config.context.iam_principals,
{
for k, v in module.automation-service-accounts :
k => v.iam_email
},
# module.service-accounts are excluded here, as adding them here results in dependency cycles
)
}
service_accounts_names = {
for k, v in module.service-accounts : k => v.name
}
service_agents_email = {
for v in local._service_agent_emails : v.key => v.value
}
}
module "projects" {
source = "../project"
for_each = local.projects
billing_account = each.value.billing_account
deletion_policy = each.value.deletion_policy
name = each.value.name
parent = lookup(
local.context.folder_ids, each.value.parent, each.value.parent
)
prefix = each.value.prefix
project_reuse = each.value.project_reuse
alerts = try(each.value.alerts, null)
auto_create_network = try(each.value.auto_create_network, false)
compute_metadata = try(each.value.compute_metadata, {})
# TODO: concat lists for each key
contacts = merge(
each.value.contacts, var.data_merges.contacts
)
default_service_account = try(each.value.default_service_account, "keep")
descriptive_name = try(each.value.descriptive_name, null)
factories_config = {
custom_roles = each.value.factories_config.custom_roles
observability = each.value.factories_config.observability
org_policies = each.value.factories_config.org_policies
quotas = each.value.factories_config.quotas
context = {
notification_channels = var.factories_config.context.notification_channels
}
}
labels = merge(
each.value.labels, var.data_merges.labels
)
lien_reason = try(each.value.lien_reason, null)
log_scopes = try(each.value.log_scopes, null)
logging_data_access = try(each.value.logging_data_access, {})
logging_exclusions = try(each.value.logging_exclusions, {})
logging_metrics = try(each.value.logging_metrics, null)
logging_sinks = try(each.value.logging_sinks, {})
metric_scopes = distinct(concat(
each.value.metric_scopes, var.data_merges.metric_scopes
))
notification_channels = try(each.value.notification_channels, null)
org_policies = each.value.org_policies
service_encryption_key_ids = {
for k, v in merge(
each.value.service_encryption_key_ids,
var.data_merges.service_encryption_key_ids
) : k => [
for key in v : lookup(var.factories_config.context.kms_keys, key, key)
]
}
services = distinct(concat(
each.value.services,
var.data_merges.services
))
shared_vpc_host_config = each.value.shared_vpc_host_config
tag_bindings = {
for k, v in merge(each.value.tag_bindings, var.data_merges.tag_bindings) :
k => lookup(var.factories_config.context.tag_values, v, v)
}
tags = each.value.tags
vpc_sc = each.value.vpc_sc == null ? null : {
perimeter_name = (
each.value.vpc_sc.perimeter_name == null
? null
: lookup(
var.factories_config.context.perimeters,
each.value.vpc_sc.perimeter_name,
each.value.vpc_sc.perimeter_name
)
)
is_dry_run = each.value.vpc_sc.is_dry_run
}
quotas = each.value.quotas
}
module "projects-iam" {
source = "../project"
for_each = local.projects
name = module.projects[each.key].project_id
project_reuse = {
use_data_source = false
attributes = {
name = module.projects[each.key].name
number = module.projects[each.key].number
services_enabled = module.projects[each.key].services
}
}
iam = {
for k, v in lookup(each.value, "iam", {}) :
lookup(var.factories_config.context.custom_roles, k, k) => [
for vv in v : try(
# project service accounts (sa)
module.service-accounts["${each.key}/${vv}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.key}/automation/${vv}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.key}/${vv}"],
# other projects service accounts (project/sa)
module.service-accounts[vv].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[vv],
# project's service identities
local.service_agents_email["${each.key}/${vv}"],
local.service_agents_email[vv],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(vv, ":")
? templatestring(
vv, { project_number = module.projects[each.key].number }
)
: tonumber("[Error] Invalid member: '${vv}' in project '${each.key}'")
)
)
]
}
iam_bindings = {
for k, v in lookup(each.value, "iam_bindings", {}) : k => merge(v, {
members = [
for vv in v.members : try(
# project service accounts (sa)
module.service-accounts["${each.key}/${vv}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.key}/automation/${vv}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.key}/${vv}"],
# other projects service accounts (project/sa)
module.service-accounts[vv].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[vv],
# project's service identities
local.service_agents_email["${each.key}/${vv}"],
local.service_agents_email[vv],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(vv, ":")
? templatestring(
vv, { project_number = module.projects[each.key].number }
)
: tonumber("[Error] Invalid member: '${vv}' in project '${each.key}'")
)
)
]
role = lookup(var.factories_config.context.custom_roles, v.role, v.role)
})
}
iam_bindings_additive = {
for k, v in lookup(each.value, "iam_bindings_additive", {}) : k => merge(v, {
member = try(
# project service accounts (sa)
module.service-accounts["${each.key}/${v.member}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.key}/automation/${v.member}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.key}/${v.member}"],
# other projects service accounts (project/sa)
module.service-accounts[v.member].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[v.member],
# project's service identities
local.service_agents_email["${each.key}/${v.member}"],
local.service_agents_email[v.member],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(v.member, ":")
? templatestring(
v.member, { project_number = module.projects[each.key].number }
)
: tonumber("[Error] Invalid member: '${v.member}' in project '${each.key}'")
)
)
role = lookup(var.factories_config.context.custom_roles, v.role, v.role)
})
}
# IAM by principals would trigger dynamic key errors so we don't interpolate
# iam_by_principals = try(each.value.iam_by_principals, {})
iam_by_principals = {
for k, v in try(each.value.iam_by_principals, {}) :
try(
# project service accounts (sa)
module.service-accounts["${each.key}/${k}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.key}/automation/${k}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.key}/${k}"],
# other projects service accounts (project/sa)
module.service-accounts[k].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[k],
# project's service identities
local.service_agents_email["${each.key}/${k}"],
local.service_agents_email[k],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(k, ":")
? templatestring(
k, { project_number = module.projects[each.key].number }
)
: tonumber("[Error] Invalid member: '${k}' in project '${each.key}'")
)
) => [
for vv in v : lookup(var.factories_config.context.custom_roles, vv, vv)
]
}
# Shared VPC configuration is done at stage 2, to avoid dependency cycle between project service accounts and
# IAM grants done for those service accounts
shared_vpc_service_config = (
try(each.value.shared_vpc_service_config.host_project, null) == null
? null
: merge(each.value.shared_vpc_service_config, {
host_project = try(
var.factories_config.context.vpc_host_projects[each.value.shared_vpc_service_config.host_project],
module.projects[each.value.shared_vpc_service_config.host_project].project_id,
each.value.shared_vpc_service_config.host_project
)
iam_bindings_additive = {
for k, v in try(each.value.shared_vpc_service_config.iam_bindings_additive, {}) : k => merge(v, {
member = try(
# project service accounts (sa)
module.service-accounts["${each.key}/${v.member}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.key}/automation/${v.member}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.key}/${v.member}"],
# other projects service accounts (project/sa)
module.service-accounts[v.member].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[v.member],
# project's service identities
local.service_agents_email["${each.key}/${v.member}"],
local.service_agents_email[v.member],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(v.member, ":")
? templatestring(
v.member, { project_number = module.projects[each.key].number }
)
: tonumber("[Error] Invalid member: '${v.member}' in project '${each.key}'")
)
)
role = lookup(var.factories_config.context.custom_roles, v.role, v.role)
})
}
network_users = [
for vv in try(each.value.shared_vpc_service_config.network_users, []) :
try(
# project service accounts (sa)
module.service-accounts["${each.key}/${vv}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.key}/automation/${vv}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.key}/${vv}"],
# other projects service accounts (project/sa)
module.service-accounts[vv].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[vv],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(vv, ":")
? templatestring(
vv, { project_number = module.projects[each.key].number }
)
: tonumber("[Error] Invalid member: '${vv}' in project '${each.key}'")
)
)
]
})
)
# add service agents config, so Service Agents can be referred in the IAM grants
service_agents_config = {
# default roles are granted in module.project
grant_default_roles = false
}
}
module "buckets" {
source = "../gcs"
for_each = {
for k in local.buckets : "${k.project_key}/${k.name}" => k
}
project_id = module.projects[each.value.project_key].project_id
prefix = each.value.prefix
name = "${each.value.project_name}-${each.value.name}"
encryption_key = each.value.encryption_key
force_destroy = each.value.force_destroy
iam = {
for k, v in each.value.iam : k => [
for vv in v : try(
# project service accounts (sa)
module.service-accounts["${each.value.project_key}/${vv}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.value.project_key}/automation/${vv}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.value.project_key}/${vv}"],
# other projects service accounts (project/sa)
module.service-accounts[vv].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[vv],
# project's service identities
local.service_agents_email["${each.value.project_key}/${vv}"],
local.service_agents_email[vv],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(vv, ":")
? templatestring(
vv, { project_number = module.projects[each.value.project_key].number }
)
: tonumber("[Error] Invalid member: '${vv}' in bucket '${each.key}'")
)
)
]
}
iam_bindings = {
for k, v in each.value.iam_bindings : k => merge(v, {
members = [
for vv in v.members : try(
# project service accounts (sa)
module.service-accounts["${each.value.project_key}/${vv}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.value.project_key}/automation/${vv}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.value.project_key}/${vv}"],
# other projects service accounts (project/sa)
module.service-accounts[vv].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[vv],
# project's service identities
local.service_agents_email["${each.value.project_key}/${vv}"],
local.service_agents_email[vv],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(vv, ":")
? templatestring(
vv, { project_number = module.projects[each.value.project_key].number }
)
: tonumber("[Error] Invalid member: '${vv}' in bucket '${each.key}'")
)
)
]
})
}
iam_bindings_additive = {
for k, v in each.value.iam_bindings_additive : k => merge(v, {
member = try(
# project service accounts (sa)
module.service-accounts["${each.value.project_key}/${v.member}"].iam_email,
# automation service account (rw)
local.context.iam_principals["${each.value.project_key}/automation/${v.member}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.value.project_key}/${v.member}"],
# other projects service accounts (project/sa)
module.service-accounts[v.member].iam_email,
# other automation service account (project/automation/rw)
local.context.iam_principals[v.member],
# project's service identities
local.service_agents_email["${each.value.project_key}/${v.member}"],
local.service_agents_email[v.member],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(v.member, ":")
? templatestring(
v.member, { project_number = module.projects[each.value.project_key].number }
)
: tonumber("[Error] Invalid member: '${v.member}' in bucket '${each.key}'")
)
)
})
}
labels = each.value.labels
location = coalesce(
var.data_overrides.storage_location,
lookup(each.value, "location", null),
var.data_defaults.storage_location
)
storage_class = each.value.storage_class
uniform_bucket_level_access = each.value.uniform_bucket_level_access
versioning = each.value.versioning
}
module "service-accounts" {
source = "../iam-service-account"
for_each = {
for k in local.service_accounts : "${k.project_key}/${k.name}" => k
}
project_id = module.projects[each.value.project_key].project_id
name = each.value.name
display_name = each.value.display_name
iam = {
for k, v in lookup(each.value, "iam", {}) : k => [
for vv in v : try(
# automation service account (rw)
local.context.iam_principals["${each.value.project_key}/automation/${vv}"],
# automation service account (automation/rw)
local.context.iam_principals["${each.value.project_key}/${vv}"],
# other automation service account (project/automation/rw)
local.context.iam_principals[vv],
# passthrough + error handling using tonumber until Terraform gets fail/raise function
(
strcontains(vv, ":")
? vv
: tonumber("[Error] Invalid member: '${vv}' in project '${each.value.project_key}'")
)
)
]
}
iam_project_roles = merge(
{
for k, v in each.value.iam_project_roles :
lookup(var.factories_config.context.vpc_host_projects, k, k) => v
},
each.value.iam_self_roles == null ? {} : {
(module.projects[each.value.project_key].project_id) = each.value.iam_self_roles
}
)
}
module "service_accounts-iam" {
source = "../iam-service-account"
for_each = {
for k in local.service_accounts : "${k.project_key}/${k.name}" => k
if k.iam_sa_roles != {}
}
project_id = module.service-accounts[each.key].service_account.project
name = each.value.name
service_account_create = false
iam_sa_roles = {
for k, v in each.value.iam_sa_roles : lookup(
local.service_accounts_names, "${each.value.project_key}/${k}", k
) => v
}
}
module "billing-account" {
source = "../billing-account"
count = var.factories_config.budgets == null ? 0 : 1
id = var.factories_config.budgets.billing_account
budget_notification_channels = (
var.factories_config.budgets.notification_channels
)
budgets = local.budgets
}