Files
hunfabric/modules/organization/logging.tf
Ludovico Magnocavallo 36648b6b63 FAST light implementation (#3255)
* data wip

* wip data

* update org schema, add note on expansion

* all schemas, workload notes

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* wip

* data wip

* wip

* wip

* wip

* wip

* org module IAM context (using lookup)

* new-style context expansion in project IAM

* remove spurious file

* project module contexts

* finalize context replacement format for project module

* revert org module changes

* fix tag id interpolation in project

* fix tag id interpolation in project

* organization module context

* organization context test

* context expansion for folder tag bindings

* test context expansion for tag bindings

* service account module context

* simplify context local

* context for iam service account

* nuke blueprints

* remove links to blueprints

* vpc sc context in project module

* Add context to GCS module

* Add inline deps to plan_summary script

* Make context a top-level variable for folder, organization, sa

* Add add context top-level to VPC-SC

* move context out of factories_config variable

* tfdoc

* fix merge

* fix merge

* fix examples

* net-vpc module context

* add parent ids to folder context

* rename folder parent context

* fix folder parent check

* new project factory stub

* wip

* wip

* refactor defaults

* project iam

* bueckts and service accounts

* start adding context replacements

* better test data

* automation resources for folders and projects

* automation

* add support for project id interpolation

* first tested apply

* improve IAM description in gcs module

* add context to billing account module

* add notification channels to billing account module context

* add billing budgets to new pf

* schemas and defaults

* bootstrap wip

* bootstrap wip

* bootstrap wip

* pf outputs

* pf fixes

* fix pf sample data

* bootstrap lite fixes

* add locations to organization module contexts

* bootstrap lite fixes

* org fixes, billing accounts

* fix default project parent

* bootstrap lite wip

* add locations to gcs module context

* add context support to logging bucket module

* add context to pubsub module

* split out iam variables in gcs module

* fix logging bucket context test

* bootstrap log sink destinations

* streamline logging-bucket module variables

* fix logging bucket context test

* align logging bucket module interface in fast bootstrap

* add support for project-level log buckets to project factory

* support full context expansion in organization module log sinks

* log buckets in fast-lite bootstrap

* make og sink type optional in organization module

* log sinks in fast-lite bootstrap

* set tag values in factory context

* bootstrap lite data

* output files schema

* billing account schema

* output files

* output providers

* gcs output files

* boilerplate

* tflint

* check documentation

* check docs

* fix project module parent variable validation

* fix log bucket examples

* allow null parent in project module

* silence folder test errors

* fix billing account sink example

* fix project example

* fix billing account module

* fix folder tests

* fix FAST

* fix fast

* tfvars outputs

* wif

* cicd service accounts

* cicd

* allow defaults in context, minimal org policies

* support gcs managed folders in project factory and bootstrap lite

* support prefix in provider output files

* rename bootstrap stage

* gitignore

* gitignore

* security folder, billing IAM

* wip tfvars

* fix typo

* security IAM

* control tag iam/context via variables in organization module

* split tag creation from tag IAM to avoid circular refs

* port organization module tag changes to project module

* implement new-style context expansion in vpc-sc module

* fix fast vpc-sc tests

* boilerplate

* vpc sc stage

* schemas

* fast-lite compatibility for vpc sc stage

* make log project number optional in vpc-sc stage

* networking

* networking

* networking

* networking

* rename and move new stage under fast

* clone pf tests

* use context replacement for internal notification channels in billing account module

* support service agents in project module iam context replacements

* support service agents in project module iam context replacements

* add support for kms keys to project module context

* experimental pf example test and fixes

* fix schemas

* fix tests

* tfdoc

* tfdoc

* pf config

* experimental pf

* remove redundant dot from gcs managed folder IAM keys

* bootstrap experimental test

* project factory exp stage test

* skip tflint for bootstrap experimental test

* tflint

* fix gcs test

* documentation work

* documentation work

* Update README.md

* tfdoc

* tfdoc

* readme

* tfdoc

* readme

* readme

* readme

* readme

* support universe in pf exp projects

* missing universe service agents

* org policies import, non-admin billing IAM

* todo

* fix test

* custom constraints

* fast classic dataset

* fix test data

* context replacements in billing module log sinks

* fix typo

* add support for billing log sinks

* update docs

* readme

* cicd fix and test

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2025-09-02 08:38:57 +02:00

163 lines
5.6 KiB
HCL

/**
* Copyright 2025 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
# tfdoc:file:description Log sinks and data access logs.
locals {
logging_sinks = {
for k, v in var.logging_sinks :
# expand destination contexts
k => merge(v,
v.type != "bigquery" ? {} : {
destination = lookup(
local.ctx.bigquery_datasets, v.destination, v.destination
)
},
v.type != "logging" ? {} : {
destination = lookup(
local.ctx.log_buckets, v.destination, v.destination
)
},
v.type != "project" ? {} : {
api = "logging"
destination = "projects/${lookup(local.ctx.project_ids, v.destination, v.destination)}"
},
v.type != "pubsub" ? {} : {
destination = lookup(
local.ctx.pubsub_topics, v.destination, v.destination
)
},
v.type != "storage" ? {} : {
destination = lookup(
local.ctx.storage_buckets, v.destination, v.destination
)
}
)
}
sink_bindings = {
for type in ["bigquery", "logging", "project", "pubsub", "storage"] :
type => {
for name, sink in local.logging_sinks :
name => sink if sink.iam && sink.type == type
}
}
}
resource "google_logging_organization_settings" "default" {
count = var.logging_settings != null ? 1 : 0
organization = local.organization_id_numeric
disable_default_sink = var.logging_settings.disable_default_sink
storage_location = lookup(
local.ctx.locations,
coalesce(var.logging_settings.storage_location, ""),
var.logging_settings.storage_location
)
}
resource "google_organization_iam_audit_config" "default" {
for_each = var.logging_data_access
org_id = local.organization_id_numeric
service = each.key
dynamic "audit_log_config" {
for_each = { for k, v in each.value : k => v if v != null }
content {
log_type = audit_log_config.key
exempted_members = audit_log_config.value.exempted_members
}
}
}
resource "google_logging_organization_sink" "sink" {
for_each = local.logging_sinks
name = each.key
description = coalesce(each.value.description, "${each.key} (Terraform-managed).")
org_id = local.organization_id_numeric
destination = "${lookup(each.value, "api", each.value.type)}.googleapis.com/${each.value.destination}"
filter = each.value.filter
include_children = each.value.include_children
intercept_children = each.value.intercept_children
disabled = each.value.disabled
dynamic "bigquery_options" {
for_each = each.value.type == "bigquery" ? [""] : []
content {
use_partitioned_tables = each.value.bq_partitioned_table
}
}
dynamic "exclusions" {
for_each = each.value.exclusions
iterator = exclusion
content {
name = exclusion.key
filter = exclusion.value
}
}
depends_on = [
google_organization_iam_binding.authoritative,
google_organization_iam_binding.bindings,
google_organization_iam_member.bindings
]
}
resource "google_storage_bucket_iam_member" "storage-sinks-binding" {
for_each = local.sink_bindings["storage"]
bucket = each.value.destination
role = "roles/storage.objectCreator"
member = google_logging_organization_sink.sink[each.key].writer_identity
}
resource "google_bigquery_dataset_iam_member" "bq-sinks-binding" {
for_each = local.sink_bindings["bigquery"]
project = split("/", each.value.destination)[1]
dataset_id = split("/", each.value.destination)[3]
role = "roles/bigquery.dataEditor"
member = google_logging_organization_sink.sink[each.key].writer_identity
}
resource "google_pubsub_topic_iam_member" "pubsub-sinks-binding" {
for_each = local.sink_bindings["pubsub"]
project = split("/", each.value.destination)[1]
topic = split("/", each.value.destination)[3]
role = "roles/pubsub.publisher"
member = google_logging_organization_sink.sink[each.key].writer_identity
}
resource "google_project_iam_member" "bucket-sinks-binding" {
for_each = local.sink_bindings["logging"]
project = split("/", each.value.destination)[1]
role = "roles/logging.bucketWriter"
member = google_logging_organization_sink.sink[each.key].writer_identity
condition {
title = "${each.key} bucket writer"
description = "Grants bucketWriter to ${google_logging_organization_sink.sink[each.key].writer_identity} used by log sink ${each.key} on ${var.organization_id}"
expression = "resource.name.endsWith('${each.value.destination}')"
}
}
resource "google_project_iam_member" "project-sinks-binding" {
for_each = local.sink_bindings["project"]
project = each.value.destination
role = "roles/logging.logWriter"
member = google_logging_organization_sink.sink[each.key].writer_identity
}
resource "google_logging_organization_exclusion" "logging-exclusion" {
for_each = var.logging_exclusions
name = each.key
org_id = local.organization_id_numeric
description = "${each.key} (Terraform-managed)."
filter = each.value
}