hunfabric/modules/gcs/README.md

Google Cloud Storage Module

• Simple bucket example
• Cloud KMS
  • External keys
  • KMS Autokey
• Retention policy, soft delete policy and logging
• Lifecycle rule
• GCS notifications
• Object upload
• IP Filter
• IAM
• Tag Bindings
• Managed Folders
• Hierarchical Namespace
• Variables
• Outputs

Simple bucket example

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  versioning = true
  labels = {
    cost-center = "devops"
  }
}
# tftest modules=1 resources=1 inventory=simple.yaml e2e

Cloud KMS

External keys

module "project" {
  source = "./fabric/modules/project"
  name   = var.project_id
  project_reuse = {
    use_data_source = false
    attributes = {
      name             = var.project_id
      number           = var.project_number
      services_enabled = ["storage.googleapis.com"]
    }
  }
}

module "kms" {
  source     = "./fabric/modules/kms"
  project_id = var.project_id
  keyring = {
    location = "europe" # location of the KMS must match location of the bucket
    name     = "${var.prefix}-test"
  }
  keys = {
    bucket_key = {
      iam_bindings = {
        bucket_key_iam = {
          members = [module.project.service_agents.storage.iam_email]
          role    = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
        }
      }
    }
  }
}

module "bucket" {
  source         = "./fabric/modules/gcs"
  project_id     = var.project_id
  prefix         = var.prefix
  name           = "my-bucket"
  location       = "EU"
  encryption_key = module.kms.keys.bucket_key.id
}

# tftest modules=2 e2e

KMS Autokey

For KMS Autokey to be used the project needs to be enabled and the principal running Terraform needs to have the roles/cloudkms.autokeyUser on the Autokey project.

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = "myproject"
  prefix     = "test"
  name       = "my-bucket"
  location   = "europe-west8"
  kms_autokeys = {
    default = {}
  }
  encryption_key = "$kms_keys:autokeys/default"
}
# tftest modules=1 resources=2

Retention policy, soft delete policy and logging

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  retention_policy = {
    retention_period = 100
    is_locked        = true
  }
  soft_delete_retention = 7776000
  logging_config = {
    log_bucket        = "log-bucket"
    log_object_prefix = null
  }
}
# tftest modules=1 resources=1 inventory=retention-logging.yaml

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  retention_policy = {
    retention_period = 100
    is_locked        = true
  }
  soft_delete_retention = 7776000
  context = {
    storage_buckets = {
      log-bucket = "log-bucket"
    }
  }
  logging_config = {
    log_bucket        = "$storage_buckets:log-bucket"
    log_object_prefix = null
  }
}
# tftest modules=1 inventory=retention-logging-context.yaml

Lifecycle rule

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  lifecycle_rules = {
    lr-0 = {
      action = {
        type          = "SetStorageClass"
        storage_class = "STANDARD"
      }
      condition = {
        age = 30
      }
    }
  }
}
# tftest modules=1 resources=1 inventory=lifecycle.yaml e2e

GCS notifications

module "project" {
  source = "./fabric/modules/project"
  name   = var.project_id
  project_reuse = {
    use_data_source = false
    attributes = {
      name             = var.project_id
      number           = var.project_number
      services_enabled = ["storage.googleapis.com"]
    }
  }
}

module "bucket-gcs-notification" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  notification_config = {
    enabled        = true
    payload_format = "JSON_API_V1"
    sa_email       = module.project.service_agents.storage.email
    topic_name     = "gcs-notification-topic"
    event_types    = ["OBJECT_FINALIZE"]
  }
}
# tftest e2e

Object upload

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  objects_to_upload = {
    sample-data = {
      name         = "example-file.csv"
      source       = "assets/example-file.csv"
      content_type = "text/csv"
    }
  }
}
# tftest modules=1 resources=2 inventory=object-upload.yaml e2e

IP Filter

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  ip_filter = {
    allow_all_service_agent_access = false
    allow_cross_org_vpcs           = false
    public_network_sources = [
      "8.8.8.8/32",
      "8.8.4.4/32"
    ]
    vpc_network_sources = {
      "projects/my-project-id/global/networks/my-vpc" = [
        "10.0.0.0/8"
      ]
    }
  }
}
# tftest modules=1 resources=1 inventory=ip-filter.yaml

IAM

IAM is managed via several variables that implement different features and levels of control:

• iam and iam_by_principals configure authoritative bindings that manage individual roles exclusively, and are internally merged
• iam_bindings configure authoritative bindings with optional support for conditions, and are not internally merged with the previous two variables
• iam_bindings_additive configure additive bindings via individual role/member pairs with optional support conditions

The authoritative and additive approaches can be used together, provided different roles are managed by each. Some care must also be taken with the iam_by_principals variable to ensure that variable keys are static values, so that Terraform is able to compute the dependency graph.

Refer to the project module for examples of the IAM interface. IAM also supports variable interpolation for both roles and principals and for the foreign resources where the service account is the principal, via the respective attributes in the var.context variable. Basic usage is shown in the example below.

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  context = {
    iam_principals = {
      mygroup = "group:${var.group_email}"
    }
  }
  iam = {
    "roles/storage.admin" = ["$iam_principals:mygroup"]
  }
}
# tftest modules=1 resources=2 inventory=iam-authoritative.yaml e2e

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  iam_bindings = {
    storage-admin-with-delegated_roles = {
      role    = "roles/storage.admin"
      members = ["group:${var.group_email}"]
      condition = {
        title = "delegated-role-grants"
        expression = format(
          "api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', []).hasOnly([%s])",
          join(",", formatlist("'%s'",
            [
              "roles/storage.objectAdmin",
              "roles/storage.objectViewer",
            ]
          ))
        )
      }
    }
  }
}
# tftest modules=1 resources=2 inventory=iam-bindings.yaml e2e

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  iam_bindings_additive = {
    storage-admin-with-delegated_roles = {
      role   = "roles/storage.admin"
      member = "group:${var.group_email}"
      condition = {
        title = "delegated-role-grants"
        expression = format(
          "api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', []).hasOnly([%s])",
          join(",", formatlist("'%s'",
            [
              "roles/storage.objectAdmin",
              "roles/storage.objectViewer",
            ]
          ))
        )
      }
    }
  }
}
# tftest modules=1 resources=2 inventory=iam-bindings-additive.yaml e2e

Tag Bindings

Refer to the Creating and managing tags documentation for details on usage.

module "org" {
  source          = "./fabric/modules/organization"
  organization_id = var.organization_id
  tags = {
    environment = {
      description = "Environment specification."
      values = {
        dev     = {}
        prod    = {}
        sandbox = {}
      }
    }
  }
}

module "bucket" {
  source     = "./fabric/modules/gcs"
  project_id = var.project_id
  prefix     = var.prefix
  name       = "my-bucket"
  location   = "EU"
  tag_bindings = {
    env-sandbox = module.org.tag_values["environment/sandbox"].id
  }
}
# tftest modules=2 resources=6

Managed Folders

module "bucket" {
  source        = "./fabric/modules/gcs"
  bucket_create = false
  prefix        = var.prefix
  name          = "my-bucket"
  location      = "EU"
  managed_folders = {
    folder1 = {
      iam = {
        "roles/storage.admin" = ["user:user1@example.com"]
      }
    }
    "folder1/subfolder" = {
      force_destroy = true
    }
  }
}
# tftest inventory=managed-folders.yaml

Hierarchical Namespace

module "bucket" {
  source                        = "./fabric/modules/gcs"
  project_id                    = var.project_id
  prefix                        = var.prefix
  name                          = "my-bucket"
  location                      = "EU"
  enable_hierarchical_namespace = true
  uniform_bucket_level_access   = true
}
# tftest inventory=hns.yaml

Variables

name	description	type	required	default
name	Bucket name suffix.	string	✓
autoclass	Enable autoclass to automatically transition objects to appropriate storage classes based on their access pattern. If set to true, storage_class must be set to STANDARD. Defaults to false.	bool		null
bucket_create	Create bucket.	bool		true
context	Context-specific interpolations.	object({…})		{}
cors	CORS configuration for the bucket. Defaults to null.	object({…})		null
custom_placement_config	The bucket's custom location configuration, which specifies the individual regions that comprise a dual-region bucket. If the bucket is designated as REGIONAL or MULTI_REGIONAL, the parameters are empty.	list(string)		null
default_event_based_hold	Enable event based hold to new objects added to specific bucket, defaults to false.	bool		null
enable_hierarchical_namespace	Enables hierarchical namespace.	bool		null
enable_object_retention	Enables object retention on a storage bucket.	bool		null
encryption_key	KMS key that will be used for encryption.	string		null
force_destroy	Optional map to set force destroy keyed by name, defaults to false.	bool		false
iam	IAM bindings in {ROLE => [MEMBERS]} format.	map(list(string))		{}
iam_bindings	Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary.	map(object({…}))		{}
iam_bindings_additive	Individual additive IAM bindings. Keys are arbitrary.	map(object({…}))		{}
iam_by_principals	Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the iam variable.	map(list(string))		{}
ip_filter	The bucket's IP filter configuration.	object({…})		null
kms_autokeys	KMS Autokey key handles. If location is not specified the bucket location will be used. Key handle names will be added to the kms_keys context with an autokeys/ prefix.	map(object({…}))		{}
labels	Labels to be attached to all buckets.	map(string)		{}
lifecycle_rules	Bucket lifecycle rule.	map(object({…}))		{}
location	Bucket location.	string		null
logging_config	Bucket logging configuration.	object({…})		null
managed_folders	Managed folders to create within the bucket in {PATH => CONFIG} format.	map(object({…}))		{}
notification_config	GCS Notification configuration.	object({…})		null
objects_to_upload	Objects to be uploaded to bucket.	map(object({…}))		{}
prefix	Optional prefix used to generate the bucket name.	string		null
project_id	Bucket project id. Only required when creating buckets, or notification config topics.	string		null
public_access_prevention	Prevents public access to the bucket.	string		null
requester_pays	Enables Requester Pays on a storage bucket.	bool		null
retention_policy	Bucket retention policy.	object({…})		null
rpo	Bucket recovery point objective.	string		null
soft_delete_retention	The duration in seconds that soft-deleted objects in the bucket will be retained and cannot be permanently deleted. Set to 0 to override the default and disable.	number		null
storage_class	Bucket storage class.	string		"STANDARD"
tag_bindings	Tag bindings for this folder, in key => tag value id format.	map(string)		{}
uniform_bucket_level_access	Allow using object ACLs (false) or not (true, this is the recommended behavior) , defaults to true (which is the recommended practice, but not the behavior of storage API).	bool		true
versioning	Enable versioning, defaults to false.	bool		null
website	Bucket website.	object({…})		null

Outputs

name	description	sensitive
bucket	Bucket resource.
id	Fully qualified bucket id.
name	Bucket name.
notification	GCS Notification self link.
objects	Objects in GCS bucket.
topic	Topic ID used by GCS.
url	Bucket URL.