Add Data Catalog Policy Tag (#520)

* First commit * Add outputs, update README, fix variables * Fix * Fix * Fix * Fix * Fix * Fix tests, for real? * Fix tests, for real. Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2022-02-09 13:05:27 +01:00
parent f4ef54da34
commit 307c29d2f8
9 changed files with 308 additions and 1 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,8 @@ All notable changes to this project will be documented in this file.
 - new `iot-core` module
 - **incompatible change** the variables for host and service Shared VPCs have changed in the project module
 - **incompatible change** the variable for service identities IAM has changed in the project factory
+- the `net-vpc` and `project` modules now use the beta provider for shared VPC-related resources
+- add `data-catalog-policy-tag` module

 ## [13.0.0] - 2022-01-27

--- a/README.md
+++ b/README.md
@@ -29,7 +29,7 @@ Currently available modules:
 - **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [naming convention](./modules/naming-convention)
 - **networking** - [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC peering](./modules/net-vpc-peering), [VPN static](./modules/net-vpn-static), [VPN dynamic](./modules/net-vpn-dynamic), [VPN HA](./modules/net-vpn-ha), [NAT](./modules/net-cloudnat), [address reservation](./modules/net-address), [DNS](./modules/dns), [L4 ILB](./modules/net-ilb), [Service Directory](./modules/service-directory), [Cloud Endpoints](./modules/endpoints)
 - **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [GKE cluster](./modules/gke-cluster), [GKE nodepool](./modules/gke-nodepool), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid)
- **data** - [GCS](./modules/gcs), [BigQuery dataset](./modules/bigquery-dataset), [Pub/Sub](./modules/pubsub), [Datafusion](./modules/datafusion), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance)
+- **data** - [GCS](./modules/gcs), [BigQuery dataset](./modules/bigquery-dataset), [Pub/Sub](./modules/pubsub), [Datafusion](./modules/datafusion), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag)
 - **development** - [Cloud Source Repository](./modules/source-repository), [Container Registry](./modules/container-registry), [Artifact Registry](./modules/artifact-registry), [Apigee Organization](./modules/apigee-organization), [Apigee X Instance](./modules/apigee-x-instance)
 - **security** - [KMS](./modules/kms), [SecretManager](./modules/secret-manager), [VPC Service Control](./modules/vpc-sc)
 - **serverless** - [Cloud Function](./modules/cloud-function), [Cloud Run](./modules/cloud-run)
--- a/modules/README.md
+++ b/modules/README.md
@@ -69,6 +69,7 @@ These modules are used in the examples included in this repository. If you are u
 - [Pub/Sub](./pubsub)
 - [Bigtable instance](./bigtable-instance)
 - [Cloud SQL instance](./cloudsql-instance)
+- [Data Catalog Policy Tag](./data-catalog-policy-tag)

 ## Development

--- a/modules/data-catalog-policy-tag/README.md
+++ b/modules/data-catalog-policy-tag/README.md
@@ -0,0 +1,62 @@
+# Data Catalog Module
+
+This module simplifies the creation of [Data Catalog](https://cloud.google.com/data-catalog) Policy Tags. Policy Tags can be used to configure [Bigquery column-level access](https://cloud.google.com/bigquery/docs/best-practices-policy-tags).
+
+Note: Data Catalog is still in beta, hence this module currently uses the beta provider.
+## Examples
+
+### Simple Taxonomy with policy tags
+
+```hcl
+module "cmn-dc" {
+  source     = "./modules/data-catalog-policy-tag"
+  name       = "my-datacatalog-policy-tags"
+  project_id = "my-project"
+  tags       = ["low", "medium", "high"]
+}
+# tftest modules=1 resources=4
+```
+
+### Simple Taxonomy with IAM binding
+
+```hcl
+module "cmn-dc" {
+  source     = "./modules/data-catalog-policy-tag"
+  name       = "my-datacatalog-policy-tags"
+  project_id = "my-project"
+  tags       = ["low", "medium", "high"]
+  iam = {
+    "roles/datacatalog.categoryAdmin" = ["group:GROUP_NAME@example.com"]
+  }
+}
+# tftest modules=1 resources=5
+```
+<!-- BEGIN TFDOC -->
+
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [name](variables.tf#L59) | Name of this taxonomy. | <code>string</code> | ✓ |  |
+| [project_id](variables.tf#L70) | GCP project id. | <code></code> | ✓ |  |
+| [activated_policy_types](variables.tf#L17) | A list of policy types that are activated for this taxonomy. | <code>list&#40;string&#41;</code> |  | <code>&#91;&#34;FINE_GRAINED_ACCESS_CONTROL&#34;&#93;</code> |
+| [description](variables.tf#L23) | Description of this taxonomy. | <code>string</code> |  | <code>&#34;Taxonomy - Terraform managed&#34;</code> |
+| [group_iam](variables.tf#L29) | Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the `iam` variable. | <code>map&#40;list&#40;string&#41;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [iam](variables.tf#L35) | IAM bindings in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [iam_additive](variables.tf#L41) | IAM additive bindings in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [iam_additive_members](variables.tf#L47) | IAM additive bindings in {MEMBERS => [ROLE]} format. This might break if members are dynamic values. | <code>map&#40;list&#40;string&#41;&#41;</code> |  | <code>&#123;&#125;</code> |
+| [location](variables.tf#L53) | Data Catalog Taxonomy location. | <code>string</code> |  | <code>&#34;eu&#34;</code> |
+| [prefix](variables.tf#L64) | Prefix used to generate project id and name. | <code>string</code> |  | <code>null</code> |
+| [tags](variables.tf#L74) | List of Data Catalog Policy tags to be created. | <code>list&#40;string&#41;</code> |  | <code>&#91;&#93;</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| [tags](outputs.tf#L17) | Policy Tags. |  |
+| [taxonomy_id](outputs.tf#L22) | Taxonomy id. |  |
+
+<!-- END TFDOC -->
+## TODO
+- Support IAM at tag level.
+- Support Child policy tags
--- a/modules/data-catalog-policy-tag/iam.tf
+++ b/modules/data-catalog-policy-tag/iam.tf
@@ -0,0 +1,67 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# tfdoc:file:description Data Catalog Taxonomy IAM definition.
+
+locals {
+  _group_iam = {
+    for r in local._group_iam_roles : r => [
+      for k, v in var.group_iam : "group:${k}" if try(index(v, r), null) != null
+    ]
+  }
+  _group_iam_roles = distinct(flatten(values(var.group_iam)))
+  _iam_additive_member_pairs = flatten([
+    for member, roles in var.iam_additive_members : [
+      for role in roles : { role = role, member = member }
+    ]
+  ])
+  _iam_additive_pairs = flatten([
+    for role, members in var.iam_additive : [
+      for member in members : { role = role, member = member }
+    ]
+  ])
+  iam = {
+    for role in distinct(concat(keys(var.iam), keys(local._group_iam))) :
+    role => concat(
+      try(var.iam[role], []),
+      try(local._group_iam[role], [])
+    )
+  }
+  iam_additive = {
+    for pair in concat(local._iam_additive_pairs, local._iam_additive_member_pairs) :
+    "${pair.role}-${pair.member}" => pair
+  }
+}
+
+resource "google_data_catalog_taxonomy_iam_binding" "authoritative" {
+  provider = google-beta
+  for_each = local.iam
+  role     = each.key
+  members  = each.value
+  taxonomy = google_data_catalog_taxonomy.default.id
+}
+
+resource "google_data_catalog_taxonomy_iam_member" "additive" {
+  provider = google-beta
+  for_each = (
+    length(var.iam_additive) + length(var.iam_additive_members) > 0
+    ? local.iam_additive
+    : {}
+  )
+  role     = each.value.role
+  member   = each.value.member
+  taxonomy = google_data_catalog_taxonomy.default.id
+}
--- a/modules/data-catalog-policy-tag/main.tf
+++ b/modules/data-catalog-policy-tag/main.tf
@@ -0,0 +1,43 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# tfdoc:file:description Data Catalog Taxonomy definition
+
+locals {
+  name = (
+    var.name != null ? var.name : "${local.prefix}taxonomy"
+  )
+  prefix = var.prefix == null ? "" : "${var.prefix}-"
+}
+
+resource "google_data_catalog_taxonomy" "default" {
+  provider               = google-beta
+  project                = var.project_id
+  region                 = var.location
+  display_name           = local.name
+  description            = var.description
+  activated_policy_types = var.activated_policy_types
+}
+
+resource "google_data_catalog_policy_tag" "default" {
+  for_each     = toset(var.tags)
+  provider     = google-beta
+  taxonomy     = google_data_catalog_taxonomy.default.id
+  display_name = each.key
+  description  = "${each.key} - Terraform managed.  "
+}
+
+#TODO Add IAM at tag level
--- a/modules/data-catalog-policy-tag/outputs.tf
+++ b/modules/data-catalog-policy-tag/outputs.tf
@@ -0,0 +1,25 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "tags" {
+  description = "Policy Tags."
+  value       = { for k, v in google_data_catalog_policy_tag.default : v.id => v.name }
+}
+
+output "taxonomy_id" {
+  description = "Taxonomy id."
+  value       = google_data_catalog_taxonomy.default.id
+}
--- a/modules/data-catalog-policy-tag/variables.tf
+++ b/modules/data-catalog-policy-tag/variables.tf
@@ -0,0 +1,78 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "activated_policy_types" {
+  description = "A list of policy types that are activated for this taxonomy."
+  type        = list(string)
+  default     = ["FINE_GRAINED_ACCESS_CONTROL"]
+}
+
+variable "description" {
+  description = "Description of this taxonomy."
+  type        = string
+  default     = "Taxonomy - Terraform managed"
+}
+
+variable "group_iam" {
+  description = "Authoritative IAM binding for organization groups, in {GROUP_EMAIL => [ROLES]} format. Group emails need to be static. Can be used in combination with the `iam` variable."
+  type        = map(list(string))
+  default     = {}
+}
+
+variable "iam" {
+  description = "IAM bindings in {ROLE => [MEMBERS]} format."
+  type        = map(list(string))
+  default     = {}
+}
+
+variable "iam_additive" {
+  description = "IAM additive bindings in {ROLE => [MEMBERS]} format."
+  type        = map(list(string))
+  default     = {}
+}
+
+variable "iam_additive_members" {
+  description = "IAM additive bindings in {MEMBERS => [ROLE]} format. This might break if members are dynamic values."
+  type        = map(list(string))
+  default     = {}
+}
+
+variable "location" {
+  description = "Data Catalog Taxonomy location."
+  type        = string
+  default     = "eu"
+}
+
+variable "name" {
+  description = "Name of this taxonomy."
+  type        = string
+}
+
+variable "prefix" {
+  description = "Prefix used to generate project id and name."
+  type        = string
+  default     = null
+}
+
+variable "project_id" {
+  description = "GCP project id."
+}
+
+variable "tags" {
+  description = "List of Data Catalog Policy tags to be created."
+  type        = list(string)
+  default     = []
+}
--- a/modules/data-catalog-policy-tag/versions.tf
+++ b/modules/data-catalog-policy-tag/versions.tf
@@ -0,0 +1,29 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 1.1.0"
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = ">= 4.0.0"
+    }
+    google-beta = {
+      source  = "hashicorp/google-beta"
+      version = ">= 4.0.0"
+    }
+  }
+}
+
+