Add module for retrieving all projects/folders under a specific parent (recursively).

README.md

@@ -26,7 +26,7 @@ The current list of modules supports most of the core foundational and networkin
 
 Currently available modules:
 
-- **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [naming convention](./modules/naming-convention)
+- **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [naming convention](./modules/naming-convention), [projects-data-source](./modules/projects-data-source)
 - **networking** - [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC peering](./modules/net-vpc-peering), [VPN static](./modules/net-vpn-static), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [NAT](./modules/net-cloudnat), [address reservation](./modules/net-address), [DNS](./modules/dns), [L4 ILB](./modules/net-ilb), [Service Directory](./modules/service-directory), [Cloud Endpoints](./modules/endpoints)
 - **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [GKE cluster](./modules/gke-cluster), [GKE nodepool](./modules/gke-nodepool), [GKE hub](./modules/gke-hub), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid)
 - **data** - [GCS](./modules/gcs), [BigQuery dataset](./modules/bigquery-dataset), [Pub/Sub](./modules/pubsub), [Datafusion](./modules/datafusion), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag)

modules/README.md

@@ -35,6 +35,7 @@ These modules are used in the examples included in this repository. If you are u
 - [naming convention](./naming-convention)
 - [organization](./organization)
 - [project](./project)
+- [projects-data-source](./projects-data-source)
 - [service account](./iam-service-account)
 
 ## Networking modules

modules/projects-data-source/README.md

@@ -0,0 +1,63 @@
+# Projects Data Source Module 
+
+This module extends functionality of [google_projects](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/projects) data source by retrieving all the projects and folders under a specific `parent` recursively. 
+
+A good usage pattern would be when we want all the projects under a specific folder (including nested subfolders) to be included into [VPC Service Controls](../vpc-sc/). Instead of manually maintaining the list of project numbers as an input to the `vpc-sc` module we can use that module to retrieve all the project numbers dynamically.
+
+## Examples
+
+### All projects in my org
+
+```hcl
+module "my-org" {
+  source     = "./modules/projects-data-source"
+  parent     = "organizations/123456789"
+}
+
+output "projects" {
+  value = module.my_org.projects
+}
+
+output "folders" {
+  value = module.my-org.folders
+}
+
+# tftest modules=1 resources=0
+```
+
+### My dev projects based on parent and label
+
+```hcl
+module "my-dev" {
+  source = "./modules/projects-data-source"
+  parent = "folders/123456789"
+  filter = "labels.env:DEV lifecycleState:ACTIVE"   
+}
+
+output "dev-projects" {
+  value = module.my-dev.projects
+}
+
+output "dev-folders" {
+  value = module.my-dev.folders
+}
+# tftest modules=1 resources=0
+```
+<!-- BEGIN TFDOC -->
+
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [parent](variables.tf#L17) | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | <code>string</code> | ✓ |  |
+| [filter](variables.tf#L26) | A string filter as defined in the [REST API](https://cloud.google.com/resource-manager/reference/rest/v1/projects/list#query-parameters). | <code>string</code> |  | <code>&#34;lifecycleState:ACTIVE&#34;</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| [folders](outputs.tf#L17) | Map of folders attributes keyed by folder id. |  |
+| [project_numbers](outputs.tf#L27) | List of project numbers. |  |
+| [projects](outputs.tf#L22) | Map of projects attributes keyed by projects id. |  |
+
+<!-- END TFDOC -->

modules/projects-data-source/main.tf

@@ -0,0 +1,143 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  folders_l1_map = { for item in data.google_folders.folders_l1.folders : item.name => item }
+
+  folders_l2_map = merge([
+    for _, v in data.google_folders.folders_l2 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l3_map = merge([
+    for _, v in data.google_folders.folders_l3 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l4_map = merge([
+    for _, v in data.google_folders.folders_l4 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l5_map = merge([
+    for _, v in data.google_folders.folders_l5 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l6_map = merge([
+    for _, v in data.google_folders.folders_l6 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l7_map = merge([
+    for _, v in data.google_folders.folders_l7 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l8_map = merge([
+    for _, v in data.google_folders.folders_l8 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l9_map = merge([
+    for _, v in data.google_folders.folders_l9 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  folders_l10_map = merge([
+    for _, v in data.google_folders.folders_l10 :
+    { for item in v.folders : item.name => item }
+  ]...)
+
+  all_folders = merge(
+    local.folders_l1_map,
+    local.folders_l2_map,
+    local.folders_l3_map,
+    local.folders_l4_map,
+    local.folders_l5_map,
+    local.folders_l6_map,
+    local.folders_l7_map,
+    local.folders_l8_map,
+    local.folders_l9_map,
+    local.folders_l10_map
+  )
+
+  parent_ids = toset(concat(
+    [split("/", var.parent)[1]],
+    [for k, _ in local.all_folders : split("/", k)[1]]
+  ))
+
+  projects = merge([
+    for _, v in data.google_projects.projects :
+    { for item in v.projects : item.project_id => item }
+  ]...)
+}
+
+# 10 datasources are used to cover 10 possible nested layers in GCP organization hirerarcy. 
+data "google_folders" "folders_l1" {
+  parent_id = var.parent
+}
+
+data "google_folders" "folders_l2" {
+  for_each  = local.folders_l1_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l3" {
+  for_each  = local.folders_l2_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l4" {
+  for_each  = local.folders_l3_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l5" {
+  for_each  = local.folders_l4_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l6" {
+  for_each  = local.folders_l5_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l7" {
+  for_each  = local.folders_l6_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l8" {
+  for_each  = local.folders_l7_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l9" {
+  for_each  = local.folders_l8_map
+  parent_id = each.value.name
+}
+
+data "google_folders" "folders_l10" {
+  for_each  = local.folders_l9_map
+  parent_id = each.value.name
+}
+
+# Getting all projects parented by any of the folders in the tree including root prg/folder provided by `parent` variable.
+data "google_projects" "projects" {
+  for_each = local.parent_ids
+  filter   = "parent.id:${each.value} ${var.filter}"
+}

modules/projects-data-source/outputs.tf

@@ -0,0 +1,30 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "folders" {
+  description = "Map of folders attributes keyed by folder id."
+  value       = local.all_folders
+}
+
+output "projects" {
+  description = "Map of projects attributes keyed by projects id."
+  value       = local.projects
+}
+
+output "project_numbers" {
+  description = "List of project numbers."
+  value       = [for _, v in local.projects : v.number]
+}

modules/projects-data-source/variables.tf

@@ -0,0 +1,30 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "parent" {
+  description = "Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format."
+  type        = string
+  validation {
+    condition     = can(regex("(organizations|folders)/[0-9]+", var.parent))
+    error_message = "Parent must be of the form folders/folder_id or organizations/organization_id."
+  }
+}
+
+variable "filter" {
+  description = "A string filter as defined in the [REST API](https://cloud.google.com/resource-manager/reference/rest/v1/projects/list#query-parameters)."
+  type        = string
+  default     = "lifecycleState:ACTIVE"
+}

modules/projects-data-source/versions.tf

@@ -0,0 +1,29 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 1.0.0"
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = ">= 4.0.0"
+    }
+    google-beta = {
+      source  = "hashicorp/google-beta"
+      version = ">= 4.0.0"
+    }
+  }
+}
+
+

tests/modules/projects_data_source/__init__.py

@@ -0,0 +1,13 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

tests/modules/projects_data_source/fixture/main.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+module "my-org" {
+  source = "../../../../modules/projects-data-source"
+  parent = var.parent
+}

tests/modules/projects_data_source/fixture/variables.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "parent" {
+  type    = string
+  default = "organizations/123456789"
+}

tests/modules/projects_data_source/test_plan.py

@@ -0,0 +1,26 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pytest
+
+
+@pytest.fixture
+def resources(plan_runner):
+  _, resources = plan_runner()
+  return resources
+
+
+def test_resource_count(resources):
+  "Test number of resources created."
+  assert len(resources) == 0