kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into example-wordpress

Browse Source
This commit is contained in:
Natalia Strelkova
2022-09-21 15:47:04 +02:00
committed by GitHub
parent dbdaeda376 d0aad6f95c
commit 256b1b77f9
37 changed files with 79 additions and 1181 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@@ -183,7 +183,7 @@ All notable changes to this project will be documented in this file.
- **incompatible change** removed `iam` key from logging sink configuration in the `project` and `organization` modules
- remove GCS to BQ with Dataflow example, replace by GCS to BQ with least privileges
- the `net-vpc` and `project` modules now use the beta provider for shared VPC-related resources
- new [iot-core](modules/iot-core) module
- new iot-core module
- **incompatible change** the variables for host and service Shared VPCs have changed in the project module
- **incompatible change** the variable for service identities IAM has changed in the project factory
- add `data-catalog-policy-tag` module

2
README.md
View File

@@ -29,7 +29,7 @@ The current list of modules supports most of the core foundational and networkin
Currently available modules:
- **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [naming convention](./modules/naming-convention), [projects-data-source](./modules/projects-data-source), [organization-policy](./modules/organization-policy)
- **foundational** - [folder](./modules/folder), [organization](./modules/organization), [project](./modules/project), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [billing budget](./modules/billing-budget), [projects-data-source](./modules/projects-data-source), [organization-policy](./modules/organization-policy)
- **networking** - [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC peering](./modules/net-vpc-peering), [VPN static](./modules/net-vpn-static), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [NAT](./modules/net-cloudnat), [address reservation](./modules/net-address), [DNS](./modules/dns), [L4 ILB](./modules/net-ilb), [L7 ILB](./modules/net-ilb-l7), [Service Directory](./modules/service-directory), [Cloud Endpoints](./modules/endpoints)
- **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [GKE cluster](./modules/gke-cluster), [GKE nodepool](./modules/gke-nodepool), [GKE hub](./modules/gke-hub), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid)
- **data** - [GCS](./modules/gcs), [BigQuery dataset](./modules/bigquery-dataset), [Pub/Sub](./modules/pubsub), [Datafusion](./modules/datafusion), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag)

2
blueprints/data-solutions/data-platform-foundations/README.md
View File

@@ -222,7 +222,7 @@ module "data-platform" {
prefix = "myprefix"
}
# tftest modules=42 resources=314
# tftest modules=42 resources=315
```
## Customizations

2
blueprints/data-solutions/data-platform-foundations/variables.tf
View File

@@ -28,7 +28,7 @@ variable "composer_config" {
})
default = {
node_count = 3
airflow_version = "composer-1.17.5-airflow-2.1.4"
airflow_version = "composer-1-airflow-2"
env_variables = {}
}
}

8
fast/stages/02-networking-nva/data/firewall-rules/dev/rules.yaml
View File

@@ -4,8 +4,8 @@ ingress-allow-composer-nodes:
description: "Allow traffic to Composer nodes."
direction: INGRESS
action: allow
sources: []
ranges: ["0.0.0.0/0"]
sources:
- composer-worker
targets:
- composer-worker
use_service_accounts: false
@@ -17,8 +17,8 @@ ingress-allow-dataflow-load:
description: "Allow traffic to Dataflow nodes."
direction: INGRESS
action: allow
sources: []
ranges: ["0.0.0.0/0"]
sources:
- dataflow
targets:
- dataflow
use_service_accounts: false

8
fast/stages/02-networking-peering/data/firewall-rules/dev/rules.yaml
View File

@@ -4,8 +4,8 @@ ingress-allow-composer-nodes:
description: "Allow traffic to Composer nodes."
direction: INGRESS
action: allow
sources: []
ranges: ["0.0.0.0/0"]
sources:
- composer-worker
targets:
- composer-worker
use_service_accounts: false
@@ -17,8 +17,8 @@ ingress-allow-dataflow-load:
description: "Allow traffic to Dataflow nodes."
direction: INGRESS
action: allow
sources: []
ranges: ["0.0.0.0/0"]
sources:
- dataflow
targets:
- dataflow
use_service_accounts: false

8
fast/stages/02-networking-vpn/data/firewall-rules/dev/rules.yaml
View File

@@ -4,8 +4,8 @@ ingress-allow-composer-nodes:
description: "Allow traffic to Composer nodes."
direction: INGRESS
action: allow
sources: []
ranges: ["0.0.0.0/0"]
sources:
- composer-worker
targets:
- composer-worker
use_service_accounts: false
@@ -17,8 +17,8 @@ ingress-allow-dataflow-load:
description: "Allow traffic to Dataflow nodes."
direction: INGRESS
action: allow
sources: []
ranges: ["0.0.0.0/0"]
sources:
- dataflow
targets:
- dataflow
use_service_accounts: false

1
fast/stages/03-data-platform/dev/demo Symbolic link
View File

@@ -0,0 +1 @@
../../../../blueprints/data-solutions/data-platform-foundations/demo/

1
modules/README.md
View File

@@ -32,7 +32,6 @@ These modules are used in the examples included in this repository. If you are u
- [billing budget](./billing-budget)
- [folder](./folder)
- [logging bucket](./logging-bucket)
- [naming convention](./naming-convention)
- [organization](./organization)
- [project](./project)
- [projects-data-source](./projects-data-source)

9
modules/cloudsql-instance/main.tf
View File

@@ -18,10 +18,11 @@ locals {
prefix = var.prefix == null ? "" : "${var.prefix}-"
is_mysql = can(regex("^MYSQL", var.database_version))
has_replicas = try(length(var.replicas) > 0, false)
is_regional = var.availability_type == "REGIONAL" ? true : false
// Enable backup if the user asks for it or if the user is deploying
// MySQL with replicas
enable_backup = var.backup_configuration.enabled || (local.is_mysql && local.has_replicas)
// MySQL in HA configuration (regional or with specified replicas)
enable_backup = var.backup_configuration.enabled || (local.is_mysql && local.has_replicas) || (local.is_mysql && local.is_regional)
users = {
for user, password in coalesce(var.users, {}) :
@@ -76,11 +77,11 @@ resource "google_sql_database_instance" "primary" {
content {
enabled = true
// enable binary log if the user asks for it or we have replicas,
// enable binary log if the user asks for it or we have replicas (default in regional),
// but only for MySQL
binary_log_enabled = (
local.is_mysql
? var.backup_configuration.binary_log_enabled || local.has_replicas
? var.backup_configuration.binary_log_enabled || local.has_replicas || local.is_regional
: null
)
start_time = var.backup_configuration.start_time

57
modules/folders-unit/README.md
View File

@@ -1,57 +0,0 @@
# Google Cloud Unit Folders Module
This module allows creation and management of an organizational hierarchy "unit" composed of a parent folder (usually mapped to a business unit or team), and a set of child folders (usually mapped to environments) each with a corresponding set of service accounts, IAM bindings and GCS buckets.
## Example
```hcl
module "folders-unit" {
source = "./fabric/modules/folders-unit"
name = "Business Intelligence"
short_name = "bi"
automation_project_id = "automation-project-394yr923811"
billing_account_id = "015617-16GHBC-AF02D9"
organization_id = "506128240800"
root_node = "folders/93469270123701"
prefix = "unique-prefix"
environments = {
dev = "Development",
test = "Testing",
prod = "Production"
}
service_account_keys = true
}
# tftest modules=1 resources=37
```
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [automation_project_id](variables.tf#L17) | Project id used for automation service accounts. | <code>string</code> | ✓ | |
| [billing_account_id](variables.tf#L22) | Country billing account account. | <code>string</code> | ✓ | |
| [name](variables.tf#L86) | Top folder name. | <code>string</code> | ✓ | |
| [organization_id](variables.tf#L91) | Organization id in organizations/nnnnnn format. | <code>string</code> | ✓ | |
| [root_node](variables.tf#L102) | Root node in folders/folder_id or organizations/org_id format. | <code>string</code> | ✓ | |
| [short_name](variables.tf#L113) | Short name used as GCS bucket and service account prefixes, do not use capital letters or spaces. | <code>string</code> | ✓ | |
| [environments](variables.tf#L27) | Unit environments short names. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; non-prod &#61; &#34;Non production&#34;&#10; prod &#61; &#34;Production&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [gcs_defaults](variables.tf#L36) | Defaults use for the state GCS buckets. | <code>map&#40;string&#41;</code> | | <code title="&#123;&#10; location &#61; &#34;EU&#34;&#10; storage_class &#61; &#34;MULTI_REGIONAL&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [iam](variables.tf#L45) | IAM bindings for the top-level folder in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [iam_billing_config](variables.tf#L51) | Grant billing user role to service accounts, defaults to granting on the billing account. | <code title="object&#40;&#123;&#10; grant &#61; bool&#10; target_org &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; grant &#61; true&#10; target_org &#61; false&#10;&#125;">&#123;&#8230;&#125;</code> |
| [iam_enviroment_roles](variables.tf#L63) | IAM roles granted to the environment service account on the environment sub-folder. | <code>list&#40;string&#41;</code> | | <code title="&#91;&#10; &#34;roles&#47;compute.networkAdmin&#34;,&#10; &#34;roles&#47;owner&#34;,&#10; &#34;roles&#47;resourcemanager.folderAdmin&#34;,&#10; &#34;roles&#47;resourcemanager.projectCreator&#34;,&#10;&#93;">&#91;&#8230;&#93;</code> |
| [iam_xpn_config](variables.tf#L74) | Grant Shared VPC creation roles to service accounts, defaults to granting at folder level. | <code title="object&#40;&#123;&#10; grant &#61; bool&#10; target_org &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; grant &#61; true&#10; target_org &#61; false&#10;&#125;">&#123;&#8230;&#125;</code> |
| [prefix](variables.tf#L96) | Optional prefix used for GCS bucket names to ensure uniqueness. | <code>string</code> | | <code>null</code> |
| [service_account_keys](variables.tf#L107) | Generate and store service account keys in the state file. | <code>bool</code> | | <code>false</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [env_folders](outputs.tf#L17) | Unit environments folders. | |
| [env_gcs_buckets](outputs.tf#L28) | Unit environments tfstate gcs buckets. | |
| [env_sa_keys](outputs.tf#L36) | Unit environments service account keys. | ✓ |
| [env_service_accounts](outputs.tf#L45) | Unit environments service accounts. | |
| [unit_folder](outputs.tf#L53) | Unit top level folder. | |
<!-- END TFDOC -->

58
modules/folders-unit/locals.tf
View File

@@ -1,58 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
folder_roles = concat(var.iam_enviroment_roles, local.sa_xpn_folder_roles)
iam = var.iam == null ? {} : var.iam
folder_iam_service_account_bindings = {
for pair in setproduct(keys(var.environments), local.folder_roles) :
"${pair.0}-${pair.1}" => { environment = pair.0, role = pair.1 }
}
org_iam_service_account_bindings = {
for pair in setproduct(keys(var.environments), concat(
local.sa_xpn_org_roles,
local.sa_billing_org_roles,
local.sa_billing_org_roles)) :
"${pair.0}-${pair.1}" => { environment = pair.0, role = pair.1 }
}
billing_iam_service_account_bindings = {
for pair in setproduct(keys(var.environments), local.sa_billing_account_roles) :
"${pair.0}-${pair.1}" => { environment = pair.0, role = pair.1 }
}
service_accounts = {
for key, sa in google_service_account.environment :
key => "serviceAccount:${sa.email}"
}
sa_billing_account_roles = (
var.iam_billing_config.target_org ? [] : ["roles/billing.user"]
)
sa_billing_org_roles = (
!var.iam_billing_config.target_org ? [] : ["roles/billing.user"]
)
sa_xpn_folder_roles = (
local.sa_xpn_target_org ? [] : ["roles/compute.xpnAdmin"]
)
sa_xpn_org_roles = (
local.sa_xpn_target_org
? ["roles/compute.xpnAdmin", "roles/resourcemanager.organizationViewer"]
: ["roles/resourcemanager.organizationViewer"]
)
sa_xpn_target_org = (
var.iam_xpn_config.target_org
||
substr(var.root_node, 0, 13) == "organizations"
)
}

109
modules/folders-unit/main.tf
View File

@@ -1,109 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
organization_id = element(split("/", var.organization_id), 1)
}
###############################################################################
# Folders and folder IAM #
###############################################################################
resource "google_folder" "unit" {
display_name = var.name
parent = var.root_node
}
resource "google_folder" "environment" {
for_each = var.environments
display_name = each.value
parent = google_folder.unit.name
}
resource "google_folder_iam_binding" "unit" {
for_each = var.iam
folder = google_folder.unit.name
role = each.key
members = each.value
}
resource "google_folder_iam_binding" "environment" {
for_each = local.folder_iam_service_account_bindings
folder = google_folder.environment[each.value.environment].name
role = each.value.role
members = [local.service_accounts[each.value.environment]]
}
###############################################################################
# Billing account and org IAM #
###############################################################################
resource "google_organization_iam_member" "org_iam_member" {
for_each = local.org_iam_service_account_bindings
org_id = local.organization_id
role = each.value.role
member = local.service_accounts[each.value.environment]
}
resource "google_billing_account_iam_member" "billing_iam_member" {
for_each = var.iam_billing_config.grant ? local.billing_iam_service_account_bindings : {}
billing_account_id = var.billing_account_id
role = each.value.role
member = local.service_accounts[each.value.environment]
}
################################################################################
# Service Accounts #
################################################################################
resource "google_service_account" "environment" {
for_each = var.environments
project = var.automation_project_id
account_id = "${var.short_name}-${each.key}"
display_name = "${var.short_name} ${each.key} (Terraform managed)."
}
resource "google_service_account_key" "keys" {
for_each = var.service_account_keys ? var.environments : {}
service_account_id = google_service_account.environment[each.key].email
}
################################################################################
# GCS and GCS IAM #
################################################################################
resource "google_storage_bucket" "tfstate" {
for_each = var.environments
project = var.automation_project_id
name = join("", [
var.prefix == null ? "" : "${var.prefix}-",
"${var.short_name}-${each.key}-tf"
])
location = var.gcs_defaults.location
storage_class = var.gcs_defaults.storage_class
force_destroy = false
uniform_bucket_level_access = true
versioning {
enabled = true
}
}
resource "google_storage_bucket_iam_binding" "bindings" {
for_each = var.environments
bucket = google_storage_bucket.tfstate[each.key].name
role = "roles/storage.objectAdmin"
members = [local.service_accounts[each.key]]
}

59
modules/folders-unit/outputs.tf
View File

@@ -1,59 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "env_folders" {
description = "Unit environments folders."
value = {
for key, folder in google_folder.environment
: key => {
id = folder.name,
name = folder.display_name
}
}
}
output "env_gcs_buckets" {
description = "Unit environments tfstate gcs buckets."
value = {
for key, bucket in google_storage_bucket.tfstate
: key => bucket.name
}
}
output "env_sa_keys" {
description = "Unit environments service account keys."
sensitive = true
value = {
for key, sa_key in google_service_account_key.keys :
key => sa_key.private_key
}
}
output "env_service_accounts" {
description = "Unit environments service accounts."
value = {
for key, sa in google_service_account.environment
: key => sa.email
}
}
output "unit_folder" {
description = "Unit top level folder."
value = {
id = google_folder.unit.name,
name = google_folder.unit.display_name
}
}

116
modules/folders-unit/variables.tf
View File

@@ -1,116 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "automation_project_id" {
description = "Project id used for automation service accounts."
type = string
}
variable "billing_account_id" {
description = "Country billing account account."
type = string
}
variable "environments" {
description = "Unit environments short names."
type = map(string)
default = {
non-prod = "Non production"
prod = "Production"
}
}
variable "gcs_defaults" {
description = "Defaults use for the state GCS buckets."
type = map(string)
default = {
location = "EU"
storage_class = "MULTI_REGIONAL"
}
}
variable "iam" {
description = "IAM bindings for the top-level folder in {ROLE => [MEMBERS]} format."
type = map(list(string))
default = {}
}
variable "iam_billing_config" {
description = "Grant billing user role to service accounts, defaults to granting on the billing account."
type = object({
grant = bool
target_org = bool
})
default = {
grant = true
target_org = false
}
}
variable "iam_enviroment_roles" {
description = "IAM roles granted to the environment service account on the environment sub-folder."
type = list(string)
default = [
"roles/compute.networkAdmin",
"roles/owner",
"roles/resourcemanager.folderAdmin",
"roles/resourcemanager.projectCreator",
]
}
variable "iam_xpn_config" {
description = "Grant Shared VPC creation roles to service accounts, defaults to granting at folder level."
type = object({
grant = bool
target_org = bool
})
default = {
grant = true
target_org = false
}
}
variable "name" {
description = "Top folder name."
type = string
}
variable "organization_id" {
description = "Organization id in organizations/nnnnnn format."
type = string
}
variable "prefix" {
description = "Optional prefix used for GCS bucket names to ensure uniqueness."
type = string
default = null
}
variable "root_node" {
description = "Root node in folders/folder_id or organizations/org_id format."
type = string
}
variable "service_account_keys" {
description = "Generate and store service account keys in the state file."
type = bool
default = false
}
variable "short_name" {
description = "Short name used as GCS bucket and service account prefixes, do not use capital letters or spaces."
type = string
}

29
modules/folders-unit/versions.tf
View File

@@ -1,29 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
required_version = ">= 1.1.0"
required_providers {
google = {
source = "hashicorp/google"
version = ">= 4.32.0" # tftest
}
google-beta = {
source = "hashicorp/google-beta"
version = ">= 4.32.0" # tftest
}
}
}

59
modules/gke-cluster/README.md
View File

@@ -68,13 +68,13 @@ module "cluster-1" {
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [location](variables.tf#L155) | Cluster zone or region. | <code>string</code> | ✓ | |
| [name](variables.tf#L222) | Cluster name. | <code>string</code> | ✓ | |
| [network](variables.tf#L227) | Name or self link of the VPC used for the cluster. Use the self link for Shared VPC. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L271) | Cluster project id. | <code>string</code> | ✓ | |
| [secondary_range_pods](variables.tf#L294) | Subnet secondary range name used for pods. | <code>string</code> | ✓ | |
| [secondary_range_services](variables.tf#L299) | Subnet secondary range name used for services. | <code>string</code> | ✓ | |
| [subnetwork](variables.tf#L304) | VPC subnetwork name or self link. | <code>string</code> | ✓ | |
| [location](variables.tf#L161) | Cluster zone or region. | <code>string</code> | ✓ | |
| [name](variables.tf#L228) | Cluster name. | <code>string</code> | ✓ | |
| [network](variables.tf#L233) | Name or self link of the VPC used for the cluster. Use the self link for Shared VPC. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L277) | Cluster project id. | <code>string</code> | ✓ | |
| [secondary_range_pods](variables.tf#L300) | Subnet secondary range name used for pods. | <code>string</code> | ✓ | |
| [secondary_range_services](variables.tf#L305) | Subnet secondary range name used for services. | <code>string</code> | ✓ | |
| [subnetwork](variables.tf#L310) | VPC subnetwork name or self link. | <code>string</code> | ✓ | |
| [addons](variables.tf#L17) | Addons enabled in the cluster (true means enabled). | <code title="object&#40;&#123;&#10; cloudrun_config &#61; bool&#10; dns_cache_config &#61; bool&#10; horizontal_pod_autoscaling &#61; bool&#10; http_load_balancing &#61; bool&#10; istio_config &#61; object&#40;&#123;&#10; enabled &#61; bool&#10; tls &#61; bool&#10; &#125;&#41;&#10; network_policy_config &#61; bool&#10; gce_persistent_disk_csi_driver_config &#61; bool&#10; gcp_filestore_csi_driver_config &#61; bool&#10; config_connector_config &#61; bool&#10; kalm_config &#61; bool&#10; gke_backup_agent_config &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; cloudrun_config &#61; false&#10; dns_cache_config &#61; false&#10; horizontal_pod_autoscaling &#61; true&#10; http_load_balancing &#61; true&#10; istio_config &#61; &#123;&#10; enabled &#61; false&#10; tls &#61; false&#10; &#125;&#10; network_policy_config &#61; false&#10; gce_persistent_disk_csi_driver_config &#61; false&#10; gcp_filestore_csi_driver_config &#61; false&#10; config_connector_config &#61; false&#10; kalm_config &#61; false&#10; gke_backup_agent_config &#61; false&#10;&#125;">&#123;&#8230;&#125;</code> |
| [authenticator_security_group](variables.tf#L53) | RBAC security group for Google Groups for GKE, format is gke-security-groups@yourdomain.com. | <code>string</code> | | <code>null</code> |
| [cluster_autoscaling](variables.tf#L59) | Enable and configure limits for Node Auto-Provisioning with Cluster Autoscaler. | <code title="object&#40;&#123;&#10; enabled &#61; bool&#10; cpu_min &#61; number&#10; cpu_max &#61; number&#10; memory_min &#61; number&#10; memory_max &#61; number&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; enabled &#61; false&#10; cpu_min &#61; 0&#10; cpu_max &#61; 0&#10; memory_min &#61; 0&#10; memory_max &#61; 0&#10;&#125;">&#123;&#8230;&#125;</code> |
@@ -83,28 +83,29 @@ module "cluster-1" {
| [description](variables.tf#L97) | Cluster description. | <code>string</code> | | <code>null</code> |
| [dns_config](variables.tf#L103) | Configuration for Using Cloud DNS for GKE. | <code title="object&#40;&#123;&#10; cluster_dns &#61; string&#10; cluster_dns_scope &#61; string&#10; cluster_dns_domain &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [enable_autopilot](variables.tf#L113) | Create cluster in autopilot mode. With autopilot there's no need to create node-pools and some features are not supported (e.g. setting default_max_pods_per_node). | <code>bool</code> | | <code>false</code> |
| [enable_dataplane_v2](variables.tf#L119) | Enable Dataplane V2 on the cluster, will disable network_policy addons config. | <code>bool</code> | | <code>false</code> |
| [enable_intranode_visibility](variables.tf#L125) | Enable intra-node visibility to make same node pod to pod traffic visible. | <code>bool</code> | | <code>null</code> |
| [enable_l4_ilb_subsetting](variables.tf#L131) | Enable L4ILB Subsetting. | <code>bool</code> | | <code>null</code> |
| [enable_shielded_nodes](variables.tf#L137) | Enable Shielded Nodes features on all nodes in this cluster. | <code>bool</code> | | <code>null</code> |
| [enable_tpu](variables.tf#L143) | Enable Cloud TPU resources in this cluster. | <code>bool</code> | | <code>null</code> |
| [labels](variables.tf#L149) | Cluster resource labels. | <code>map&#40;string&#41;</code> | | <code>null</code> |
| [logging_config](variables.tf#L160) | Logging configuration (enabled components). | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [logging_service](variables.tf#L166) | Logging service (disable with an empty string). | <code>string</code> | | <code>&#34;logging.googleapis.com&#47;kubernetes&#34;</code> |
| [maintenance_config](variables.tf#L172) | Maintenance window configuration. | <code title="object&#40;&#123;&#10; daily_maintenance_window &#61; object&#40;&#123;&#10; start_time &#61; string&#10; &#125;&#41;&#10; recurring_window &#61; object&#40;&#123;&#10; start_time &#61; string&#10; end_time &#61; string&#10; recurrence &#61; string&#10; &#125;&#41;&#10; maintenance_exclusion &#61; list&#40;object&#40;&#123;&#10; exclusion_name &#61; string&#10; start_time &#61; string&#10; end_time &#61; string&#10; &#125;&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; daily_maintenance_window &#61; &#123;&#10; start_time &#61; &#34;03:00&#34;&#10; &#125;&#10; recurring_window &#61; null&#10; maintenance_exclusion &#61; &#91;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [master_authorized_ranges](variables.tf#L198) | External Ip address ranges that can access the Kubernetes cluster master through HTTPS. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [min_master_version](variables.tf#L204) | Minimum version of the master, defaults to the version of the most recent official release. | <code>string</code> | | <code>null</code> |
| [monitoring_config](variables.tf#L210) | Monitoring configuration (enabled components). | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [monitoring_service](variables.tf#L216) | Monitoring service (disable with an empty string). | <code>string</code> | | <code>&#34;monitoring.googleapis.com&#47;kubernetes&#34;</code> |
| [node_locations](variables.tf#L232) | Zones in which the cluster's nodes are located. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [notification_config](variables.tf#L238) | GKE Cluster upgrade notifications via PubSub. | <code>bool</code> | | <code>false</code> |
| [peering_config](variables.tf#L244) | Configure peering with the master VPC for private clusters. | <code title="object&#40;&#123;&#10; export_routes &#61; bool&#10; import_routes &#61; bool&#10; project_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [pod_security_policy](variables.tf#L254) | Enable the PodSecurityPolicy feature. | <code>bool</code> | | <code>null</code> |
| [private_cluster_config](variables.tf#L260) | Enable and configure private cluster, private nodes must be true if used. | <code title="object&#40;&#123;&#10; enable_private_nodes &#61; bool&#10; enable_private_endpoint &#61; bool&#10; master_ipv4_cidr_block &#61; string&#10; master_global_access &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [release_channel](variables.tf#L276) | Release channel for GKE upgrades. | <code>string</code> | | <code>null</code> |
| [resource_usage_export_config](variables.tf#L282) | Configure the ResourceUsageExportConfig feature. | <code title="object&#40;&#123;&#10; enabled &#61; bool&#10; dataset &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; enabled &#61; null&#10; dataset &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [vertical_pod_autoscaling](variables.tf#L309) | Enable the Vertical Pod Autoscaling feature. | <code>bool</code> | | <code>null</code> |
| [workload_identity](variables.tf#L315) | Enable the Workload Identity feature. | <code>bool</code> | | <code>true</code> |
| [enable_binary_authorization](variables.tf#L119) | Enable Google Binary Authorization. | <code>bool</code> | | <code>false</code> |
| [enable_dataplane_v2](variables.tf#L125) | Enable Dataplane V2 on the cluster, will disable network_policy addons config. | <code>bool</code> | | <code>false</code> |
| [enable_intranode_visibility](variables.tf#L131) | Enable intra-node visibility to make same node pod to pod traffic visible. | <code>bool</code> | | <code>null</code> |
| [enable_l4_ilb_subsetting](variables.tf#L137) | Enable L4ILB Subsetting. | <code>bool</code> | | <code>null</code> |
| [enable_shielded_nodes](variables.tf#L143) | Enable Shielded Nodes features on all nodes in this cluster. | <code>bool</code> | | <code>null</code> |
| [enable_tpu](variables.tf#L149) | Enable Cloud TPU resources in this cluster. | <code>bool</code> | | <code>null</code> |
| [labels](variables.tf#L155) | Cluster resource labels. | <code>map&#40;string&#41;</code> | | <code>null</code> |
| [logging_config](variables.tf#L166) | Logging configuration (enabled components). | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [logging_service](variables.tf#L172) | Logging service (disable with an empty string). | <code>string</code> | | <code>&#34;logging.googleapis.com&#47;kubernetes&#34;</code> |
| [maintenance_config](variables.tf#L178) | Maintenance window configuration. | <code title="object&#40;&#123;&#10; daily_maintenance_window &#61; object&#40;&#123;&#10; start_time &#61; string&#10; &#125;&#41;&#10; recurring_window &#61; object&#40;&#123;&#10; start_time &#61; string&#10; end_time &#61; string&#10; recurrence &#61; string&#10; &#125;&#41;&#10; maintenance_exclusion &#61; list&#40;object&#40;&#123;&#10; exclusion_name &#61; string&#10; start_time &#61; string&#10; end_time &#61; string&#10; &#125;&#41;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; daily_maintenance_window &#61; &#123;&#10; start_time &#61; &#34;03:00&#34;&#10; &#125;&#10; recurring_window &#61; null&#10; maintenance_exclusion &#61; &#91;&#93;&#10;&#125;">&#123;&#8230;&#125;</code> |
| [master_authorized_ranges](variables.tf#L204) | External Ip address ranges that can access the Kubernetes cluster master through HTTPS. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [min_master_version](variables.tf#L210) | Minimum version of the master, defaults to the version of the most recent official release. | <code>string</code> | | <code>null</code> |
| [monitoring_config](variables.tf#L216) | Monitoring configuration (enabled components). | <code>list&#40;string&#41;</code> | | <code>null</code> |
| [monitoring_service](variables.tf#L222) | Monitoring service (disable with an empty string). | <code>string</code> | | <code>&#34;monitoring.googleapis.com&#47;kubernetes&#34;</code> |
| [node_locations](variables.tf#L238) | Zones in which the cluster's nodes are located. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [notification_config](variables.tf#L244) | GKE Cluster upgrade notifications via PubSub. | <code>bool</code> | | <code>false</code> |
| [peering_config](variables.tf#L250) | Configure peering with the master VPC for private clusters. | <code title="object&#40;&#123;&#10; export_routes &#61; bool&#10; import_routes &#61; bool&#10; project_id &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [pod_security_policy](variables.tf#L260) | Enable the PodSecurityPolicy feature. | <code>bool</code> | | <code>null</code> |
| [private_cluster_config](variables.tf#L266) | Enable and configure private cluster, private nodes must be true if used. | <code title="object&#40;&#123;&#10; enable_private_nodes &#61; bool&#10; enable_private_endpoint &#61; bool&#10; master_ipv4_cidr_block &#61; string&#10; master_global_access &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [release_channel](variables.tf#L282) | Release channel for GKE upgrades. | <code>string</code> | | <code>null</code> |
| [resource_usage_export_config](variables.tf#L288) | Configure the ResourceUsageExportConfig feature. | <code title="object&#40;&#123;&#10; enabled &#61; bool&#10; dataset &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; enabled &#61; null&#10; dataset &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [vertical_pod_autoscaling](variables.tf#L315) | Enable the Vertical Pod Autoscaling feature. | <code>bool</code> | | <code>null</code> |
| [workload_identity](variables.tf#L321) | Enable the Workload Identity feature. | <code>bool</code> | | <code>true</code> |
## Outputs

7
modules/gke-cluster/main.tf
View File

@@ -292,6 +292,13 @@ resource "google_container_cluster" "cluster" {
}
}
dynamic "binary_authorization" {
for_each = var.enable_binary_authorization ? [""] : []
content {
evaluation_mode = "PROJECT_SINGLETON_POLICY_ENFORCE"
}
}
dynamic "dns_config" {
for_each = var.dns_config != null ? [""] : []
content {

6
modules/gke-cluster/variables.tf
View File

@@ -116,6 +116,12 @@ variable "enable_autopilot" {
default = false
}
variable "enable_binary_authorization" {
description = "Enable Google Binary Authorization."
type = bool
default = false
}
variable "enable_dataplane_v2" {
description = "Enable Dataplane V2 on the cluster, will disable network_policy addons config."
type = bool

145
modules/iot-core/README.md
View File

@@ -1,145 +0,0 @@
# Google Cloud IoT Core Module
This module sets up Cloud IoT Core Registry, registers IoT Devices and configures Pub/Sub topics required in Cloud IoT Core.
To use this module, ensure the following APIs are enabled:
* pubsub.googleapis.com
* cloudiot.googleapis.com
## Simple Example
Basic example showing how to create an IoT Platform (IoT Core), connected to a set of given Pub/Sub topics and provision IoT devices.
Devices certificates must exist before calling this module. You can generate these certificates using the following command
```
openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem -nodes -out rsa_cert.pem -subj "/CN=unused"
```
And then provision public certificate path, together with the rest of device configuration in a devices yaml file following the following format
```yaml
device_id: # id of your IoT Device
is_blocked: # false to allow device connection with IoT Registry
is_gateway: # true to indicate the device connecting acts as a gateway for other IoT Devices
log_level: # device logs level
certificate_file: # public certificate path, generated as explained in the previous step
certificate_format: # Certificates format values are RSA_PEM, RSA_X509_PEM, ES256_PEM, and ES256_X509_PEM
```
Example Device config yaml configuration
```yaml
device_1:
is_blocked: false
is_gateway: false
log_level: INFO
certificate_file: device_certs/rsa_cert5.pem
certificate_format: RSA_X509_PEM
device_2:
is_blocked: true
is_gateway: false
log_level: INFO
certificate_file: device_certs/rsa_cert5.pem
certificate_format: RSA_X509_PEM
```
```hcl
module "iot-platform" {
source = "./fabric/modules/iot-core"
project_id = "my_project_id"
region = "europe-west1"
telemetry_pubsub_topic_id = "telemetry_topic_id"
status_pubsub_topic_id = "status_topic_id"
protocols = {
http = false,
mqtt = true
}
devices_config_directory = "./devices_config_folder"
}
# tftest:skip
```
Now, we can test sending telemetry messages from devices to our IoT Platform, for example using the MQTT demo client at https://github.com/googleapis/nodejs-iot/tree/main/samples/mqtt_example
## Example with specific PubSub topics for custom MQTT topics
If you need to match specific MQTT topics (eg, /temperature) into specific PubSub topics, you can use extra_telemetry_pubsub_topic_ids for that, as in the following example:
```hcl
module "iot-platform" {
source = "./fabric/modules/iot-core"
project_id = "my_project_id"
region = "europe-west1"
telemetry_pubsub_topic_id = "telemetry_topic_id"
status_pubsub_topic_id = "status_topic_id"
extra_telemetry_pubsub_topic_ids = {
"temperature" = "temp_topic_id",
"humidity" = "hum_topic_id"
}
protocols = {
http = false,
mqtt = true
}
devices_config_directory = "./devices_config_folder"
}
# tftest:skip
```
## Example integrated with Data Foundation Platform
In this example, we will show how to extend the **[Data Foundations Platform](../../blueprints/data-solutions/data-platform-foundations/)** to include IoT Platform as a new source of data.
![Target architecture](./diagram_iot.png)
1. First, we will setup Environment following instructions in **[Environment Setup](../../blueprints/data-solutions/data-platform-foundations/)** to setup projects and SAs required. Get output variable project_ids.landing as will be used later
1. Second, execute instructions in **[Environment Setup](../../blueprints/data-solutions/data-platform-foundations/)** to provision PubSub, DataFlow, BQ,... Get variable landing-pubsub as will be used later to create IoT Registry
1. Now it is time to provision IoT Platform. Modify landing-project-id and landing_pubsub_topic_id with output variables obtained before. Create device certificates as shown in the Simple Example and register them in devices.yaml file together with deviceids.
```hcl
module "iot-platform" {
source = "./fabric/modules/iot-core"
project_id = "landing-project-id"
region = "europe-west1"
telemetry_pubsub_topic_id = "landing_pubsub_topic_id"
status_pubsub_topic_id = "status_pubsub_topic_id"
protocols = {
http = false,
mqtt = true
}
devices_config_directory = "./devices_config_folder"
}
# tftest:skip
```
1. After that, we can setup the pipeline "PubSub to BigQuery" shown at **[Pipeline Setup](../../blueprints/data-solutions/data-platform-foundations/)**
1. Finally, instead of testing the pipeline by sending messages to PubSub, we can now test sending telemetry messages from simulated IoT devices to our IoT Platform, for example using the MQTT demo client at https://github.com/googleapis/nodejs-iot/tree/main/samples/mqtt_example . We shall edit the client script cloudiot_mqtt_example_nodejs.js to send messages following the pipeline message format, so they are processed by DataFlow job and inserted in the BigQuery table.
```
const payload = '{"name": "device4", "surname": "NA", "timestamp":"'+Math.floor(Date.now()/1000)+'"}';
```
Or even better, create a new BigQuery table with our IoT sensors data columns and modify the DataFlow job to push data to it.
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [devices_config_directory](variables.tf#L17) | Path to folder where devices configs are stored in yaml format. Folder may include subfolders with configuration files. Files suffix must be `.yaml`. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L34) | Project were resources will be deployed | <code>string</code> | ✓ | |
| [region](variables.tf#L48) | Region were resources will be deployed | <code>string</code> | ✓ | |
| [status_pubsub_topic_id](variables.tf#L59) | pub sub topic for status messages (GCP-->Device) | <code>string</code> | ✓ | |
| [telemetry_pubsub_topic_id](variables.tf#L64) | pub sub topic for telemetry messages (Device-->GCP) | <code>string</code> | ✓ | |
| [extra_telemetry_pubsub_topic_ids](variables.tf#L22) | additional pubsub topics linked to adhoc MQTT topics (Device-->GCP) in the format MQTT_TOPIC: PUBSUB_TOPIC_ID | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [log_level](variables.tf#L28) | IoT Registry Log level | <code>string</code> | | <code>&#34;INFO&#34;</code> |
| [protocols](variables.tf#L39) | IoT protocols (HTTP / MQTT) activation | <code title="object&#40;&#123;&#10; http &#61; bool,&#10; mqtt &#61; bool&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123; http &#61; true, mqtt &#61; true &#125;</code> |
| [registry_name](variables.tf#L53) | Name for the IoT Core Registry | <code>string</code> | | <code>&#34;cloudiot-registry&#34;</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [iot_registry](outputs.tf#L17) | Cloud IoT Core Registry | |
<!-- END TFDOC -->

BIN
modules/iot-core/diagram.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 470 KiB

BIN
modules/iot-core/diagram_iot.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 52 KiB

95
modules/iot-core/main.tf
View File

@@ -1,95 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
devices_config_files = [
for config_file in fileset("${path.root}/${var.devices_config_directory}", "**/*.yaml") :
"${path.root}/${var.devices_config_directory}/${config_file}"
]
device_config = merge(
[
for config_file in local.devices_config_files :
try(yamldecode(file(config_file)), {})
]...
)
}
#---------------------------------------------------------
# Create IoT Core Registry
#---------------------------------------------------------
resource "google_cloudiot_registry" "registry" {
name = var.registry_name
project = var.project_id
region = var.region
dynamic "event_notification_configs" {
for_each = var.extra_telemetry_pubsub_topic_ids
content {
pubsub_topic_name = event_notification_configs.value
subfolder_matches = event_notification_configs.key
}
}
event_notification_configs {
pubsub_topic_name = var.telemetry_pubsub_topic_id
subfolder_matches = ""
}
state_notification_config = {
pubsub_topic_name = var.status_pubsub_topic_id
}
mqtt_config = {
mqtt_enabled_state = var.protocols.mqtt ? "MQTT_ENABLED" : "MQTT_DISABLED"
}
http_config = {
http_enabled_state = var.protocols.http ? "HTTP_ENABLED" : "HTTP_DISABLED"
}
log_level = var.log_level
}
#---------------------------------------------------------
# Create IoT Core Device
# certificate created using: openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem -nodes -out rsa_cert.pem -subj "/CN=unused"
#---------------------------------------------------------
resource "google_cloudiot_device" "device" {
for_each = local.device_config
name = each.key
registry = google_cloudiot_registry.registry.id
credentials {
public_key {
format = try(each.value.certificate_format, null)
key = try(file(each.value.certificate_file), null)
}
}
blocked = try(each.value.is_blocked, null)
log_level = try(each.value.log_level, null)
gateway_config {
gateway_type = try(each.value.is_gateway, null) ? "GATEWAY" : "NON_GATEWAY"
}
}

20
modules/iot-core/outputs.tf
View File

@@ -1,20 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "iot_registry" {
description = "Cloud IoT Core Registry"
value = google_cloudiot_registry.registry
}

67
modules/iot-core/variables.tf
View File

@@ -1,67 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "devices_config_directory" {
description = "Path to folder where devices configs are stored in yaml format. Folder may include subfolders with configuration files. Files suffix must be `.yaml`."
type = string
}
variable "extra_telemetry_pubsub_topic_ids" {
description = "additional pubsub topics linked to adhoc MQTT topics (Device-->GCP) in the format MQTT_TOPIC: PUBSUB_TOPIC_ID"
type = map(string)
default = {}
}
variable "log_level" {
description = "IoT Registry Log level"
type = string
default = "INFO"
}
variable "project_id" {
description = "Project were resources will be deployed"
type = string
}
variable "protocols" {
description = "IoT protocols (HTTP / MQTT) activation"
type = object({
http = bool,
mqtt = bool
})
default = { http = true, mqtt = true }
}
variable "region" {
description = "Region were resources will be deployed"
type = string
}
variable "registry_name" {
description = "Name for the IoT Core Registry"
type = string
default = "cloudiot-registry"
}
variable "status_pubsub_topic_id" {
description = "pub sub topic for status messages (GCP-->Device)"
type = string
}
variable "telemetry_pubsub_topic_id" {
description = "pub sub topic for telemetry messages (Device-->GCP)"
type = string
}

89
modules/naming-convention/README.md
View File

@@ -1,89 +0,0 @@
# Naming Convention Module
This module allows defining a naming convention in a single place, and enforcing it by pre-creating resource names based on a set of tokens (environment, team name, etc.).
It implements a fairly common naming convention with optional prefix or suffix, but is really meant to be forked, and modified to adapt to individual use cases: just replace the `environment` and `team` variables with whatever makes sense for you, and edit the few lines in `main.tf` marked by comments.
The module also supports labels, generating sets of per-resource labels that combine the passed in tokens with optional resource-level labels.
It's completely static, using no provider resources, so its outputs are safe to use where dynamic values are not supported, like in `for_each` statements.
## Example
In its default configuration, the module supports an option prefix and suffix, and two tokens: one for the environment, and one for the team name.
```hcl
module "names-org" {
source = "./fabric/modules/naming-convention"
prefix = "myco"
environment = "dev"
team = "cloud"
resources = {
bucket = ["tf-org", "tf-sec", "tf-log"]
project = ["tf", "sec", "log"]
}
labels = {
project = {
tf = {scope = "global"}
}
}
}
module "project-tf" {
source = "./fabric/modules/project"
# myco-cloud-dev-tf
name = module.names-org.names.project.tf
# { environment = "dev", scope = "global", team = "cloud" }
labels = module.names-org.labels.project.tf
}
```
You can also enable resource type naming, useful with some legacy CMDB setups. When doing this, resource type names become part of the final resource names and are usually shorted (e.g. `prj` instead of `project`):
```hcl
module "names-org" {
source = "./fabric/modules/naming-convention"
prefix = "myco"
environment = "dev"
team = "cloud"
resources = {
bkt = ["tf-org", "tf-sec", "tf-log"]
prj = ["tf", "sec", "log"]
}
labels = {
prj = {
tf = {scope = "global"}
}
}
use_resource_prefixes = true
}
module "project-tf" {
source = "./fabric/modules/project"
# prj-myco-cloud-dev-tf
name = module.names-org.names.prj.tf
}
```
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [environment](variables.tf#L17) | Environment abbreviation used in names and labels. | <code>string</code> | ✓ | |
| [resources](variables.tf#L34) | Short resource names by type. | <code>map&#40;list&#40;string&#41;&#41;</code> | ✓ | |
| [team](variables.tf#L51) | Team name. | <code>string</code> | ✓ | |
| [labels](variables.tf#L22) | Per-resource labels. | <code>map&#40;map&#40;map&#40;string&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [prefix](variables.tf#L28) | Optional name prefix. | <code>string</code> | | <code>null</code> |
| [separator_override](variables.tf#L39) | Optional separator override for specific resource types. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [suffix](variables.tf#L45) | Optional name suffix. | <code>string</code> | | <code>null</code> |
| [use_resource_prefixes](variables.tf#L56) | Prefix names with the resource type. | <code>bool</code> | | <code>false</code> |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [labels](outputs.tf#L17) | Per resource labels. | |
| [names](outputs.tf#L22) | Per resource names. | |
<!-- END TFDOC -->

46
modules/naming-convention/main.tf
View File

@@ -1,46 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
locals {
prefix = var.prefix == null ? "" : "${var.prefix}-"
suffix = var.suffix == null ? "" : "-${var.suffix}"
// merge common and per-resource labels
labels = {
for resource_type, resources in var.resources : resource_type => {
for name in resources : name => merge(
// change this line if you need different tokens
{ environment = var.environment, team = var.team },
try(var.labels[resource_type][name], {})
)
}
}
// create per-resource names by assembling tokens
names = {
for resource_type, resources in var.resources : resource_type => {
for name in resources : name => join(
try(var.separator_override[resource_type], "-"),
compact([
var.use_resource_prefixes ? resource_type : "",
var.prefix,
// change these lines if you need different tokens
var.team,
var.environment,
name,
var.suffix
]))
}
}
}

25
modules/naming-convention/outputs.tf
View File

@@ -1,25 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
output "labels" {
description = "Per resource labels."
value = local.labels
}
output "names" {
description = "Per resource names."
value = local.names
}

60
modules/naming-convention/variables.tf
View File

@@ -1,60 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "environment" {
description = "Environment abbreviation used in names and labels."
type = string
}
variable "labels" {
description = "Per-resource labels."
type = map(map(map(string)))
default = {}
}
variable "prefix" {
description = "Optional name prefix."
type = string
default = null
}
variable "resources" {
description = "Short resource names by type."
type = map(list(string))
}
variable "separator_override" {
description = "Optional separator override for specific resource types."
type = map(string)
default = {}
}
variable "suffix" {
description = "Optional name suffix."
type = string
default = null
}
variable "team" {
description = "Team name."
type = string
}
variable "use_resource_prefixes" {
description = "Prefix names with the resource type."
type = bool
default = false
}

29
modules/naming-convention/versions.tf
View File

@@ -1,29 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
required_version = ">= 1.1.0"
required_providers {
google = {
source = "hashicorp/google"
version = ">= 4.32.0" # tftest
}
google-beta = {
source = "hashicorp/google-beta"
version = ">= 4.32.0" # tftest
}
}
}

1
modules/project/service-accounts.tf
View File

@@ -67,6 +67,7 @@ locals {
}
)
service_accounts_jit_services = [
"artifactregistry.googleapis.com",
"cloudasset.googleapis.com",
"gkehub.googleapis.com",
"pubsub.googleapis.com",

2
tests/blueprints/data_solutions/data_platform_foundations/test_plan.py
View File

@@ -24,4 +24,4 @@ def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner(FIXTURES_DIR)
assert len(modules) == 41
assert len(resources) == 313
assert len(resources) == 314

2
tests/blueprints/gke/binauthz/test_plan.py
View File

@@ -16,4 +16,4 @@ def test_resources(e2e_plan_runner):
"Test that plan works and the numbers of resources is as expected."
modules, resources = e2e_plan_runner()
assert len(modules) == 13
assert len(resources) == 42
assert len(resources) == 43

11
tests/modules/cloudsql_instance/test_plan.py
View File

@@ -90,6 +90,17 @@ def test_mysql_replicas_enables_backup(plan_runner):
assert backup_config['binary_log_enabled']
def test_mysql_binary_log_for_regional(plan_runner):
"Test that the binary log will be enabled for regional MySQL DBs."
_, resources = plan_runner(database_version="MYSQL_8_0", availability_type="REGIONAL")
assert len(resources) == 1
primary = [r for r in resources if r['name'] == 'primary'][0]
backup_config = primary['values']['settings'][0]['backup_configuration'][0]
assert backup_config['enabled']
assert backup_config['binary_log_enabled']
def test_users(plan_runner):
"Test user creation."

13
tests/modules/naming_convention/__init__.py
View File

@@ -1,13 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

65
tests/modules/naming_convention/fixture/main.tf
View File

@@ -1,65 +0,0 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
module "test" {
source = "../../../../modules/naming-convention"
prefix = var.prefix
suffix = var.suffix
use_resource_prefixes = var.use_resource_prefixes
environment = "dev"
team = "cloud"
resources = {
bucket = ["tf-org", "tf-sec", "tf-log"]
dataset = ["foobar", "frobniz"]
project = ["tf", "sec", "log"]
}
labels = {
project = {
tf = { scope = "global" }
}
}
separator_override = {
dataset = "_"
}
}
output "labels" {
value = module.test.labels
}
output "names" {
value = module.test.names
}
variable "prefix" {
type = string
default = null
}
variable "separator_override" {
type = map(string)
default = {}
}
variable "suffix" {
type = string
default = null
}
variable "use_resource_prefixes" {
type = bool
default = false
}

47
tests/modules/naming_convention/test_plan.py
View File

@@ -1,47 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
def test_no_prefix_suffix(apply_runner):
_, output = apply_runner()
assert output['names']['project']['tf'] == 'cloud-dev-tf'
assert output['names']['bucket']['tf-org'] == 'cloud-dev-tf-org'
assert output['labels']['project']['tf'] == {
'environment': 'dev', 'scope': 'global', 'team': 'cloud'}
assert output['labels']['bucket']['tf-org'] == {
'environment': 'dev', 'team': 'cloud'}
def test_prefix(apply_runner):
_, output = apply_runner(prefix='myco')
assert output['names']['project']['tf'] == 'myco-cloud-dev-tf'
assert output['names']['bucket']['tf-org'] == 'myco-cloud-dev-tf-org'
def test_suffix(apply_runner):
_, output = apply_runner(suffix='myco')
assert output['names']['project']['tf'] == 'cloud-dev-tf-myco'
assert output['names']['bucket']['tf-org'] == 'cloud-dev-tf-org-myco'
def test_resource_prefix(apply_runner):
_, output = apply_runner(prefix='myco',
use_resource_prefixes='true')
assert output['names']['project']['tf'] == 'project-myco-cloud-dev-tf'
assert output['names']['bucket']['tf-org'] == 'bucket-myco-cloud-dev-tf-org'
def test_separator(apply_runner):
_, output = apply_runner(separator_override='{ dataset = "_" }')
assert output['names']['dataset'] == {
'foobar': 'cloud_dev_foobar', 'frobniz': 'cloud_dev_frobniz'}