merge fast-dev
This commit is contained in:
Ludo
@@ -8,7 +8,7 @@ Currently available blueprints:
|
||||
- **cloud operations** - [Active Directory Federation Services](./cloud-operations/adfs), [Cloud Asset Inventory feeds for resource change tracking and remediation](./cloud-operations/asset-inventory-feed-remediation), [Fine-grained Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Cloud DNS & Shared VPC design](./cloud-operations/dns-shared-vpc), [Delegated Role Grants](./cloud-operations/iam-delegated-role-grants), [Network Quota Monitoring](./cloud-operations/network-quota-monitoring), [Managing on-prem service account keys by uploading public keys](./cloud-operations/onprem-sa-key-management), [Compute Image builder with Hashicorp Packer](./cloud-operations/packer-image-builder), [Packer example](./cloud-operations/packer-image-builder/packer), [Compute Engine quota monitoring](./cloud-operations/compute-quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Configuring workload identity federation with Terraform Cloud/Enterprise workflows](./cloud-operations/terraform-cloud-dynamic-credentials), [TCP healthcheck and restart for unmanaged GCE instances](./cloud-operations/unmanaged-instances-healthcheck), [Migrate for Compute Engine (v5) blueprints](./cloud-operations/vm-migration), [Configuring workload identity federation to access Google Cloud resources from apps running on Azure](./cloud-operations/workload-identity-federation)
|
||||
- **data solutions** - [GCE and GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms), [Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key](./data-solutions/composer-2), [Cloud SQL instance with multi-region read replicas](./data-solutions/cloudsql-multiregion), [Data Platform](./data-solutions/data-platform-foundations), [Minimal Data Platform](./data-solutions/data-platform-minimal), [Spinning up a foundation data pipeline on Google Cloud using Cloud Storage, Dataflow and BigQuery](./data-solutions/gcs-to-bq-with-least-privileges), [#SQL Server Always On Groups blueprint](./data-solutions/sqlserver-alwayson), [Data Playground](./data-solutions/data-playground), [MLOps with Vertex AI](./data-solutions/vertex-mlops), [Shielded Folder](./data-solutions/shielded-folder), [BigQuery ML and Vertex AI Pipeline](./data-solutions/bq-ml)
|
||||
- **factories** - [Fabric resource factories](./factories)
|
||||
- **GKE** - [Binary Authorization Pipeline Blueprint](./gke/binauthz), [Storage API](./gke/binauthz/image), [Multi-cluster mesh on GKE (fleet API)](./gke/multi-cluster-mesh-gke-fleet-api), [GKE Multitenant Blueprint](./gke/multitenant-fleet), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [GKE Autopilot](./gke/autopilot)
|
||||
- **GKE** - [Binary Authorization Pipeline Blueprint](./gke/binauthz), [Storage API](./gke/binauthz/image), [Multi-cluster mesh on GKE (fleet API)](./gke/multi-cluster-mesh-gke-fleet-api), [GKE Multitenant](../fast/stages/3-gke-dev), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [GKE Autopilot](./gke/autopilot)
|
||||
- **networking** - [Calling a private Cloud Function from On-premises](./networking/private-cloud-function-from-onprem), [HA VPN over Interconnect](./networking/ha-vpn-over-interconnect/), [GLB and multi-regional daisy-chaining through hybrid NEGs](./networking/glb-hybrid-neg-internal), [Hybrid connectivity to on-premise services through PSC](./networking/psc-hybrid), [HTTP Load Balancer with Cloud Armor](./networking/glb-and-armor), [Internal Load Balancer as Next Hop](./networking/ilb-next-hop), On-prem DNS and Google Private Access, [PSC Producer](./networking/psc-hybrid/psc-producer), [PSC Consumer](./networking/psc-hybrid/psc-consumer), [Shared VPC with optional GKE cluster](./networking/shared-vpc-gke), [VPC Connectivity Lab](./networking/vpc-connectivity-lab/)
|
||||
- **SecOps** - [SecOps GKE Forwarder](./secops/secops-gke-forwarder)
|
||||
- **serverless** - [Cloud Run series](./serverless/cloud-run-explore)
|
||||
|
||||
@@ -23,7 +23,7 @@ The approach adapts to different high-level requirements:
|
||||
- least privilege principle
|
||||
- rely on service account impersonation
|
||||
|
||||
The code in this blueprint doesn't address Organization-level configurations (Organization policy, VPC-SC, centralized logs). We expect those elements to be managed by automation stages external to this script like those in [FAST](../../../fast) and this blueprint deployed on top of them as one of the [stages](../../../fast/stages/3-data-platform/dev/README.md).
|
||||
The code in this blueprint doesn't address Organization-level configurations (Organization policy, VPC-SC, centralized logs). We expect those elements to be managed by automation stages external to this script like those in [FAST](../../../fast).
|
||||
|
||||
### Project structure
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ The approach adapts to different high-level requirements:
|
||||
- least privilege principle
|
||||
- rely on service account impersonation
|
||||
|
||||
The code in this blueprint doesn't address Organization-level configurations (Organization policy, VPC-SC, centralized logs). We expect those elements to be managed by automation stages external to this script like those in [FAST](../../../fast) and this blueprint deployed on top of them as one of the [stages](../../../fast/stages/3-data-platform/dev/README.md).
|
||||
The code in this blueprint doesn't address Organization-level configurations (Organization policy, VPC-SC, centralized logs). We expect those elements to be managed by automation stages external to this script like those in [FAST](../../../fast).
|
||||
|
||||
## Project structure
|
||||
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
# GCVE Private Cloud Minimal
|
||||
|
||||
This blueprint presents an opinionated architecture to handle different Google VMware Engine deployment scenarios: from a simple single region private cloud to multi-region private clouds spread across different locations. The general idea behind this blueprint is to deploy a single project hosting one or more GCVE private clouds connected to a shared VMware Engine Network (VEN).
|
||||
Optionally this blueprint can deploy the VMWare Engine Network peerings to pre-existing VPCs.
|
||||
|
||||
Multiple deployments of this blueprint allow the user to achieve more complex design solutions as for example GCVE private clouds deployed on different projects or connected to independent VMWare Engine Networks.
|
||||
|
||||
This blueprint is used as part of the [FAST GCVE stage](../../../fast/stages/3-gcve/) but it can also be used independently if desired.
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GCVE single region private cloud">
|
||||
</p>
|
||||
|
||||
The blueprint manages:
|
||||
- project creation
|
||||
- project-level organization policy definitions
|
||||
- billing setup (billing account attachment)
|
||||
- API/services enablement
|
||||
- IAM role assignment for groups
|
||||
- VMware Engine private clouds creation
|
||||
- [VMware Engine Network](https://cloud.google.com/vmware-engine/docs/networking/vmware-engine-network#standard_networks) creation
|
||||
- VPC attachment (Optional)
|
||||
|
||||
### User groups
|
||||
|
||||
Based on our GCP best practices, a GCVE private cloud relies on user groups to assign roles to human identities. These are the specific groups bound to the main GCVE [predefined roles](https://cloud.google.com/vmware-engine/docs/iam#vmware-engine-roles):
|
||||
- *VMware Engine Administrators*. They have full access to the VMWare Engine Service.
|
||||
- *VMware Engine Viewers*. They have read-only access to the VMware Engine Service.
|
||||
|
||||
|
||||
### Network
|
||||
|
||||
This blueprint expects the user to provision a VPC upfront, either from one of the FAST networking stages (e.g. [Networking with separated single environment](../../../fast/stages/2-networking-c-separate-envs)) or from an external source.
|
||||
The blueprint can optionally configure the [VMware Engine Network peering](https://cloud.google.com/vmware-engine/docs/networking/peer-vpc-network) on the peer VPC by granting the following permissions on the project that hosts the VPC:
|
||||
- vmwareengine.networkPeerings.create
|
||||
- vmwareengine.networkPeerings.get
|
||||
- vmwareengine.networkPeerings.list
|
||||
- vmwareengine.operations.get
|
||||
The permissions can be assigned through the predefined role *vmwareengine.vmwareengineAdmin*. The creation of a dedicated custom role is strongly recommended anyway to comply with the least privilege principle.
|
||||
|
||||
## Basic usage
|
||||
|
||||
The following example shows how to deploy a CGVE private cloud and connect it to a VPC
|
||||
|
||||
```hcl
|
||||
module "gcve-pc" {
|
||||
source = "./fabric/blueprints/gcve/pc-minimal"
|
||||
billing_account_id = "000000-000000-000000"
|
||||
folder_id = "folders/000000000000"
|
||||
project_id = "myprojectid"
|
||||
groups = {
|
||||
gcp-gcve-admins = "group:gcp-gcve-admins@acme.com"
|
||||
gcp-gcve-viewers = "group:gcp-gcve-viewers@acme.com"
|
||||
}
|
||||
|
||||
prefix = "myprefix"
|
||||
|
||||
network_peerings = {
|
||||
dev-spoke-ven = {
|
||||
peer_network = "projects/spokeproject/regions/europe-west1/subnetworks/dev-default-ew1"
|
||||
peer_project_id = "peerprojectid"
|
||||
}
|
||||
}
|
||||
|
||||
private_cloud_configs = {
|
||||
dev-pc = {
|
||||
cidr = "172.26.16.0/22"
|
||||
zone = "europe-west1-a"
|
||||
management_cluster_config = {
|
||||
name = "mgmt-cluster"
|
||||
node_count = 1
|
||||
node_type_id = "standard-72"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest modules=3 resources=9
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Files
|
||||
|
||||
| name | description | modules | resources |
|
||||
|---|---|---|---|
|
||||
| [gcve-pc.tf](./gcve-pc.tf) | GCVE private cloud. | <code>gcve-private-cloud</code> | <code>google_vmwareengine_network_peering</code> |
|
||||
| [main.tf](./main.tf) | Project. | <code>project</code> | |
|
||||
| [output.tf](./output.tf) | Output variables. | | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [billing_account_id](variables.tf#L17) | Billing account ID. | <code>string</code> | ✓ | |
|
||||
| [folder_id](variables.tf#L22) | Folder used for the GCVE project in folders/nnnnnnnnnnn format. | <code>string</code> | ✓ | |
|
||||
| [groups](variables.tf#L27) | GCVE groups. | <code title="object({ gcp-gcve-admins = string gcp-gcve-viewers = string })">object({…})</code> | ✓ | |
|
||||
| [prefix](variables.tf#L81) | Prefix used for resource names. | <code>string</code> | ✓ | |
|
||||
| [private_cloud_configs](variables.tf#L90) | The VMware private cloud configurations. The key is the unique private cloud name suffix. | <code title="map(object({ cidr = string zone = string additional_cluster_configs = optional(map(object({ custom_core_count = optional(number) node_count = optional(number, 3) node_type_id = optional(string, "standard-72") })), {}) management_cluster_config = optional(object({ custom_core_count = optional(number) name = optional(string, "mgmt-cluster") node_count = optional(number, 3) node_type_id = optional(string, "standard-72") }), {}) description = optional(string, "Managed by Terraform.") }))">map(object({…}))</code> | ✓ | |
|
||||
| [project_id](variables.tf#L112) | ID of the project that will contain the GCVE private cloud. | <code>string</code> | ✓ | |
|
||||
| [iam](variables.tf#L36) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [iam_by_principals](variables.tf#L43) | Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [labels](variables.tf#L50) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [network_peerings](variables.tf#L56) | The network peerings between users' VPCs and the VMware Engine networks. The key is the peering name suffix. | <code title="map(object({ peer_network = string configure_peer_network = optional(bool, false) custom_routes = optional(object({ export_to_peer = optional(bool, false) import_from_peer = optional(bool, false) export_to_ven = optional(bool, false) import_from_ven = optional(bool, false) }), {}) custom_routes_with_public_ip = optional(object({ export_to_peer = optional(bool, false) import_from_peer = optional(bool, false) export_to_ven = optional(bool, false) import_from_ven = optional(bool, false) }), {}) description = optional(string, "Managed by Terraform.") peer_project_id = optional(string) peer_to_vmware_engine_network = optional(bool, false) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [project_services](variables.tf#L117) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> |
|
||||
<!-- END TFDOC -->
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 46 KiB |
@@ -1,58 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GCVE private cloud.
|
||||
locals {
|
||||
ven_peerings = {
|
||||
for k, v in var.network_peerings : k => {
|
||||
peer_network = v.peer_network
|
||||
description = v.description
|
||||
export_custom_routes = v.custom_routes.export_to_peer
|
||||
export_custom_routes_with_public_ip = v.custom_routes_with_public_ip.export_to_peer
|
||||
import_custom_routes = v.custom_routes.import_from_peer
|
||||
import_custom_routes_with_public_ip = v.custom_routes_with_public_ip.import_from_peer
|
||||
peer_to_vmware_engine_network = v.peer_to_vmware_engine_network
|
||||
}
|
||||
}
|
||||
}
|
||||
module "gcve-pc" {
|
||||
source = "../../../modules/gcve-private-cloud"
|
||||
prefix = var.prefix
|
||||
project_id = module.gcve-project-0.id
|
||||
|
||||
vmw_network_config = {
|
||||
create = true
|
||||
name = "default"
|
||||
}
|
||||
vmw_network_peerings = local.ven_peerings
|
||||
|
||||
vmw_private_cloud_configs = var.private_cloud_configs
|
||||
}
|
||||
|
||||
resource "google_vmwareengine_network_peering" "vmw_engine_network_peerings" {
|
||||
provider = google-beta
|
||||
for_each = { for k, v in var.network_peerings : k => v if v.configure_peer_network }
|
||||
peer_network = each.value.peer_network
|
||||
name = "${var.prefix}-${each.key}"
|
||||
description = each.value.description
|
||||
export_custom_routes = each.value.custom_routes.export_to_ven
|
||||
export_custom_routes_with_public_ip = each.value.custom_routes_with_public_ip.export_to_ven
|
||||
import_custom_routes = each.value.custom_routes.import_from_ven
|
||||
import_custom_routes_with_public_ip = each.value.custom_routes_with_public_ip.import_from_ven
|
||||
peer_network_type = "STANDARD"
|
||||
project = each.value.peer_project_id
|
||||
vmware_engine_network = module.gcve-pc.vmw_private_cloud_network.id
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Project.
|
||||
|
||||
module "gcve-project-0" {
|
||||
source = "../../../modules/project"
|
||||
billing_account = var.billing_account_id
|
||||
name = var.project_id
|
||||
parent = var.folder_id
|
||||
prefix = var.prefix
|
||||
iam_by_principals = merge({
|
||||
(var.groups.gcp-gcve-admins) = ["roles/vmwareengine.vmwareengineAdmin"]
|
||||
(var.groups.gcp-gcve-viewers) = ["roles/vmwareengine.vmwareengineViewer"]
|
||||
},
|
||||
var.iam_by_principals
|
||||
)
|
||||
iam = var.iam
|
||||
labels = var.labels
|
||||
services = concat([
|
||||
"vmwareengine.googleapis.com",
|
||||
],
|
||||
var.project_services
|
||||
)
|
||||
# specify project-level org policies here if you need them
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
# Copyright 2024 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# tfdoc:file:description Output variables.
|
||||
|
||||
output "project_id" {
|
||||
description = "GCVE project id."
|
||||
value = module.gcve-project-0.project_id
|
||||
}
|
||||
|
||||
output "vmw_engine_network_config" {
|
||||
description = "VMware engine network configuration."
|
||||
value = module.gcve-pc.vmw_engine_network_config
|
||||
}
|
||||
|
||||
output "vmw_engine_network_peerings" {
|
||||
description = "The peerings created towards the user VPC or other VMware engine networks."
|
||||
value = module.gcve-pc.vmw_engine_network_peerings
|
||||
}
|
||||
|
||||
output "vmw_engine_private_clouds" {
|
||||
description = "VMware engine private cloud resources."
|
||||
value = module.gcve-pc.vmw_engine_private_clouds
|
||||
}
|
||||
|
||||
output "vmw_private_cloud_network" {
|
||||
description = "VMware engine network."
|
||||
value = module.gcve-pc.vmw_private_cloud_network
|
||||
}
|
||||
@@ -1,122 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "billing_account_id" {
|
||||
description = "Billing account ID."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "folder_id" {
|
||||
description = "Folder used for the GCVE project in folders/nnnnnnnnnnn format."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "groups" {
|
||||
description = "GCVE groups."
|
||||
type = object({
|
||||
gcp-gcve-admins = string
|
||||
gcp-gcve-viewers = string
|
||||
})
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam_by_principals" {
|
||||
description = "Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "labels" {
|
||||
description = "Project-level labels."
|
||||
type = map(string)
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "network_peerings" {
|
||||
description = "The network peerings between users' VPCs and the VMware Engine networks. The key is the peering name suffix."
|
||||
type = map(object({
|
||||
peer_network = string
|
||||
configure_peer_network = optional(bool, false)
|
||||
custom_routes = optional(object({
|
||||
export_to_peer = optional(bool, false)
|
||||
import_from_peer = optional(bool, false)
|
||||
export_to_ven = optional(bool, false)
|
||||
import_from_ven = optional(bool, false)
|
||||
}), {})
|
||||
custom_routes_with_public_ip = optional(object({
|
||||
export_to_peer = optional(bool, false)
|
||||
import_from_peer = optional(bool, false)
|
||||
export_to_ven = optional(bool, false)
|
||||
import_from_ven = optional(bool, false)
|
||||
}), {})
|
||||
description = optional(string, "Managed by Terraform.")
|
||||
peer_project_id = optional(string)
|
||||
peer_to_vmware_engine_network = optional(bool, false)
|
||||
}))
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
description = "Prefix used for resource names."
|
||||
type = string
|
||||
validation {
|
||||
condition = var.prefix != ""
|
||||
error_message = "Prefix cannot be empty."
|
||||
}
|
||||
}
|
||||
|
||||
variable "private_cloud_configs" {
|
||||
description = "The VMware private cloud configurations. The key is the unique private cloud name suffix."
|
||||
type = map(object({
|
||||
cidr = string
|
||||
zone = string
|
||||
# The key is the unique additional cluster name suffix
|
||||
additional_cluster_configs = optional(map(object({
|
||||
custom_core_count = optional(number)
|
||||
node_count = optional(number, 3)
|
||||
node_type_id = optional(string, "standard-72")
|
||||
})), {})
|
||||
management_cluster_config = optional(object({
|
||||
custom_core_count = optional(number)
|
||||
name = optional(string, "mgmt-cluster")
|
||||
node_count = optional(number, 3)
|
||||
node_type_id = optional(string, "standard-72")
|
||||
}), {})
|
||||
description = optional(string, "Managed by Terraform.")
|
||||
}))
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "ID of the project that will contain the GCVE private cloud."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_services" {
|
||||
description = "Additional project services to enable."
|
||||
type = list(string)
|
||||
default = []
|
||||
nullable = false
|
||||
}
|
||||
@@ -20,7 +20,7 @@ They are meant to be used as minimal but complete starting points to create actu
|
||||
|
||||
### Multitenant GKE fleet
|
||||
|
||||
<a href="./multitenant-fleet/" title="GKE multitenant fleet"><img src="./multitenant-fleet/diagram.png" align="left" width="280px"></a> This [blueprint](./multitenant-fleet/) allows simple centralized management of similar sets of GKE clusters and their nodepools in a single project, and optional fleet management via GKE Hub templated configurations.
|
||||
<a href="../../fast/stages/3-gke-dev/" title="GKE multitenant fleet"><img src="../../fast/stages/3-gke-dev/diagram.png" align="left" width="280px"></a> This [blueprint](../../fast/stages/3-gke-dev/) allows simple centralized management of similar sets of GKE clusters and their nodepools in a single project, and optional fleet management via GKE Hub templated configurations.
|
||||
|
||||
<br clear="left">
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
juliocc
|
||||
@@ -1,265 +0,0 @@
|
||||
# GKE Multitenant Blueprint
|
||||
|
||||
This blueprint presents an opinionated architecture to handle multiple homogeneous GKE clusters. The general idea behind this blueprint is to deploy a single project hosting multiple clusters leveraging several useful GKE features.
|
||||
|
||||
The pattern used in this design is useful, for blueprint, in cases where multiple clusters host/support the same workloads, such as in the case of a multi-regional deployment. Furthermore, combined with Anthos Config Sync and proper RBAC, this architecture can be used to host multiple tenants (e.g. teams, applications) sharing the clusters.
|
||||
|
||||
This blueprint is used as part of the [FAST GKE stage](../../../fast/stages/3-gke-multitenant/) but it can also be used independently if desired.
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GKE multitenant">
|
||||
</p>
|
||||
|
||||
The overall architecture is based on the following design decisions:
|
||||
|
||||
- All clusters are assumed to be [private](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters), therefore only [VPC-native clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips) are supported.
|
||||
- Logging and monitoring configured to use Cloud Operations for system components and user workloads.
|
||||
- [GKE metering](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering) enabled by default and stored in a bigquery dataset created within the project.
|
||||
- Optional [GKE Fleet](https://cloud.google.com/kubernetes-engine/docs/fleets-overview) support with the possibility to enable any of the following features:
|
||||
- [Fleet workload identity](https://cloud.google.com/anthos/fleet-management/docs/use-workload-identity)
|
||||
- [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/overview)
|
||||
- [Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/overview)
|
||||
- [Anthos Identity Service](https://cloud.google.com/anthos/identity/setup/fleet)
|
||||
- [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services)
|
||||
- [Multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress).
|
||||
- Support for [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview), [Hierarchy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/hierarchy-controller), and [Policy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/policy-controller) when using Anthos Config Management.
|
||||
- [Groups for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac) can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
|
||||
- Support for [application layer secret encryption](https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets).
|
||||
- Support to customize peering configuration of the control plane VPC (e.g. to import/export routes to the peered network)
|
||||
- Some features are enabled by default in all clusters:
|
||||
- [Intranode visibility](https://cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility)
|
||||
- [Dataplane v2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2)
|
||||
- [Shielded GKE nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes)
|
||||
- [Workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
|
||||
- [Node local DNS cache](https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache)
|
||||
- [Use of the GCE persistent disk CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
|
||||
- Node [auto-upgrade](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) and [auto-repair](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair) for all node pools
|
||||
|
||||
<!--
|
||||
- [GKE subsetting for L4 internal load balancers](https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer#subsetting) enabled by default in all clusters
|
||||
-->
|
||||
|
||||
## Basic usage
|
||||
|
||||
The following example shows how to deploy two clusters and one node pool for each
|
||||
|
||||
```hcl
|
||||
locals {
|
||||
cluster_defaults = {
|
||||
private_cluster_config = {
|
||||
enable_private_endpoint = true
|
||||
master_global_access = true
|
||||
}
|
||||
}
|
||||
subnet_self_links = {
|
||||
ew1 = "projects/prj-host/regions/europe-west1/subnetworks/gke-0"
|
||||
ew3 = "projects/prj-host/regions/europe-west3/subnetworks/gke-0"
|
||||
}
|
||||
}
|
||||
|
||||
module "gke-fleet" {
|
||||
source = "./fabric/blueprints/gke/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
iam_by_principals = {
|
||||
"group:gke-admin@example.com" = [
|
||||
"roles/container.admin"
|
||||
]
|
||||
}
|
||||
iam = {
|
||||
"roles/container.clusterAdmin" = [
|
||||
"serviceAccount:cicd@my-cicd-project.iam.gserviceaccount.com"
|
||||
]
|
||||
}
|
||||
clusters = {
|
||||
cluster-0 = {
|
||||
location = "europe-west1"
|
||||
private_cluster_config = local.cluster_defaults.private_cluster_config
|
||||
vpc_config = {
|
||||
subnetwork = local.subnet_self_links.ew1
|
||||
master_ipv4_cidr_block = "172.16.10.0/28"
|
||||
}
|
||||
}
|
||||
cluster-1 = {
|
||||
location = "europe-west3"
|
||||
private_cluster_config = local.cluster_defaults.private_cluster_config
|
||||
vpc_config = {
|
||||
subnetwork = local.subnet_self_links.ew3
|
||||
master_ipv4_cidr_block = "172.16.20.0/28"
|
||||
}
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
cluster-0 = {
|
||||
nodepool-0 = {
|
||||
node_config = {
|
||||
disk_type = "pd-balanced"
|
||||
machine_type = "n2-standard-4"
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
cluster-1 = {
|
||||
nodepool-0 = {
|
||||
node_config = {
|
||||
disk_type = "pd-balanced"
|
||||
machine_type = "n2-standard-4"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/prj-host/global/networks/prod-0"
|
||||
}
|
||||
}
|
||||
# tftest modules=8 resources=46
|
||||
```
|
||||
|
||||
## GKE Fleet
|
||||
|
||||
This example deploys two clusters and configures several GKE Fleet features:
|
||||
|
||||
- Enables [multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress) and sets the configuration cluster to be `cluster-eu1`.
|
||||
- Enables [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services) and assigns the [required roles](https://cloud.google.com/kubernetes-engine/docs/how-to/multi-cluster-services#authenticating) to its service accounts.
|
||||
- A `default` Config Management template is created with binary authorization, config sync enabled with a git repository, hierarchy controller, and policy controller.
|
||||
- The two clusters are configured to use the `default` Config Management template.
|
||||
|
||||
```hcl
|
||||
locals {
|
||||
subnet_self_links = {
|
||||
ew1 = "projects/prj-host/regions/europe-west1/subnetworks/gke-0"
|
||||
ew3 = "projects/prj-host/regions/europe-west3/subnetworks/gke-0"
|
||||
}
|
||||
}
|
||||
|
||||
module "gke" {
|
||||
source = "./fabric/blueprints/gke/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
clusters = {
|
||||
cluster-0 = {
|
||||
location = "europe-west1"
|
||||
vpc_config = {
|
||||
subnetwork = local.subnet_self_links.ew1
|
||||
}
|
||||
}
|
||||
cluster-1 = {
|
||||
location = "europe-west3"
|
||||
vpc_config = {
|
||||
subnetwork = local.subnet_self_links.ew3
|
||||
}
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
cluster-0 = {
|
||||
nodepool-0 = {
|
||||
node_config = {
|
||||
disk_type = "pd-balanced"
|
||||
machine_type = "n2-standard-4"
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
cluster-1 = {
|
||||
nodepool-0 = {
|
||||
node_config = {
|
||||
disk_type = "pd-balanced"
|
||||
machine_type = "n2-standard-4"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
fleet_features = {
|
||||
configmanagement = true
|
||||
identityservice = true
|
||||
multiclusteringress = "cluster-0"
|
||||
multiclusterservicediscovery = true
|
||||
servicemesh = true
|
||||
}
|
||||
fleet_workload_identity = true
|
||||
fleet_configmanagement_templates = {
|
||||
default = {
|
||||
binauthz = true
|
||||
config_sync = {
|
||||
git = {
|
||||
policy_dir = "configsync"
|
||||
secret_type = "none"
|
||||
source_format = "hierarchy"
|
||||
sync_branch = "main"
|
||||
sync_repo = "https://github.com/myorg/myrepo"
|
||||
}
|
||||
prevent_drift = true
|
||||
source_format = "hierarchy"
|
||||
}
|
||||
hierarchy_controller = {
|
||||
enable_hierarchical_resource_quota = true
|
||||
enable_pod_tree_labels = true
|
||||
}
|
||||
policy_controller = {
|
||||
audit_interval_seconds = 30
|
||||
exemptable_namespaces = ["kube-system"]
|
||||
log_denies_enabled = true
|
||||
referential_rules_enabled = true
|
||||
template_library_installed = true
|
||||
}
|
||||
version = "1.10.2"
|
||||
}
|
||||
}
|
||||
fleet_configmanagement_clusters = {
|
||||
default = ["cluster-0", "cluster-1"]
|
||||
}
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/prj-host/global/networks/prod-0"
|
||||
}
|
||||
}
|
||||
# tftest modules=9 resources=57
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Files
|
||||
|
||||
| name | description | modules |
|
||||
|---|---|---|
|
||||
| [gke-clusters.tf](./gke-clusters.tf) | GKE clusters. | <code>gke-cluster-standard</code> |
|
||||
| [gke-hub.tf](./gke-hub.tf) | GKE hub configuration. | <code>gke-hub</code> |
|
||||
| [gke-nodepools.tf](./gke-nodepools.tf) | GKE nodepools. | <code>gke-nodepool</code> |
|
||||
| [main.tf](./main.tf) | Project and usage dataset. | <code>bigquery-dataset</code> · <code>iam-service-account</code> · <code>project</code> |
|
||||
| [outputs.tf](./outputs.tf) | Output variables. | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [billing_account_id](variables.tf#L17) | Billing account ID. | <code>string</code> | ✓ | |
|
||||
| [folder_id](variables.tf#L131) | Folder used for the GKE project in folders/nnnnnnnnnnn format. | <code>string</code> | ✓ | |
|
||||
| [prefix](variables.tf#L189) | Prefix used for resource names. | <code>string</code> | ✓ | |
|
||||
| [project_id](variables.tf#L198) | ID of the project that will contain all the clusters. | <code>string</code> | ✓ | |
|
||||
| [vpc_config](variables.tf#L210) | Shared VPC project and VPC details. | <code title="object({ host_project_id = string vpc_self_link = string })">object({…})</code> | ✓ | |
|
||||
| [clusters](variables.tf#L22) | Clusters configuration. Refer to the gke-cluster module for type details. | <code title="map(object({ cluster_autoscaling = optional(any) description = optional(string) enable_addons = optional(any, { horizontal_pod_autoscaling = true, http_load_balancing = true }) enable_features = optional(any, { shielded_nodes = true workload_identity = true }) issue_client_certificate = optional(bool, false) labels = optional(map(string)) location = string logging_config = optional(object({ enable_system_logs = optional(bool, true) enable_workloads_logs = optional(bool, true) enable_api_server_logs = optional(bool, false) enable_scheduler_logs = optional(bool, false) enable_controller_manager_logs = optional(bool, false) }), {}) maintenance_config = optional(any, { daily_window_start_time = "03:00" recurring_window = null maintenance_exclusion = [] }) max_pods_per_node = optional(number, 110) min_master_version = optional(string) monitoring_config = optional(object({ enable_system_metrics = optional(bool, true) enable_api_server_metrics = optional(bool, false) enable_controller_manager_metrics = optional(bool, false) enable_scheduler_metrics = optional(bool, false) enable_daemonset_metrics = optional(bool, false) enable_deployment_metrics = optional(bool, false) enable_hpa_metrics = optional(bool, false) enable_pod_metrics = optional(bool, false) enable_statefulset_metrics = optional(bool, false) enable_storage_metrics = optional(bool, false) enable_managed_prometheus = optional(bool, true) }), {}) node_locations = optional(list(string)) private_cluster_config = optional(any) release_channel = optional(string) vpc_config = object({ subnetwork = string network = optional(string) secondary_range_blocks = optional(object({ pods = string services = string })) secondary_range_names = optional(object({ pods = string services = string }), { pods = "pods", services = "services" }) master_authorized_ranges = optional(map(string)) master_ipv4_cidr_block = optional(string) }) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [deletion_protection](variables.tf#L89) | Prevent Terraform from destroying data storage resources (storage buckets, GKE clusters, CloudSQL instances) in this blueprint. When this field is set in Terraform state, a terraform destroy or terraform apply that would delete data storage resources will fail. | <code>bool</code> | | <code>false</code> |
|
||||
| [fleet_configmanagement_clusters](variables.tf#L96) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [fleet_configmanagement_templates](variables.tf#L103) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code>map(any)</code> | | <code>{}</code> |
|
||||
| [fleet_features](variables.tf#L111) | Enable and configure fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = optional(bool, false) configmanagement = optional(bool, false) identityservice = optional(bool, false) multiclusteringress = optional(string, null) multiclusterservicediscovery = optional(bool, false) servicemesh = optional(bool, false) })">object({…})</code> | | <code>null</code> |
|
||||
| [fleet_workload_identity](variables.tf#L124) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>false</code> |
|
||||
| [iam](variables.tf#L136) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [iam_by_principals](variables.tf#L143) | Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [labels](variables.tf#L150) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [nodepools](variables.tf#L156) | Nodepools configuration. Refer to the gke-nodepool module for type details. | <code title="map(map(object({ gke_version = optional(string) k8s_labels = optional(map(string), {}) max_pods_per_node = optional(number) name = optional(string) node_config = optional(any, { disk_type = "pd-balanced" shielded_instance_config = { enable_integrity_monitoring = true enable_secure_boot = true } }) node_count = optional(map(number), { initial = 1 }) node_locations = optional(list(string)) nodepool_config = optional(any) pod_range = optional(any) reservation_affinity = optional(any) service_account = optional(any) sole_tenant_nodegroup = optional(string) tags = optional(list(string)) taints = optional(map(object({ value = string effect = string }))) })))">map(map(object({…})))</code> | | <code>{}</code> |
|
||||
| [project_services](variables.tf#L203) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive |
|
||||
|---|---|:---:|
|
||||
| [cluster_ids](outputs.tf#L17) | Cluster ids. | |
|
||||
| [clusters](outputs.tf#L24) | Cluster resources. | |
|
||||
| [project_id](outputs.tf#L29) | GKE project id. | |
|
||||
<!-- END TFDOC -->
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 43 KiB |
@@ -1,45 +0,0 @@
|
||||
/**
|
||||
* Copyright 2023 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE clusters.
|
||||
|
||||
module "gke-cluster" {
|
||||
source = "../../../modules/gke-cluster-standard"
|
||||
for_each = var.clusters
|
||||
name = each.key
|
||||
project_id = module.gke-project-0.project_id
|
||||
cluster_autoscaling = each.value.cluster_autoscaling
|
||||
description = each.value.description
|
||||
enable_features = each.value.enable_features
|
||||
enable_addons = each.value.enable_addons
|
||||
issue_client_certificate = each.value.issue_client_certificate
|
||||
labels = each.value.labels
|
||||
location = each.value.location
|
||||
logging_config = each.value.logging_config
|
||||
maintenance_config = each.value.maintenance_config
|
||||
max_pods_per_node = each.value.max_pods_per_node
|
||||
min_master_version = each.value.min_master_version
|
||||
monitoring_config = each.value.monitoring_config
|
||||
node_locations = each.value.node_locations
|
||||
private_cluster_config = each.value.private_cluster_config
|
||||
release_channel = each.value.release_channel
|
||||
vpc_config = merge(each.value.vpc_config, {
|
||||
network = coalesce(
|
||||
each.value.vpc_config.network, var.vpc_config.vpc_self_link
|
||||
)
|
||||
})
|
||||
deletion_protection = var.deletion_protection
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE hub configuration.
|
||||
|
||||
locals {
|
||||
fleet_enabled = (
|
||||
var.fleet_features != null || var.fleet_workload_identity
|
||||
)
|
||||
fleet_mcs_enabled = (
|
||||
try(var.fleet_features.multiclusterservicediscovery, false) == true
|
||||
)
|
||||
}
|
||||
|
||||
module "gke-hub" {
|
||||
source = "../../../modules/gke-hub"
|
||||
count = local.fleet_enabled ? 1 : 0
|
||||
project_id = module.gke-project-0.project_id
|
||||
clusters = {
|
||||
for cluster_id in keys(var.clusters) :
|
||||
cluster_id => module.gke-cluster[cluster_id].id
|
||||
}
|
||||
features = var.fleet_features
|
||||
configmanagement_templates = var.fleet_configmanagement_templates
|
||||
configmanagement_clusters = var.fleet_configmanagement_clusters
|
||||
workload_identity_clusters = (
|
||||
var.fleet_workload_identity ? keys(var.clusters) : []
|
||||
)
|
||||
|
||||
depends_on = [
|
||||
module.gke-nodepool
|
||||
]
|
||||
}
|
||||
@@ -1,55 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE nodepools.
|
||||
|
||||
locals {
|
||||
nodepools = merge([
|
||||
for cluster, nodepools in var.nodepools : {
|
||||
for nodepool, config in nodepools :
|
||||
"${cluster}/${nodepool}" => merge(config, {
|
||||
name = nodepool
|
||||
cluster = cluster
|
||||
})
|
||||
}
|
||||
]...)
|
||||
}
|
||||
|
||||
module "gke-nodepool" {
|
||||
source = "../../../modules/gke-nodepool"
|
||||
for_each = local.nodepools
|
||||
name = each.value.name
|
||||
project_id = module.gke-project-0.project_id
|
||||
cluster_name = module.gke-cluster[each.value.cluster].name
|
||||
location = module.gke-cluster[each.value.cluster].location
|
||||
gke_version = each.value.gke_version
|
||||
k8s_labels = each.value.k8s_labels
|
||||
max_pods_per_node = each.value.max_pods_per_node
|
||||
node_config = each.value.node_config
|
||||
node_count = each.value.node_count
|
||||
node_locations = each.value.node_locations
|
||||
nodepool_config = each.value.nodepool_config
|
||||
pod_range = each.value.pod_range
|
||||
reservation_affinity = each.value.reservation_affinity
|
||||
service_account = (
|
||||
each.value.service_account == null
|
||||
? { email = module.gke-nodes-service-account.email }
|
||||
: each.value.service_account
|
||||
)
|
||||
sole_tenant_nodegroup = each.value.sole_tenant_nodegroup
|
||||
tags = each.value.tags
|
||||
taints = each.value.taints
|
||||
}
|
||||
@@ -1,97 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Project and usage dataset.
|
||||
|
||||
locals {
|
||||
gke_nodes_sa_roles = [
|
||||
"autoscaling.metricsWriter",
|
||||
"logging.logWriter",
|
||||
"monitoring.viewer",
|
||||
"monitoring.metricWriter",
|
||||
"stackdriver.resourceMetadata.writer"
|
||||
]
|
||||
}
|
||||
|
||||
module "gke-project-0" {
|
||||
source = "../../../modules/project"
|
||||
billing_account = var.billing_account_id
|
||||
name = var.project_id
|
||||
parent = var.folder_id
|
||||
prefix = var.prefix
|
||||
iam_by_principals = var.iam_by_principals
|
||||
labels = var.labels
|
||||
iam = merge(var.iam, {
|
||||
"roles/gkehub.serviceAgent" = [
|
||||
module.gke-project-0.service_agents.fleet.iam_email
|
||||
] }
|
||||
)
|
||||
iam_bindings_additive = {
|
||||
for r in local.gke_nodes_sa_roles : "gke-nodes-sa-${r}" => {
|
||||
member = module.gke-nodes-service-account.iam_email
|
||||
role = "roles/${r}"
|
||||
}
|
||||
}
|
||||
services = concat(
|
||||
[
|
||||
"anthos.googleapis.com",
|
||||
"anthosconfigmanagement.googleapis.com",
|
||||
"cloudresourcemanager.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"dns.googleapis.com",
|
||||
"gkeconnect.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
"iam.googleapis.com",
|
||||
"multiclusteringress.googleapis.com",
|
||||
"multiclusterservicediscovery.googleapis.com",
|
||||
"stackdriver.googleapis.com",
|
||||
"trafficdirector.googleapis.com"
|
||||
],
|
||||
var.project_services
|
||||
)
|
||||
shared_vpc_service_config = {
|
||||
attach = true
|
||||
host_project = var.vpc_config.host_project_id
|
||||
service_agent_iam = merge({
|
||||
"roles/compute.networkUser" = [
|
||||
"cloudservices", "container-engine"
|
||||
]
|
||||
"roles/container.hostServiceAgentUser" = [
|
||||
"container-engine"
|
||||
]
|
||||
},
|
||||
!local.fleet_mcs_enabled ? {} : {
|
||||
"roles/multiclusterservicediscovery.serviceAgent" = ["mcsd"]
|
||||
"roles/compute.networkViewer" = [
|
||||
"serviceAccount:${var.prefix}-${var.project_id}.svc.id.goog[gke-mcs/gke-mcs-importer]"
|
||||
]
|
||||
})
|
||||
}
|
||||
# specify project-level org policies here if you need them
|
||||
}
|
||||
|
||||
module "gke-dataset-resource-usage" {
|
||||
source = "../../../modules/bigquery-dataset"
|
||||
project_id = module.gke-project-0.project_id
|
||||
id = "gke_resource_usage"
|
||||
friendly_name = "GKE resource usage."
|
||||
}
|
||||
|
||||
module "gke-nodes-service-account" {
|
||||
source = "../../../modules/iam-service-account"
|
||||
project_id = module.gke-project-0.project_id
|
||||
name = "gke-node-default"
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
# Copyright 2023 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# tfdoc:file:description Output variables.
|
||||
|
||||
output "cluster_ids" {
|
||||
description = "Cluster ids."
|
||||
value = {
|
||||
for k, v in module.gke-cluster : k => v.id
|
||||
}
|
||||
}
|
||||
|
||||
output "clusters" {
|
||||
description = "Cluster resources."
|
||||
value = module.gke-cluster
|
||||
}
|
||||
|
||||
output "project_id" {
|
||||
description = "GKE project id."
|
||||
value = module.gke-project-0.project_id
|
||||
}
|
||||
@@ -1,216 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "billing_account_id" {
|
||||
description = "Billing account ID."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "clusters" {
|
||||
description = "Clusters configuration. Refer to the gke-cluster module for type details."
|
||||
type = map(object({
|
||||
cluster_autoscaling = optional(any)
|
||||
description = optional(string)
|
||||
enable_addons = optional(any, {
|
||||
horizontal_pod_autoscaling = true, http_load_balancing = true
|
||||
})
|
||||
enable_features = optional(any, {
|
||||
shielded_nodes = true
|
||||
workload_identity = true
|
||||
})
|
||||
issue_client_certificate = optional(bool, false)
|
||||
labels = optional(map(string))
|
||||
location = string
|
||||
logging_config = optional(object({
|
||||
enable_system_logs = optional(bool, true)
|
||||
enable_workloads_logs = optional(bool, true)
|
||||
enable_api_server_logs = optional(bool, false)
|
||||
enable_scheduler_logs = optional(bool, false)
|
||||
enable_controller_manager_logs = optional(bool, false)
|
||||
}), {})
|
||||
maintenance_config = optional(any, {
|
||||
daily_window_start_time = "03:00"
|
||||
recurring_window = null
|
||||
maintenance_exclusion = []
|
||||
})
|
||||
max_pods_per_node = optional(number, 110)
|
||||
min_master_version = optional(string)
|
||||
monitoring_config = optional(object({
|
||||
enable_system_metrics = optional(bool, true)
|
||||
# (Optional) control plane metrics
|
||||
enable_api_server_metrics = optional(bool, false)
|
||||
enable_controller_manager_metrics = optional(bool, false)
|
||||
enable_scheduler_metrics = optional(bool, false)
|
||||
# (Optional) kube state metrics
|
||||
enable_daemonset_metrics = optional(bool, false)
|
||||
enable_deployment_metrics = optional(bool, false)
|
||||
enable_hpa_metrics = optional(bool, false)
|
||||
enable_pod_metrics = optional(bool, false)
|
||||
enable_statefulset_metrics = optional(bool, false)
|
||||
enable_storage_metrics = optional(bool, false)
|
||||
# Google Cloud Managed Service for Prometheus
|
||||
enable_managed_prometheus = optional(bool, true)
|
||||
}), {})
|
||||
node_locations = optional(list(string))
|
||||
private_cluster_config = optional(any)
|
||||
release_channel = optional(string)
|
||||
vpc_config = object({
|
||||
subnetwork = string
|
||||
network = optional(string)
|
||||
secondary_range_blocks = optional(object({
|
||||
pods = string
|
||||
services = string
|
||||
}))
|
||||
secondary_range_names = optional(object({
|
||||
pods = string
|
||||
services = string
|
||||
}), { pods = "pods", services = "services" })
|
||||
master_authorized_ranges = optional(map(string))
|
||||
master_ipv4_cidr_block = optional(string)
|
||||
})
|
||||
}))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
description = "Prevent Terraform from destroying data storage resources (storage buckets, GKE clusters, CloudSQL instances) in this blueprint. When this field is set in Terraform state, a terraform destroy or terraform apply that would delete data storage resources will fail."
|
||||
type = bool
|
||||
default = false
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_clusters" {
|
||||
description = "Config management features enabled on specific sets of member clusters, in config name => [cluster name] format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_templates" {
|
||||
description = "Sets of config management configurations that can be applied to member clusters, in config name => {options} format."
|
||||
# refer to the gke-hub module for the full type
|
||||
type = map(any)
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_features" {
|
||||
description = "Enable and configure fleet features. Set to null to disable GKE Hub if fleet workload identity is not used."
|
||||
type = object({
|
||||
appdevexperience = optional(bool, false)
|
||||
configmanagement = optional(bool, false)
|
||||
identityservice = optional(bool, false)
|
||||
multiclusteringress = optional(string, null)
|
||||
multiclusterservicediscovery = optional(bool, false)
|
||||
servicemesh = optional(bool, false)
|
||||
})
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_workload_identity" {
|
||||
description = "Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true."
|
||||
type = bool
|
||||
default = false
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "folder_id" {
|
||||
description = "Folder used for the GKE project in folders/nnnnnnnnnnn format."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam_by_principals" {
|
||||
description = "Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "labels" {
|
||||
description = "Project-level labels."
|
||||
type = map(string)
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "nodepools" {
|
||||
description = "Nodepools configuration. Refer to the gke-nodepool module for type details."
|
||||
type = map(map(object({
|
||||
gke_version = optional(string)
|
||||
k8s_labels = optional(map(string), {})
|
||||
max_pods_per_node = optional(number)
|
||||
name = optional(string)
|
||||
node_config = optional(any, {
|
||||
disk_type = "pd-balanced"
|
||||
shielded_instance_config = {
|
||||
enable_integrity_monitoring = true
|
||||
enable_secure_boot = true
|
||||
}
|
||||
})
|
||||
node_count = optional(map(number), {
|
||||
initial = 1
|
||||
})
|
||||
node_locations = optional(list(string))
|
||||
nodepool_config = optional(any)
|
||||
pod_range = optional(any)
|
||||
reservation_affinity = optional(any)
|
||||
service_account = optional(any)
|
||||
sole_tenant_nodegroup = optional(string)
|
||||
tags = optional(list(string))
|
||||
taints = optional(map(object({
|
||||
value = string
|
||||
effect = string
|
||||
})))
|
||||
})))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
description = "Prefix used for resource names."
|
||||
type = string
|
||||
validation {
|
||||
condition = var.prefix != ""
|
||||
error_message = "Prefix cannot be empty."
|
||||
}
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "ID of the project that will contain all the clusters."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_services" {
|
||||
description = "Additional project services to enable."
|
||||
type = list(string)
|
||||
default = []
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "vpc_config" {
|
||||
description = "Shared VPC project and VPC details."
|
||||
type = object({
|
||||
host_project_id = string
|
||||
vpc_self_link = string
|
||||
})
|
||||
}
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Fabric release: v35.1.0
|
||||
=======
|
||||
# Fabric release: v36.0.0
|
||||
>>>>>>> fast-dev
|
||||
|
||||
terraform {
|
||||
required_version = ">= 1.7.4"
|
||||
|
||||
Reference in New Issue
Block a user