Rename gke-serverless to gke and add test for fast gke stage
This commit is contained in:
Julio Castillo
12
blueprints/gke/README.md
Normal file
12
blueprints/gke/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# GKE and Serverless blueprints
|
||||
|
||||
The blueprints in this folder show implement **end-to-end scenarios** for GKE or Serveless topologies that show how to automate common configurations or leverage specific products.
|
||||
|
||||
They are meant to be used as minimal but complete starting points to create actual infrastructure, and as playgrounds to experiment with Google Cloud features.
|
||||
|
||||
## Blueprints
|
||||
|
||||
### Multitenant GKE fleet
|
||||
|
||||
<a href="./multitenant-fleet/" title="GKE multitenant fleet"><img src="./multitenant-fleet/diagram.png" align="left" width="280px"></a> This [blueprint](./multitenant-fleet/) allows simple centralized management of similar sets of GKE clusters and their nodepools in a single project, and optional fleet management via GKE Hub templated configurations.
|
||||
<br clear="left">
|
||||
359
blueprints/gke/multitenant-fleet/README.md
Normal file
359
blueprints/gke/multitenant-fleet/README.md
Normal file
@@ -0,0 +1,359 @@
|
||||
# GKE Multitenant Blueprint
|
||||
|
||||
This blueprint presents an opinionated architecture to handle multiple homogeneous GKE clusters. The general idea behind this blueprint is to deploy a single project hosting multiple clusters leveraging several useful GKE features.
|
||||
|
||||
The pattern used in this design is useful, for blueprint, in cases where multiple clusters host/support the same workloads, such as in the case of a multi-regional deployment. Furthermore, combined with Anthos Config Sync and proper RBAC, this architecture can be used to host multiple tenants (e.g. teams, applications) sharing the clusters.
|
||||
|
||||
This blueprint is used as part of the [FAST GKE stage](../../../fast/stages/03-gke-multitenant/) but it can also be used independently if desired.
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GKE multitenant">
|
||||
</p>
|
||||
|
||||
The overall architecture is based on the following design decisions:
|
||||
|
||||
- All clusters are assumed to be [private](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters), therefore only [VPC-native clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips) are supported.
|
||||
- Logging and monitoring configured to use Cloud Operations for system components and user workloads.
|
||||
- [GKE metering](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering) enabled by default and stored in a bigquery dataset created within the project.
|
||||
- Optional [GKE Fleet](https://cloud.google.com/kubernetes-engine/docs/fleets-overview) support with the possibility to enable any of the following features:
|
||||
- [Fleet workload identity](https://cloud.google.com/anthos/fleet-management/docs/use-workload-identity)
|
||||
- [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/overview)
|
||||
- [Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/overview)
|
||||
- [Anthos Identity Service](https://cloud.google.com/anthos/identity/setup/fleet)
|
||||
- [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services)
|
||||
- [Multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress).
|
||||
- Support for [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview), [Hierarchy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/hierarchy-controller), and [Policy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/policy-controller) when using Anthos Config Management.
|
||||
- [Groups for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac) can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
|
||||
- Support for [application layer secret encryption](https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets).
|
||||
- Support to customize peering configuration of the control plane VPC (e.g. to import/export routes to the peered network)
|
||||
- Some features are enabled by default in all clusters:
|
||||
- [Intranode visibility](https://cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility)
|
||||
- [Dataplane v2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2)
|
||||
- [Shielded GKE nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes)
|
||||
- [Workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
|
||||
- [Node local DNS cache](https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache)
|
||||
- [Use of the GCE persistent disk CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
|
||||
- Node [auto-upgrade](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) and [auto-repair](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair) for all node pools
|
||||
|
||||
<!--
|
||||
- [GKE subsetting for L4 internal load balancers](https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer#subsetting) enabled by default in all clusters
|
||||
-->
|
||||
|
||||
## Basic usage
|
||||
|
||||
The following example shows how to deploy a single cluster and a single node pool
|
||||
|
||||
```hcl
|
||||
module "gke" {
|
||||
source = "./fabric/blueprints/gke-serverless/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/my-host-project-id/global/networks/my-network"
|
||||
}
|
||||
|
||||
authenticator_security_group = "gke-rbac-base@example.com"
|
||||
group_iam = {
|
||||
"gke-admin@example.com" = [
|
||||
"roles/container.admin"
|
||||
]
|
||||
}
|
||||
iam = {
|
||||
"roles/container.clusterAdmin" = [
|
||||
"cicd@my-cicd-project.iam.gserviceaccount.com"
|
||||
]
|
||||
}
|
||||
|
||||
clusters = {
|
||||
mycluster = {
|
||||
cluster_autoscaling = null
|
||||
description = "My cluster"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west1/subnetworks/mycluster-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
mycluster = {
|
||||
mynodepool = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest modules=1 resources=0
|
||||
```
|
||||
|
||||
## Creating Multiple Clusters
|
||||
|
||||
The following example shows how to deploy two clusters with different configurations.
|
||||
|
||||
The first cluster `cluster-euw1` defines the mandatory configuration parameters (description, location, network setup) and inherits the some defaults from the `cluster_defaults` and `nodepool_deaults` variables. These two variables are used whenever the `override` key of the `clusters` and `nodepools` variables are set to `null`.
|
||||
|
||||
On the other hand, the second cluster (`cluster-euw3`) defines its own configuration by providing a value to the `overrides` key.
|
||||
|
||||
|
||||
```hcl
|
||||
module "gke" {
|
||||
source = "./fabric/blueprints/gke-serverless/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/my-host-project-id/global/networks/my-network"
|
||||
}
|
||||
clusters = {
|
||||
cluster-euw1 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europ-west1"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west1/subnetworks/euw1-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
cluster-euw3 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europe-west3"
|
||||
dns_domain = null
|
||||
location = "europe-west3"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.17.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west3/subnetworks/euw3-subnet"
|
||||
}
|
||||
overrides = {
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
gcp_filestore_csi_driver_config = true
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
}
|
||||
max_pods_per_node = 64
|
||||
pod_security_policy = true
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
}
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
cluster-euw1 = {
|
||||
pool-euw1 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = false
|
||||
}
|
||||
}
|
||||
cluster-euw3 = {
|
||||
pool-euw3 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = {
|
||||
image_type = "UBUNTU_CONTAINERD"
|
||||
max_pods_per_node = 64
|
||||
node_locations = []
|
||||
node_tags = []
|
||||
node_taints = []
|
||||
}
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest modules=1 resources=0
|
||||
```
|
||||
|
||||
## Multiple clusters with GKE Fleet
|
||||
|
||||
This example deploys two clusters and configures several GKE Fleet features:
|
||||
|
||||
- Enables [multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress) and sets the configuration cluster to be `cluster-eu1`.
|
||||
- Enables [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services) and assigns the [required roles](https://cloud.google.com/kubernetes-engine/docs/how-to/multi-cluster-services#authenticating) to its service accounts.
|
||||
- A `default` Config Management template is created with binary authorization, config sync enabled with a git repository, hierarchy controller, and policy controller.
|
||||
- The two clusters are configured to use the `default` Config Management template.
|
||||
|
||||
```hcl
|
||||
module "gke" {
|
||||
source = "./fabric/blueprints/gke-serverless/multitenant-fleet/"
|
||||
project_id = var.project_id
|
||||
billing_account_id = var.billing_account_id
|
||||
folder_id = var.folder_id
|
||||
prefix = "myprefix"
|
||||
vpc_config = {
|
||||
host_project_id = "my-host-project-id"
|
||||
vpc_self_link = "projects/my-host-project-id/global/networks/my-network"
|
||||
}
|
||||
clusters = {
|
||||
cluster-euw1 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europe-west1"
|
||||
dns_domain = null
|
||||
location = "europe-west1"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.16.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west1/subnetworks/euw1-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
cluster-euw3 = {
|
||||
cluster_autoscaling = null
|
||||
description = "Cluster for europe-west3"
|
||||
dns_domain = null
|
||||
location = "europe-west3"
|
||||
labels = {}
|
||||
net = {
|
||||
master_range = "172.17.17.0/28"
|
||||
pods = "pods"
|
||||
services = "services"
|
||||
subnet = "projects/my-host-project-id/regions/europe-west3/subnetworks/euw3-subnet"
|
||||
}
|
||||
overrides = null
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
cluster-euw1 = {
|
||||
pool-euw1 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = false
|
||||
}
|
||||
}
|
||||
cluster-euw3 = {
|
||||
pool-euw3 = {
|
||||
initial_node_count = 1
|
||||
node_count = 1
|
||||
node_type = "n2-standard-4"
|
||||
overrides = null
|
||||
spot = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fleet_features = {
|
||||
appdevexperience = false
|
||||
configmanagement = true
|
||||
identityservice = true
|
||||
multiclusteringress = "cluster-euw1"
|
||||
multiclusterservicediscovery = true
|
||||
servicemesh = true
|
||||
}
|
||||
fleet_workload_identity = true
|
||||
fleet_configmanagement_templates = {
|
||||
default = {
|
||||
binauthz = true
|
||||
config_sync = {
|
||||
git = {
|
||||
gcp_service_account_email = null
|
||||
https_proxy = null
|
||||
policy_dir = "configsync"
|
||||
secret_type = "none"
|
||||
source_format = "hierarchy"
|
||||
sync_branch = "main"
|
||||
sync_repo = "https://github.com/myorg/myrepo"
|
||||
sync_rev = null
|
||||
sync_wait_secs = null
|
||||
}
|
||||
prevent_drift = true
|
||||
source_format = "hierarchy"
|
||||
}
|
||||
hierarchy_controller = {
|
||||
enable_hierarchical_resource_quota = true
|
||||
enable_pod_tree_labels = true
|
||||
}
|
||||
policy_controller = {
|
||||
audit_interval_seconds = 30
|
||||
exemptable_namespaces = ["kube-system"]
|
||||
log_denies_enabled = true
|
||||
referential_rules_enabled = true
|
||||
template_library_installed = true
|
||||
}
|
||||
version = "1.10.2"
|
||||
}
|
||||
}
|
||||
fleet_configmanagement_clusters = {
|
||||
default = ["cluster-euw1", "cluster-euw3"]
|
||||
}
|
||||
}
|
||||
|
||||
# tftest modules=1 resources=0
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
|
||||
## Files
|
||||
|
||||
| name | description | modules |
|
||||
|---|---|---|
|
||||
| [gke-clusters.tf](./gke-clusters.tf) | None | <code>gke-cluster</code> |
|
||||
| [gke-hub.tf](./gke-hub.tf) | None | <code>gke-hub</code> |
|
||||
| [gke-nodepools.tf](./gke-nodepools.tf) | None | <code>gke-nodepool</code> |
|
||||
| [main.tf](./main.tf) | Module-level locals and resources. | <code>bigquery-dataset</code> · <code>project</code> |
|
||||
| [outputs.tf](./outputs.tf) | Output variables. | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [billing_account_id](variables.tf#L23) | Billing account id. | <code>string</code> | ✓ | | |
|
||||
| [clusters](variables.tf#L57) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
||||
| [folder_id](variables.tf#L158) | Folder used for the GKE project in folders/nnnnnnnnnnn format. | <code>string</code> | ✓ | | |
|
||||
| [nodepools](variables.tf#L201) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
||||
| [prefix](variables.tf#L231) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
||||
| [project_id](variables.tf#L236) | ID of the project that will contain all the clusters. | <code>string</code> | ✓ | | |
|
||||
| [vpc_config](variables.tf#L248) | Shared VPC project and VPC details. | <code title="object({ host_project_id = string vpc_self_link = string })">object({…})</code> | ✓ | | |
|
||||
| [authenticator_security_group](variables.tf#L17) | Optional group used for Groups for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [cluster_defaults](variables.tf#L28) | Default values for optional cluster configurations. | <code title="object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool })">object({…})</code> | | <code title="{ cloudrun_config = false database_encryption_key = null master_authorized_ranges = { rfc1918_1 = "10.0.0.0/8" rfc1918_2 = "172.16.0.0/12" rfc1918_3 = "192.168.0.0/16" } max_pods_per_node = 110 pod_security_policy = false release_channel = "STABLE" vertical_pod_autoscaling = false gcp_filestore_csi_driver_config = false }">{…}</code> | |
|
||||
| [dns_domain](variables.tf#L90) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
||||
| [fleet_configmanagement_clusters](variables.tf#L96) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [fleet_configmanagement_templates](variables.tf#L103) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [fleet_features](variables.tf#L138) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
||||
| [fleet_workload_identity](variables.tf#L151) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>false</code> | |
|
||||
| [group_iam](variables.tf#L163) | Project-level IAM bindings for groups. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam](variables.tf#L170) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [labels](variables.tf#L177) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
||||
| [nodepool_defaults](variables.tf#L183) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
||||
| [peering_config](variables.tf#L218) | Configure peering with the control plane VPC. Requires compute.networks.updatePeering. Set to null if you don't want to update the default peering configuration. | <code title="object({ export_routes = bool import_routes = bool })">object({…})</code> | | <code title="{ export_routes = true // TODO(jccb) is there any situation where the control plane VPC would export any routes? import_routes = false }">{…}</code> | |
|
||||
| [project_services](variables.tf#L241) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [cluster_ids](outputs.tf#L22) | Cluster ids. | | |
|
||||
| [clusters](outputs.tf#L17) | Cluster resources. | | |
|
||||
| [project_id](outputs.tf#L29) | GKE project id. | | |
|
||||
|
||||
<!-- END TFDOC -->
|
||||
BIN
blueprints/gke/multitenant-fleet/diagram.png
Normal file
BIN
blueprints/gke/multitenant-fleet/diagram.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
116
blueprints/gke/multitenant-fleet/gke-clusters.tf
Normal file
116
blueprints/gke/multitenant-fleet/gke-clusters.tf
Normal file
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
clusters = {
|
||||
for name, config in var.clusters :
|
||||
name => merge(config, {
|
||||
overrides = coalesce(config.overrides, var.cluster_defaults)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
module "gke-cluster" {
|
||||
source = "../../../modules/gke-cluster"
|
||||
for_each = local.clusters
|
||||
name = each.key
|
||||
project_id = module.gke-project-0.project_id
|
||||
description = each.value.description
|
||||
location = each.value.location
|
||||
network = var.vpc_config.vpc_self_link
|
||||
subnetwork = each.value.net.subnet
|
||||
secondary_range_pods = each.value.net.pods
|
||||
secondary_range_services = each.value.net.services
|
||||
labels = each.value.labels
|
||||
addons = {
|
||||
cloudrun_config = each.value.overrides.cloudrun_config
|
||||
dns_cache_config = true
|
||||
http_load_balancing = true
|
||||
gce_persistent_disk_csi_driver_config = true
|
||||
horizontal_pod_autoscaling = true
|
||||
config_connector_config = true
|
||||
kalm_config = false
|
||||
gcp_filestore_csi_driver_config = each.value.overrides.gcp_filestore_csi_driver_config
|
||||
gke_backup_agent_config = false
|
||||
# enable only if enable_dataplane_v2 is changed to false below
|
||||
network_policy_config = false
|
||||
istio_config = {
|
||||
enabled = false
|
||||
tls = false
|
||||
}
|
||||
}
|
||||
# change these here for all clusters if absolutely needed
|
||||
authenticator_security_group = var.authenticator_security_group
|
||||
enable_dataplane_v2 = true
|
||||
enable_l4_ilb_subsetting = false
|
||||
enable_intranode_visibility = true
|
||||
enable_shielded_nodes = true
|
||||
workload_identity = true
|
||||
private_cluster_config = {
|
||||
enable_private_nodes = true
|
||||
enable_private_endpoint = false
|
||||
master_ipv4_cidr_block = each.value.net.master_range
|
||||
master_global_access = true
|
||||
}
|
||||
dns_config = each.value.dns_domain == null ? null : {
|
||||
cluster_dns = "CLOUD_DNS"
|
||||
cluster_dns_scope = "VPC_SCOPE"
|
||||
cluster_dns_domain = "${each.key}.${var.dns_domain}"
|
||||
}
|
||||
logging_config = ["SYSTEM_COMPONENTS", "WORKLOADS"]
|
||||
monitoring_config = ["SYSTEM_COMPONENTS", "WORKLOADS"]
|
||||
|
||||
peering_config = var.peering_config == null ? null : {
|
||||
export_routes = var.peering_config.export_routes
|
||||
import_routes = var.peering_config.import_routes
|
||||
project_id = var.vpc_config.host_project_id
|
||||
}
|
||||
resource_usage_export_config = {
|
||||
enabled = true
|
||||
dataset = module.gke-dataset-resource-usage.dataset_id
|
||||
}
|
||||
# TODO: the attributes below are "primed" from project-level defaults
|
||||
# in locals, merge defaults with cluster-level stuff
|
||||
# TODO(jccb): change fabric module
|
||||
database_encryption = (
|
||||
each.value.overrides.database_encryption_key == null
|
||||
? {
|
||||
enabled = false
|
||||
state = null
|
||||
key_name = null
|
||||
}
|
||||
: {
|
||||
enabled = true
|
||||
state = "ENCRYPTED"
|
||||
key_name = each.value.overrides.database_encryption_key
|
||||
}
|
||||
)
|
||||
default_max_pods_per_node = each.value.overrides.max_pods_per_node
|
||||
master_authorized_ranges = each.value.overrides.master_authorized_ranges
|
||||
pod_security_policy = each.value.overrides.pod_security_policy
|
||||
release_channel = each.value.overrides.release_channel
|
||||
vertical_pod_autoscaling = each.value.overrides.vertical_pod_autoscaling
|
||||
# dynamic "cluster_autoscaling" {
|
||||
# for_each = each.value.cluster_autoscaling == null ? {} : { 1 = 1 }
|
||||
# content {
|
||||
# enabled = true
|
||||
# cpu_min = each.value.cluster_autoscaling.cpu_min
|
||||
# cpu_max = each.value.cluster_autoscaling.cpu_max
|
||||
# memory_min = each.value.cluster_autoscaling.memory_min
|
||||
# memory_max = each.value.cluster_autoscaling.memory_max
|
||||
# }
|
||||
# }
|
||||
}
|
||||
44
blueprints/gke/multitenant-fleet/gke-hub.tf
Normal file
44
blueprints/gke/multitenant-fleet/gke-hub.tf
Normal file
@@ -0,0 +1,44 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
fleet_enabled = (
|
||||
var.fleet_features != null || var.fleet_workload_identity
|
||||
)
|
||||
fleet_mcs_enabled = (
|
||||
try(var.fleet_features.multiclusterservicediscovery, false) == true
|
||||
)
|
||||
}
|
||||
|
||||
module "gke-hub" {
|
||||
source = "../../../modules/gke-hub"
|
||||
count = local.fleet_enabled ? 1 : 0
|
||||
project_id = module.gke-project-0.project_id
|
||||
clusters = {
|
||||
for cluster_id in keys(var.clusters) :
|
||||
cluster_id => module.gke-cluster[cluster_id].id
|
||||
}
|
||||
features = var.fleet_features
|
||||
configmanagement_templates = var.fleet_configmanagement_templates
|
||||
configmanagement_clusters = var.fleet_configmanagement_clusters
|
||||
workload_identity_clusters = (
|
||||
var.fleet_workload_identity ? keys(var.clusters) : []
|
||||
)
|
||||
|
||||
depends_on = [
|
||||
module.gke-nodepool
|
||||
]
|
||||
}
|
||||
66
blueprints/gke/multitenant-fleet/gke-nodepools.tf
Normal file
66
blueprints/gke/multitenant-fleet/gke-nodepools.tf
Normal file
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
locals {
|
||||
nodepools = merge([
|
||||
for cluster, nodepools in var.nodepools : {
|
||||
for nodepool, config in nodepools :
|
||||
"${cluster}/${nodepool}" => merge(config, {
|
||||
name = nodepool
|
||||
cluster = cluster
|
||||
overrides = coalesce(config.overrides, var.nodepool_defaults)
|
||||
})
|
||||
}
|
||||
]...)
|
||||
}
|
||||
|
||||
module "gke-nodepool" {
|
||||
source = "../../../modules/gke-nodepool"
|
||||
for_each = local.nodepools
|
||||
name = each.value.name
|
||||
project_id = module.gke-project-0.project_id
|
||||
cluster_name = module.gke-cluster[each.value.cluster].name
|
||||
location = module.gke-cluster[each.value.cluster].location
|
||||
initial_node_count = each.value.initial_node_count
|
||||
node_machine_type = each.value.node_type
|
||||
node_spot = each.value.spot
|
||||
|
||||
node_count = each.value.node_count
|
||||
# node_count = (
|
||||
# each.value.autoscaling_config == null ? each.value.node_count : null
|
||||
# )
|
||||
# dynamic "autoscaling_config" {
|
||||
# for_each = each.value.autoscaling_config == null ? {} : { 1 = 1 }
|
||||
# content {
|
||||
# min_node_count = each.value.autoscaling_config.min_node_count
|
||||
# max_node_count = each.value.autoscaling_config.max_node_count
|
||||
# }
|
||||
# }
|
||||
|
||||
# overrides
|
||||
node_locations = each.value.overrides.node_locations
|
||||
max_pods_per_node = each.value.overrides.max_pods_per_node
|
||||
node_image_type = each.value.overrides.image_type
|
||||
node_tags = each.value.overrides.node_tags
|
||||
node_taints = each.value.overrides.node_taints
|
||||
|
||||
management_config = {
|
||||
auto_repair = true
|
||||
auto_upgrade = true
|
||||
}
|
||||
|
||||
node_service_account_create = true
|
||||
}
|
||||
82
blueprints/gke/multitenant-fleet/main.tf
Normal file
82
blueprints/gke/multitenant-fleet/main.tf
Normal file
@@ -0,0 +1,82 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
module "gke-project-0" {
|
||||
source = "../../../modules/project"
|
||||
billing_account = var.billing_account_id
|
||||
name = var.project_id
|
||||
parent = var.folder_id
|
||||
prefix = var.prefix
|
||||
group_iam = var.group_iam
|
||||
labels = var.labels
|
||||
iam = merge(var.iam, {
|
||||
"roles/gkehub.serviceAgent" = [
|
||||
"serviceAccount:${module.gke-project-0.service_accounts.robots.fleet}"
|
||||
] }
|
||||
)
|
||||
services = concat(
|
||||
[
|
||||
"anthos.googleapis.com",
|
||||
"anthosconfigmanagement.googleapis.com",
|
||||
"cloudresourcemanager.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"dns.googleapis.com",
|
||||
"gkeconnect.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
"iam.googleapis.com",
|
||||
"multiclusteringress.googleapis.com",
|
||||
"multiclusterservicediscovery.googleapis.com",
|
||||
"stackdriver.googleapis.com",
|
||||
"trafficdirector.googleapis.com"
|
||||
],
|
||||
var.project_services
|
||||
)
|
||||
shared_vpc_service_config = {
|
||||
attach = true
|
||||
host_project = var.vpc_config.host_project_id
|
||||
service_identity_iam = merge({
|
||||
"roles/compute.networkUser" = [
|
||||
"cloudservices", "container-engine"
|
||||
]
|
||||
"roles/container.hostServiceAgentUser" = [
|
||||
"container-engine"
|
||||
]
|
||||
},
|
||||
!local.fleet_mcs_enabled ? {} : {
|
||||
"roles/multiclusterservicediscovery.serviceAgent" = ["gke-mcs"]
|
||||
"roles/compute.networkViewer" = ["gke-mcs-importer"]
|
||||
})
|
||||
}
|
||||
# specify project-level org policies here if you need them
|
||||
# policy_boolean = {
|
||||
# "constraints/compute.disableGuestAttributesAccess" = true
|
||||
# }
|
||||
# policy_list = {
|
||||
# "constraints/compute.trustedImageProjects" = {
|
||||
# inherit_from_parent = null
|
||||
# suggested_value = null
|
||||
# status = true
|
||||
# values = ["projects/fl01-prod-iac-core-0"]
|
||||
# }
|
||||
# }
|
||||
}
|
||||
|
||||
module "gke-dataset-resource-usage" {
|
||||
source = "../../../modules/bigquery-dataset"
|
||||
project_id = module.gke-project-0.project_id
|
||||
id = "gke_resource_usage"
|
||||
friendly_name = "GKE resource usage."
|
||||
}
|
||||
32
blueprints/gke/multitenant-fleet/outputs.tf
Normal file
32
blueprints/gke/multitenant-fleet/outputs.tf
Normal file
@@ -0,0 +1,32 @@
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# tfdoc:file:description Output variables.
|
||||
|
||||
output "clusters" {
|
||||
description = "Cluster resources."
|
||||
value = module.gke-cluster
|
||||
}
|
||||
|
||||
output "cluster_ids" {
|
||||
description = "Cluster ids."
|
||||
value = {
|
||||
for k, v in module.gke-cluster : k => v.id
|
||||
}
|
||||
}
|
||||
|
||||
output "project_id" {
|
||||
description = "GKE project id."
|
||||
value = module.gke-project-0.project_id
|
||||
}
|
||||
254
blueprints/gke/multitenant-fleet/variables.tf
Normal file
254
blueprints/gke/multitenant-fleet/variables.tf
Normal file
@@ -0,0 +1,254 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "authenticator_security_group" {
|
||||
description = "Optional group used for Groups for GKE."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "billing_account_id" {
|
||||
description = "Billing account id."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "cluster_defaults" {
|
||||
description = "Default values for optional cluster configurations."
|
||||
type = object({
|
||||
cloudrun_config = bool
|
||||
database_encryption_key = string
|
||||
master_authorized_ranges = map(string)
|
||||
max_pods_per_node = number
|
||||
pod_security_policy = bool
|
||||
release_channel = string
|
||||
vertical_pod_autoscaling = bool
|
||||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
default = {
|
||||
# TODO: review defaults
|
||||
cloudrun_config = false
|
||||
database_encryption_key = null
|
||||
master_authorized_ranges = {
|
||||
rfc1918_1 = "10.0.0.0/8"
|
||||
rfc1918_2 = "172.16.0.0/12"
|
||||
rfc1918_3 = "192.168.0.0/16"
|
||||
}
|
||||
max_pods_per_node = 110
|
||||
pod_security_policy = false
|
||||
release_channel = "STABLE"
|
||||
vertical_pod_autoscaling = false
|
||||
gcp_filestore_csi_driver_config = false
|
||||
}
|
||||
}
|
||||
|
||||
variable "clusters" {
|
||||
description = ""
|
||||
type = map(object({
|
||||
cluster_autoscaling = object({
|
||||
cpu_min = number
|
||||
cpu_max = number
|
||||
memory_min = number
|
||||
memory_max = number
|
||||
})
|
||||
description = string
|
||||
dns_domain = string
|
||||
labels = map(string)
|
||||
location = string
|
||||
net = object({
|
||||
master_range = string
|
||||
pods = string
|
||||
services = string
|
||||
subnet = string
|
||||
})
|
||||
overrides = object({
|
||||
cloudrun_config = bool
|
||||
database_encryption_key = string
|
||||
# binary_authorization = bool
|
||||
master_authorized_ranges = map(string)
|
||||
max_pods_per_node = number
|
||||
pod_security_policy = bool
|
||||
release_channel = string
|
||||
vertical_pod_autoscaling = bool
|
||||
gcp_filestore_csi_driver_config = bool
|
||||
})
|
||||
}))
|
||||
}
|
||||
|
||||
variable "dns_domain" {
|
||||
description = "Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE."
|
||||
type = string
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_clusters" {
|
||||
description = "Config management features enabled on specific sets of member clusters, in config name => [cluster name] format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_templates" {
|
||||
description = "Sets of config management configurations that can be applied to member clusters, in config name => {options} format."
|
||||
type = map(object({
|
||||
binauthz = bool
|
||||
config_sync = object({
|
||||
git = object({
|
||||
gcp_service_account_email = string
|
||||
https_proxy = string
|
||||
policy_dir = string
|
||||
secret_type = string
|
||||
sync_branch = string
|
||||
sync_repo = string
|
||||
sync_rev = string
|
||||
sync_wait_secs = number
|
||||
})
|
||||
prevent_drift = string
|
||||
source_format = string
|
||||
})
|
||||
hierarchy_controller = object({
|
||||
enable_hierarchical_resource_quota = bool
|
||||
enable_pod_tree_labels = bool
|
||||
})
|
||||
policy_controller = object({
|
||||
audit_interval_seconds = number
|
||||
exemptable_namespaces = list(string)
|
||||
log_denies_enabled = bool
|
||||
referential_rules_enabled = bool
|
||||
template_library_installed = bool
|
||||
})
|
||||
version = string
|
||||
}))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "fleet_features" {
|
||||
description = "Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used."
|
||||
type = object({
|
||||
appdevexperience = bool
|
||||
configmanagement = bool
|
||||
identityservice = bool
|
||||
multiclusteringress = string
|
||||
multiclusterservicediscovery = bool
|
||||
servicemesh = bool
|
||||
})
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_workload_identity" {
|
||||
description = "Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true."
|
||||
type = bool
|
||||
default = false
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "folder_id" {
|
||||
description = "Folder used for the GKE project in folders/nnnnnnnnnnn format."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "group_iam" {
|
||||
description = "Project-level IAM bindings for groups. Use group emails as keys, list of roles as values."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "labels" {
|
||||
description = "Project-level labels."
|
||||
type = map(string)
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "nodepool_defaults" {
|
||||
description = ""
|
||||
type = object({
|
||||
image_type = string
|
||||
max_pods_per_node = number
|
||||
node_locations = list(string)
|
||||
node_tags = list(string)
|
||||
node_taints = list(string)
|
||||
})
|
||||
default = {
|
||||
image_type = "COS_CONTAINERD"
|
||||
max_pods_per_node = 110
|
||||
node_locations = null
|
||||
node_tags = null
|
||||
node_taints = []
|
||||
}
|
||||
}
|
||||
|
||||
variable "nodepools" {
|
||||
description = ""
|
||||
type = map(map(object({
|
||||
node_count = number
|
||||
node_type = string
|
||||
initial_node_count = number
|
||||
overrides = object({
|
||||
image_type = string
|
||||
max_pods_per_node = number
|
||||
node_locations = list(string)
|
||||
node_tags = list(string)
|
||||
node_taints = list(string)
|
||||
})
|
||||
spot = bool
|
||||
})))
|
||||
}
|
||||
|
||||
variable "peering_config" {
|
||||
description = "Configure peering with the control plane VPC. Requires compute.networks.updatePeering. Set to null if you don't want to update the default peering configuration."
|
||||
type = object({
|
||||
export_routes = bool
|
||||
import_routes = bool
|
||||
})
|
||||
default = {
|
||||
export_routes = true
|
||||
// TODO(jccb) is there any situation where the control plane VPC would export any routes?
|
||||
import_routes = false
|
||||
}
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
description = "Prefix used for resources that need unique names."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "ID of the project that will contain all the clusters."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_services" {
|
||||
description = "Additional project services to enable."
|
||||
type = list(string)
|
||||
default = []
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "vpc_config" {
|
||||
description = "Shared VPC project and VPC details."
|
||||
type = object({
|
||||
host_project_id = string
|
||||
vpc_self_link = string
|
||||
})
|
||||
}
|
||||
Reference in New Issue
Block a user