diff --git a/fast/stages/03-gke-multitenant/_module/README.md b/fast/stages/03-gke-multitenant/_module/README.md index 6189adaa3..d4f956ce8 100644 --- a/fast/stages/03-gke-multitenant/_module/README.md +++ b/fast/stages/03-gke-multitenant/_module/README.md @@ -1,26 +1,6 @@ # GKE Multitenant Module -TODO: add description and diagram - -

- GKE multitenant -

- -## Design overview and choices - -TODO - -### Cluster and nodepool configuration - -TODO - -### Fleet management - -TODO - -### Variable configuration - -TODO +TODO: add brief explanation and refer back to dev folder? diff --git a/fast/stages/03-gke-multitenant/_module/gke-clusters.tf b/fast/stages/03-gke-multitenant/_module/gke-clusters.tf index 79794e566..4b102bd31 100644 --- a/fast/stages/03-gke-multitenant/_module/gke-clusters.tf +++ b/fast/stages/03-gke-multitenant/_module/gke-clusters.tf @@ -43,7 +43,7 @@ module "gke-cluster" { horizontal_pod_autoscaling = true config_connector_config = true kalm_config = false - gcp_filestore_csi_driver_config = false + gcp_filestore_csi_driver_config = each.value.overrides.gcp_filestore_csi_driver_config gke_backup_agent_config = false # enable only if enable_dataplane_v2 is changed to false below network_policy_config = false diff --git a/fast/stages/03-gke-multitenant/dev/README.md b/fast/stages/03-gke-multitenant/dev/README.md index 2c6eb8bae..db65df14e 100644 --- a/fast/stages/03-gke-multitenant/dev/README.md +++ b/fast/stages/03-gke-multitenant/dev/README.md @@ -10,34 +10,33 @@ TODO: add description and diagram TODO -### Cluster and nodepool configuration +## How to run this stage -Clusters and nodepools configuration comes with variables that let you define common configuration (cluster_default) and define the single cluster details when needed. This stage is designed with multi-tenancy in mind, this means that the GKE clusters within this stage will be similar more often than not. +TODO -``` -cluster_defaults = { - cloudrun_config = false - database_encryption_key = null - gcp_filestore_csi_driver_config = false - master_authorized_ranges = { - rfc1918_1 = "10.0.0.0/8" - rfc1918_2 = "172.16.0.0/12" - rfc1918_3 = "192.168.0.0/16" - on_prem = "100.xxx.xxx.xxx/xx" - } - max_pods_per_node = 110 - pod_security_policy = false - release_channel = "STABLE" - vertical_pod_autoscaling = false -} -``` -The variable cluster_defaults is used as a shared configuration between all the clusters within the projects, clusters variable instead can be used to define the required specific details: +### Providers configuration -``` +TODO + +### Variable configuration + +#### Cluster and nodepools + +This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Stage variables are designed to support this approach for both clusters and nodepools: + +- the `cluster_default` variable allows defining common defaults for cluster +- the `clusters` variable is used to declare the actual GKE clusters and allows overriding defaults on a per-cluster basis +- the `nodepool_defaults` variable allows definining common defaults for nodepools +- the `nodepools` variable is used to declare cluster nodepools and allows overriding defaults on a per-cluster basis + +This is an example of that shows the use of the above variables: + +```hcl +# the `cluster_defaults` variable defaults are used and not shown here clusters = { - "gke-1" = { + "gke-00" = { cluster_autoscaling = null - description = "gke-1" + description = "gke-00" dns_domain = null location = "europe-west1" labels = {} @@ -45,13 +44,13 @@ clusters = { master_range = "172.17.16.0/28" pods = "pods" services = "services" - subnet = "https://www.googleapis.com/compute/v1/projects/XXXX-dev-net-spoke-0/regions/europe-west1/subnetworks/dev-gke-nodes-ew1" + subnet = local.vpc.subnet_self_links["europe-west3/gke-dev-0"] } overrides = null } - "gke-2" = { + "gke-01" = { cluster_autoscaling = null - description = "gke-2" + description = "gke-01" dns_domain = null location = "europe-west3" labels = {} @@ -59,19 +58,25 @@ clusters = { master_range = "172.17.17.0/28" pods = "pods" services = "services" - subnet = "https://www.googleapis.com/compute/v1/projects/XXXX-dev-net-spoke-0/regions/europe-west3/subnetworks/dev-gke-nodes-ew3" + subnet = local.vpc.subnet_self_links["europe-west3/gke-dev-0"] + } + overrides = { + cloudrun_config = false + database_encryption_key = null + gcp_filestore_csi_driver_config = true + master_authorized_ranges = { + rfc1918_1 = "10.0.0.0/8" + } + max_pods_per_node = 64 + pod_security_policy = true + release_channel = "STABLE" + vertical_pod_autoscaling = false } - overrides = null } } - -``` -The same design principle used for the clusters variable is used to define the nodepool and attach them to the right GKE cluster (previously defined) - -``` nodepools = { - "gke-1" = { - "np-001" = { + "gke-0" = { + "gke-00-000" = { initial_node_count = 1 node_count = 1 node_type = "n2-standard-4" @@ -79,21 +84,27 @@ nodepools = { spot = false } } - "gke-2" = { - "np-002" = { + "gke-1" = { + "gke-01-000" = { initial_node_count = 1 node_count = 1 node_type = "n2-standard-4" - overrides = null + overrides = { + image_type = "UBUNTU_CONTAINERD" + max_pods_per_node = 64 + node_locations = [] + node_tags = [] + node_taints = [] + } spot = true } } } ``` -On the top of all the clusters configuration, the variable authenticator_security_group can be used to define the google group that should be used within Google Groups for RBAC feature as authenticator security group. +There are two additional variables that influence cluster configuration: `authenticator_security_group` to configure Google Groups for RBAC, `dns_domain` to configure Cloud DNS for GKE. -### Fleet management +#### Fleet management Fleet management is achieved by the configuration of the fleet_configmanagement_templates, fleet_configmanagement_clusters and fleet_features variables exposed by the module in _module. In details fleet_features lets you activate the fleet features, fleet_configmanagement_templates lets you define one o more fleet configmanagement configuration template to be activated onto one or more GKE clusters. Configured features and settings can be applied to clusters by leveraging fleet_configmanagement_clusters where a single template can be applied to one or more clusters. @@ -142,7 +153,6 @@ fleet_features = { } ``` - ## How to run this stage This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../00-bootstrap), [`01-resman`](../../01-resman), 02-networking (either [VPN](../../02-networking-vpn) or [NVA](../../02-networking-nva)) and [`02-security`](../../02-security)) have been run. @@ -176,25 +186,25 @@ TODO |---|---|:---:|:---:|:---:|:---:| | [automation](variables.tf#L21) | Automation resources created by the bootstrap stage. | object({…}) | ✓ | | 00-bootstrap | | [billing_account](variables.tf#L35) | Billing account id and organization id ('nnnnnnnn' or null). | object({…}) | ✓ | | 00-bootstrap | -| [clusters](variables.tf#L74) | | map(object({…})) | ✓ | | | -| [folder_ids](variables.tf#L176) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | object({…}) | ✓ | | 01-resman | -| [host_project_ids](variables.tf#L198) | Host project for the shared VPC. | object({…}) | ✓ | | 02-networking | -| [nodepools](variables.tf#L230) | | map(map(object({…}))) | ✓ | | | -| [prefix](variables.tf#L253) | Prefix used for resources that need unique names. | string | ✓ | | | -| [vpc_self_links](variables.tf#L265) | Self link for the shared VPC. | object({…}) | ✓ | | 02-networking | +| [clusters](variables.tf#L73) | | map(object({…})) | ✓ | | | +| [folder_ids](variables.tf#L175) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | object({…}) | ✓ | | 01-resman | +| [host_project_ids](variables.tf#L197) | Host project for the shared VPC. | object({…}) | ✓ | | 02-networking | +| [nodepools](variables.tf#L229) | | map(map(object({…}))) | ✓ | | | +| [prefix](variables.tf#L252) | Prefix used for resources that need unique names. | string | ✓ | | | +| [vpc_self_links](variables.tf#L264) | Self link for the shared VPC. | object({…}) | ✓ | | 02-networking | | [authenticator_security_group](variables.tf#L29) | Optional group used for Groups for GKE. | string | | null | | | [cluster_defaults](variables.tf#L44) | Default values for optional cluster configurations. | object({…}) | | {…} | | -| [dns_domain](variables.tf#L107) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | string | | null | | -| [fleet_configmanagement_clusters](variables.tf#L113) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | map(list(string)) | | {} | | -| [fleet_configmanagement_templates](variables.tf#L121) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | map(object({…})) | | {} | | -| [fleet_features](variables.tf#L156) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | object({…}) | | null | | -| [fleet_workload_identity](variables.tf#L169) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | bool | | true | | -| [group_iam](variables.tf#L184) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | map(list(string)) | | {} | | -| [iam](variables.tf#L191) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | map(list(string)) | | {} | | -| [labels](variables.tf#L206) | Project-level labels. | map(string) | | {} | | -| [nodepool_defaults](variables.tf#L212) | | object({…}) | | {…} | | -| [outputs_location](variables.tf#L247) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | string | | null | | -| [project_services](variables.tf#L258) | Additional project services to enable. | list(string) | | [] | | +| [dns_domain](variables.tf#L106) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | string | | null | | +| [fleet_configmanagement_clusters](variables.tf#L112) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | map(list(string)) | | {} | | +| [fleet_configmanagement_templates](variables.tf#L120) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | map(object({…})) | | {} | | +| [fleet_features](variables.tf#L155) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | object({…}) | | null | | +| [fleet_workload_identity](variables.tf#L168) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | bool | | true | | +| [group_iam](variables.tf#L183) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | map(list(string)) | | {} | | +| [iam](variables.tf#L190) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | map(list(string)) | | {} | | +| [labels](variables.tf#L205) | Project-level labels. | map(string) | | {} | | +| [nodepool_defaults](variables.tf#L211) | | object({…}) | | {…} | | +| [outputs_location](variables.tf#L246) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | string | | null | | +| [project_services](variables.tf#L257) | Additional project services to enable. | list(string) | | [] | | ## Outputs diff --git a/fast/stages/03-gke-multitenant/dev/variables.tf b/fast/stages/03-gke-multitenant/dev/variables.tf index 6b9437709..7547f168e 100644 --- a/fast/stages/03-gke-multitenant/dev/variables.tf +++ b/fast/stages/03-gke-multitenant/dev/variables.tf @@ -54,7 +54,6 @@ variable "cluster_defaults" { gcp_filestore_csi_driver_config = bool }) default = { - # TODO: review defaults cloudrun_config = false database_encryption_key = null # binary_authorization = false