|
|
|
|
@@ -10,34 +10,33 @@ TODO: add description and diagram
|
|
|
|
|
|
|
|
|
|
TODO
|
|
|
|
|
|
|
|
|
|
### Cluster and nodepool configuration
|
|
|
|
|
## How to run this stage
|
|
|
|
|
|
|
|
|
|
Clusters and nodepools configuration comes with variables that let you define common configuration (cluster_default) and define the single cluster details when needed. This stage is designed with multi-tenancy in mind, this means that the GKE clusters within this stage will be similar more often than not.
|
|
|
|
|
TODO
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
cluster_defaults = {
|
|
|
|
|
cloudrun_config = false
|
|
|
|
|
database_encryption_key = null
|
|
|
|
|
gcp_filestore_csi_driver_config = false
|
|
|
|
|
master_authorized_ranges = {
|
|
|
|
|
rfc1918_1 = "10.0.0.0/8"
|
|
|
|
|
rfc1918_2 = "172.16.0.0/12"
|
|
|
|
|
rfc1918_3 = "192.168.0.0/16"
|
|
|
|
|
on_prem = "100.xxx.xxx.xxx/xx"
|
|
|
|
|
}
|
|
|
|
|
max_pods_per_node = 110
|
|
|
|
|
pod_security_policy = false
|
|
|
|
|
release_channel = "STABLE"
|
|
|
|
|
vertical_pod_autoscaling = false
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
The variable cluster_defaults is used as a shared configuration between all the clusters within the projects, clusters variable instead can be used to define the required specific details:
|
|
|
|
|
### Providers configuration
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
TODO
|
|
|
|
|
|
|
|
|
|
### Variable configuration
|
|
|
|
|
|
|
|
|
|
#### Cluster and nodepools
|
|
|
|
|
|
|
|
|
|
This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Stage variables are designed to support this approach for both clusters and nodepools:
|
|
|
|
|
|
|
|
|
|
- the `cluster_default` variable allows defining common defaults for cluster
|
|
|
|
|
- the `clusters` variable is used to declare the actual GKE clusters and allows overriding defaults on a per-cluster basis
|
|
|
|
|
- the `nodepool_defaults` variable allows definining common defaults for nodepools
|
|
|
|
|
- the `nodepools` variable is used to declare cluster nodepools and allows overriding defaults on a per-cluster basis
|
|
|
|
|
|
|
|
|
|
This is an example of that shows the use of the above variables:
|
|
|
|
|
|
|
|
|
|
```hcl
|
|
|
|
|
# the `cluster_defaults` variable defaults are used and not shown here
|
|
|
|
|
clusters = {
|
|
|
|
|
"gke-1" = {
|
|
|
|
|
"gke-00" = {
|
|
|
|
|
cluster_autoscaling = null
|
|
|
|
|
description = "gke-1"
|
|
|
|
|
description = "gke-00"
|
|
|
|
|
dns_domain = null
|
|
|
|
|
location = "europe-west1"
|
|
|
|
|
labels = {}
|
|
|
|
|
@@ -45,13 +44,13 @@ clusters = {
|
|
|
|
|
master_range = "172.17.16.0/28"
|
|
|
|
|
pods = "pods"
|
|
|
|
|
services = "services"
|
|
|
|
|
subnet = "https://www.googleapis.com/compute/v1/projects/XXXX-dev-net-spoke-0/regions/europe-west1/subnetworks/dev-gke-nodes-ew1"
|
|
|
|
|
subnet = local.vpc.subnet_self_links["europe-west3/gke-dev-0"]
|
|
|
|
|
}
|
|
|
|
|
overrides = null
|
|
|
|
|
}
|
|
|
|
|
"gke-2" = {
|
|
|
|
|
"gke-01" = {
|
|
|
|
|
cluster_autoscaling = null
|
|
|
|
|
description = "gke-2"
|
|
|
|
|
description = "gke-01"
|
|
|
|
|
dns_domain = null
|
|
|
|
|
location = "europe-west3"
|
|
|
|
|
labels = {}
|
|
|
|
|
@@ -59,19 +58,25 @@ clusters = {
|
|
|
|
|
master_range = "172.17.17.0/28"
|
|
|
|
|
pods = "pods"
|
|
|
|
|
services = "services"
|
|
|
|
|
subnet = "https://www.googleapis.com/compute/v1/projects/XXXX-dev-net-spoke-0/regions/europe-west3/subnetworks/dev-gke-nodes-ew3"
|
|
|
|
|
subnet = local.vpc.subnet_self_links["europe-west3/gke-dev-0"]
|
|
|
|
|
}
|
|
|
|
|
overrides = {
|
|
|
|
|
cloudrun_config = false
|
|
|
|
|
database_encryption_key = null
|
|
|
|
|
gcp_filestore_csi_driver_config = true
|
|
|
|
|
master_authorized_ranges = {
|
|
|
|
|
rfc1918_1 = "10.0.0.0/8"
|
|
|
|
|
}
|
|
|
|
|
max_pods_per_node = 64
|
|
|
|
|
pod_security_policy = true
|
|
|
|
|
release_channel = "STABLE"
|
|
|
|
|
vertical_pod_autoscaling = false
|
|
|
|
|
}
|
|
|
|
|
overrides = null
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
The same design principle used for the clusters variable is used to define the nodepool and attach them to the right GKE cluster (previously defined)
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
nodepools = {
|
|
|
|
|
"gke-1" = {
|
|
|
|
|
"np-001" = {
|
|
|
|
|
"gke-0" = {
|
|
|
|
|
"gke-00-000" = {
|
|
|
|
|
initial_node_count = 1
|
|
|
|
|
node_count = 1
|
|
|
|
|
node_type = "n2-standard-4"
|
|
|
|
|
@@ -79,21 +84,27 @@ nodepools = {
|
|
|
|
|
spot = false
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
"gke-2" = {
|
|
|
|
|
"np-002" = {
|
|
|
|
|
"gke-1" = {
|
|
|
|
|
"gke-01-000" = {
|
|
|
|
|
initial_node_count = 1
|
|
|
|
|
node_count = 1
|
|
|
|
|
node_type = "n2-standard-4"
|
|
|
|
|
overrides = null
|
|
|
|
|
overrides = {
|
|
|
|
|
image_type = "UBUNTU_CONTAINERD"
|
|
|
|
|
max_pods_per_node = 64
|
|
|
|
|
node_locations = []
|
|
|
|
|
node_tags = []
|
|
|
|
|
node_taints = []
|
|
|
|
|
}
|
|
|
|
|
spot = true
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
On the top of all the clusters configuration, the variable authenticator_security_group can be used to define the google group that should be used within Google Groups for RBAC feature as authenticator security group.
|
|
|
|
|
There are two additional variables that influence cluster configuration: `authenticator_security_group` to configure Google Groups for RBAC, `dns_domain` to configure Cloud DNS for GKE.
|
|
|
|
|
|
|
|
|
|
### Fleet management
|
|
|
|
|
#### Fleet management
|
|
|
|
|
|
|
|
|
|
Fleet management is achieved by the configuration of the fleet_configmanagement_templates, fleet_configmanagement_clusters and fleet_features variables exposed by the module in _module. In details fleet_features lets you activate the fleet features, fleet_configmanagement_templates lets you define one o more fleet configmanagement configuration template to be activated onto one or more GKE clusters. Configured features and settings can be applied to clusters by leveraging fleet_configmanagement_clusters where a single template can be applied to one or more clusters.
|
|
|
|
|
|
|
|
|
|
@@ -142,7 +153,6 @@ fleet_features = {
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## How to run this stage
|
|
|
|
|
|
|
|
|
|
This stage is meant to be executed after "foundational stages" (i.e., stages [`00-bootstrap`](../../00-bootstrap), [`01-resman`](../../01-resman), 02-networking (either [VPN](../../02-networking-vpn) or [NVA](../../02-networking-nva)) and [`02-security`](../../02-security)) have been run.
|
|
|
|
|
@@ -176,25 +186,25 @@ TODO
|
|
|
|
|
|---|---|:---:|:---:|:---:|:---:|
|
|
|
|
|
| [automation](variables.tf#L21) | Automation resources created by the bootstrap stage. | <code title="object({ outputs_bucket = string })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
|
|
|
|
| [billing_account](variables.tf#L35) | Billing account id and organization id ('nnnnnnnn' or null). | <code title="object({ id = string organization_id = number })">object({…})</code> | ✓ | | <code>00-bootstrap</code> |
|
|
|
|
|
| [clusters](variables.tf#L74) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
|
|
|
|
| [folder_ids](variables.tf#L176) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object({ gke-dev = string })">object({…})</code> | ✓ | | <code>01-resman</code> |
|
|
|
|
|
| [host_project_ids](variables.tf#L198) | Host project for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
|
|
|
|
| [nodepools](variables.tf#L230) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
|
|
|
|
| [prefix](variables.tf#L253) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
|
|
|
|
| [vpc_self_links](variables.tf#L265) | Self link for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
|
|
|
|
| [clusters](variables.tf#L73) | | <code title="map(object({ cluster_autoscaling = object({ cpu_min = number cpu_max = number memory_min = number memory_max = number }) description = string dns_domain = string labels = map(string) location = string net = object({ master_range = string pods = string services = string subnet = string }) overrides = object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool }) }))">map(object({…}))</code> | ✓ | | |
|
|
|
|
|
| [folder_ids](variables.tf#L175) | Folders to be used for the networking resources in folders/nnnnnnnnnnn format. If null, folder will be created. | <code title="object({ gke-dev = string })">object({…})</code> | ✓ | | <code>01-resman</code> |
|
|
|
|
|
| [host_project_ids](variables.tf#L197) | Host project for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
|
|
|
|
| [nodepools](variables.tf#L229) | | <code title="map(map(object({ node_count = number node_type = string initial_node_count = number overrides = object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) }) spot = bool })))">map(map(object({…})))</code> | ✓ | | |
|
|
|
|
|
| [prefix](variables.tf#L252) | Prefix used for resources that need unique names. | <code>string</code> | ✓ | | |
|
|
|
|
|
| [vpc_self_links](variables.tf#L264) | Self link for the shared VPC. | <code title="object({ dev-spoke-0 = string })">object({…})</code> | ✓ | | <code>02-networking</code> |
|
|
|
|
|
| [authenticator_security_group](variables.tf#L29) | Optional group used for Groups for GKE. | <code>string</code> | | <code>null</code> | |
|
|
|
|
|
| [cluster_defaults](variables.tf#L44) | Default values for optional cluster configurations. | <code title="object({ cloudrun_config = bool database_encryption_key = string master_authorized_ranges = map(string) max_pods_per_node = number pod_security_policy = bool release_channel = string vertical_pod_autoscaling = bool gcp_filestore_csi_driver_config = bool })">object({…})</code> | | <code title="{ cloudrun_config = false database_encryption_key = null master_authorized_ranges = { rfc1918_1 = "10.0.0.0/8" rfc1918_2 = "172.16.0.0/12" rfc1918_3 = "192.168.0.0/16" } max_pods_per_node = 110 pod_security_policy = false release_channel = "STABLE" vertical_pod_autoscaling = false gcp_filestore_csi_driver_config = false }">{…}</code> | |
|
|
|
|
|
| [dns_domain](variables.tf#L107) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
|
|
|
|
| [fleet_configmanagement_clusters](variables.tf#L113) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
|
|
|
|
| [fleet_configmanagement_templates](variables.tf#L121) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
|
|
|
|
| [fleet_features](variables.tf#L156) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
|
|
|
|
| [fleet_workload_identity](variables.tf#L169) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>true</code> | |
|
|
|
|
|
| [group_iam](variables.tf#L184) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
|
|
|
|
| [iam](variables.tf#L191) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
|
|
|
|
| [labels](variables.tf#L206) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
|
|
|
|
| [nodepool_defaults](variables.tf#L212) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
|
|
|
|
| [outputs_location](variables.tf#L247) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
|
|
|
|
|
| [project_services](variables.tf#L258) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
|
|
|
|
| [dns_domain](variables.tf#L106) | Domain name used for clusters, prefixed by each cluster name. Leave null to disable Cloud DNS for GKE. | <code>string</code> | | <code>null</code> | |
|
|
|
|
|
| [fleet_configmanagement_clusters](variables.tf#L112) | Config management features enabled on specific sets of member clusters, in config name => [cluster name] format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
|
|
|
|
| [fleet_configmanagement_templates](variables.tf#L120) | Sets of config management configurations that can be applied to member clusters, in config name => {options} format. | <code title="map(object({ binauthz = bool config_sync = object({ git = object({ gcp_service_account_email = string https_proxy = string policy_dir = string secret_type = string sync_branch = string sync_repo = string sync_rev = string sync_wait_secs = number }) prevent_drift = string source_format = string }) hierarchy_controller = object({ enable_hierarchical_resource_quota = bool enable_pod_tree_labels = bool }) policy_controller = object({ audit_interval_seconds = number exemptable_namespaces = list(string) log_denies_enabled = bool referential_rules_enabled = bool template_library_installed = bool }) version = string }))">map(object({…}))</code> | | <code>{}</code> | |
|
|
|
|
|
| [fleet_features](variables.tf#L155) | Enable and configue fleet features. Set to null to disable GKE Hub if fleet workload identity is not used. | <code title="object({ appdevexperience = bool configmanagement = bool identityservice = bool multiclusteringress = string multiclusterservicediscovery = bool servicemesh = bool })">object({…})</code> | | <code>null</code> | |
|
|
|
|
|
| [fleet_workload_identity](variables.tf#L168) | Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true. | <code>bool</code> | | <code>true</code> | |
|
|
|
|
|
| [group_iam](variables.tf#L183) | Project-level authoritative IAM bindings for groups in {GROUP_EMAIL => [ROLES]} format. Use group emails as keys, list of roles as values. | <code>map(list(string))</code> | | <code>{}</code> | |
|
|
|
|
|
| [iam](variables.tf#L190) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
|
|
|
|
| [labels](variables.tf#L205) | Project-level labels. | <code>map(string)</code> | | <code>{}</code> | |
|
|
|
|
|
| [nodepool_defaults](variables.tf#L211) | | <code title="object({ image_type = string max_pods_per_node = number node_locations = list(string) node_tags = list(string) node_taints = list(string) })">object({…})</code> | | <code title="{ image_type = "COS_CONTAINERD" max_pods_per_node = 110 node_locations = null node_tags = null node_taints = [] }">{…}</code> | |
|
|
|
|
|
| [outputs_location](variables.tf#L246) | Path where providers, tfvars files, and lists for the following stages are written. Leave empty to disable. | <code>string</code> | | <code>null</code> | |
|
|
|
|
|
| [project_services](variables.tf#L257) | Additional project services to enable. | <code>list(string)</code> | | <code>[]</code> | |
|
|
|
|
|
|
|
|
|
|
## Outputs
|
|
|
|
|
|
|
|
|
|
|
Ludovico Magnocavallo