hunfabric/fast/stages/3-gke-dev

GKE Multitenant

This stage allows creation and management of a fleet of GKE multitenant clusters for a single environment, optionally leveraging GKE Hub to configure additional features.

The following diagram illustrates the high-level design of created resources, which can be adapted to specific requirements via variables:

• Design overview and choices
• How to run this stage
  • FAST prerequisites
• Customizations
  • Clusters and node pools
  • Fleet management
• Files
• Variables
• Outputs

Design overview and choices

The general idea behind this stage is to deploy a single project hosting multiple clusters leveraging several useful GKE features like Config Sync, which lend themselves well to a multitenant approach to GKE.

Some high level choices applied here:

• all clusters are created as private clusters which then need to be VPC-native.
• Logging and monitoring uses Cloud Operations for system components and user workloads.
• GKE metering is enabled by default and stored in a BigQuery dataset created within the project.
• GKE Fleet can be optionally with support for the following features:
  • Fleet workload identity
  • Config Management
  • Service Mesh
  • Identity Service
  • Multi-cluster services
  • Multi-cluster ingress.
• Support for Config Sync and Hierarchy Controller when using Config Management.
• Groups for GKE can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
• Support for application layer secret encryption.
• Some features are enabled by default in all clusters:
  • Intranode visibility
  • Dataplane v2
  • Shielded GKE nodes
  • Workload identity
  • Node local DNS cache
  • Use of the GCE persistent disk CSI driver
  • Node auto-upgrade and auto-repair for all node pools

How to run this stage

If this stage is deployed within a FAST-based GCP organization, we recommend executing it after foundational FAST stage-2 components like networking and security. This is the recommended flow as specific data platform features in this stage might depend on configurations from these earlier stages. Although this stage can be run independently, instructions for such a standalone setup are beyond the scope of this document.

FAST prerequisites

This stage needs specific automation resources, and permissions granted on those that allow control of selective IAM roles on specific networking and security resources.

Network permissions are needed to associate data domain or product projects to Shared VPC hosts and grant network permissions to data platform managed service accounts. They are mandatory when deploying Composer.

Security permissions are only needed when using CMEK encryption, to grant the relevant IAM roles to data platform service agents on the encryption keys used.

The "Classic FAST" dataset in the bootstrap stage contains the configuration for a development Data Platform that can be easily adapted to serve for this stage.

Customizations

This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Variables allow management of clusters, nodepools, and fleet registration and configurations.

Clusters and node pools

This is an example of declaring a private cluster with one nodepool via tfvars file:

clusters = {
  test-00 = {
    description = "Cluster test 0"
    location    = "europe-west8"
    private_cluster_config = {
      enable_private_endpoint = true
      master_global_access    = true
    }
    vpc_config = {
      subnetwork             = "projects/ldj-dev-net-spoke-0/regions/europe-west8/subnetworks/gke"
      master_ipv4_cidr_block = "172.16.20.0/28"
      master_authorized_ranges = {
        private = "10.0.0.0/8"
      }
    }
  }
}
nodepools = {
  test-00 = {
    00 = {
      node_count = { initial = 1 }
    }
  }
}
# tftest skip

If clusters share similar configurations, those can be centralized via locals blocks in this stage's main.tf file, and merged in with clusters via a simple for_each loop.

Fleet management

Fleet management is entirely optional, and uses two separate variables:

• fleet_config: specifies the GKE fleet features to activate
• fleet_configmanagement_templates: defines configuration templates for specific sets of features (Config Management currently)

Clusters can then be configured for fleet registration and one of the config management templates attached via the cluster-level fleet_config attribute.

Files

name	description	modules	resources
gke-clusters.tf	GKE clusters.	gke-cluster-standard · gke-nodepool
gke-hub.tf	GKE hub configuration.	gke-hub
main.tf	Project and usage dataset.	bigquery-dataset · iam-service-account · project
outputs.tf	Module outputs.		google_storage_bucket_object
variables-fast.tf	None
variables-fleet.tf	GKE fleet configurations.
variables.tf	Module variables.

Variables

name	description	type	required	default	producer
automation	Automation resources created by the bootstrap stage.	object({…})	✓		0-org-setup
billing_account	Billing account id. If billing account is not part of the same org set is_org_level to false.	object({…})	✓		0-org-setup
environments	Long environment names.	object({…})	✓		1-resman
prefix	Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants.	string	✓		0-org-setup
clusters	Clusters configuration. Refer to the gke-cluster module for type details.	map(object({…}))		{}
deletion_protection	Prevent Terraform from destroying data resources.	bool		false
fleet_config	Fleet configuration.	object({…})		null
fleet_configmanagement_templates	Sets of fleet configurations that can be applied to member clusters, in config name => {options} format.	map(object({…}))		{}
folder_ids	Folder name => id mappings.	map(string)		{}	1-resman
host_project_ids	Shared VPC host project name => id mappings.	map(string)		{}	2-networking
iam	Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format.	map(list(string))		{}
iam_by_principals	Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the iam variable.	map(list(string))		{}
nodepools	Nodepools configuration. Refer to the gke-nodepool module for type details.	map(map(object({…})))		{}
stage_config	FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management.	object({…})		{…}
subnet_self_links	Subnet VPC name => { name => self link } mappings.	map(map(string))		{}	2-networking
vpc_config	VPC-level configuration for project and clusters.	object({…})		{…}
vpc_self_links	Shared VPC name => self link mappings.	map(string)		{}	2-networking

Outputs

name	description	sensitive	consumers
cluster_ids	Cluster ids.
clusters	Cluster resources.	✓
project_id	GKE project id.