kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fa1e390af7655d24b2b39f073843985836aca83a
hunfabric/fast/stages/3-gke-dev
History

README.md

GKE Multitenant

This stage allows creation and management of a fleet of GKE multitenant clusters for a single environment, optionally leveraging GKE Hub to configure additional features.

The following diagram illustrates the high-level design of created resources, which can be adapted to specific requirements via variables:

GKE multitenant

Design overview and choices

The general idea behind this stage is to deploy a single project hosting multiple clusters leveraging several useful GKE features like Config Sync, which lend themselves well to a multitenant approach to GKE.

Some high level choices applied here:

How to run this stage

If this stage is deployed within a FAST-based GCP organization, we recommend executing it after foundational FAST stage-2 components like networking and security. This is the recommended flow as specific data platform features in this stage might depend on configurations from these earlier stages. Although this stage can be run independently, instructions for such a standalone setup are beyond the scope of this document.

FAST prerequisites

This stage needs specific automation resources, and permissions granted on those that allow control of selective IAM roles on specific networking and security resources.

Network permissions are needed to associate data domain or product projects to Shared VPC hosts and grant network permissions to data platform managed service accounts. They are mandatory when deploying Composer.

Security permissions are only needed when using CMEK encryption, to grant the relevant IAM roles to data platform service agents on the encryption keys used.

The "Classic FAST" dataset in the bootstrap stage contains the configuration for a development Data Platform that can be easily adapted to serve for this stage.

Customizations

This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Variables allow management of clusters, nodepools, and fleet registration and configurations.

Clusters and node pools

This is an example of declaring a private cluster with one nodepool via tfvars file:

clusters = {
  test-00 = {
    description = "Cluster test 0"
    location    = "europe-west8"
    private_cluster_config = {
      enable_private_endpoint = true
      master_global_access    = true
    }
    vpc_config = {
      subnetwork             = "projects/ldj-dev-net-spoke-0/regions/europe-west8/subnetworks/gke"
      master_ipv4_cidr_block = "172.16.20.0/28"
      master_authorized_ranges = {
        private = "10.0.0.0/8"
      }
    }
  }
}
nodepools = {
  test-00 = {
    00 = {
      node_count = { initial = 1 }
    }
  }
}
# tftest skip

And here another example of declaring a private cluster following hardening guidelines with one nodepool via tfvars file:

clusters = {
  test-00 = {
    description = "Hardened Cluster test 0"
    location    = "europe-west1"
    private_cluster_config = {
      enable_private_endpoint = true
      master_global_access    = true
    }
    access_config = {
      dns_access = true
      ip_access = {
        disable_public_endpoint = true
      }
      private_nodes = true
    }
    enable_features = {
      binary_authorization = true
      groups_for_rbac      = "gke-security-groups@example.com"
      intranode_visibility = true
      rbac_binding_config = {
        enable_insecure_binding_system_unauthenticated : false
        enable_insecure_binding_system_authenticated : false
      }
      shielded_nodes = true
      upgrade_notifications = {
        event_types = ["SECURITY_BULLETIN_EVENT", "UPGRADE_AVAILABLE_EVENT", "UPGRADE_INFO_EVENT", "UPGRADE_EVENT"]
      }
      workload_identity = true
    }
    vpc_config = {
      subnetwork             = "projects/ldj-dev-net-spoke-0/regions/europe-west8/subnetworks/gke"
      master_ipv4_cidr_block = "172.16.20.0/28"
      master_authorized_ranges = {
        private = "10.0.0.0/8"
      }
    }
  }
}
nodepools = {
  test-00 = {
    00 = {
      node_count = { initial = 1 }
      node_config = {
        sandbox_config_gvisor = true
      }
    }
  }
}
# tftest skip

If clusters share similar configurations, those can be centralized via locals blocks in this stage's main.tf file, and merged in with clusters via a simple for_each loop.

Fleet management

Fleet management is entirely optional, and uses two separate variables:

  • fleet_config: specifies the GKE fleet features to activate
  • fleet_configmanagement_templates: defines configuration templates for specific sets of features (Config Management currently)

Clusters can then be configured for fleet registration and one of the config management templates attached via the cluster-level fleet_config attribute.

Files

name description modules resources
gke-clusters.tf GKE clusters. gke-cluster-standard · gke-nodepool
gke-hub.tf GKE hub configuration. gke-hub
main.tf Project and usage dataset. bigquery-dataset · iam-service-account · project
outputs.tf Module outputs. google_storage_bucket_object
variables-fast.tf None
variables-fleet.tf GKE fleet configurations.
variables.tf Module variables.

Variables

name description type required default producer
automation Automation resources created by the bootstrap stage. object({…}) ✓ 0-org-setup
billing_account Billing account id. If billing account is not part of the same org set is_org_level to false. object({…}) ✓ 0-org-setup
environments Long environment names. object({…}) ✓ 0-org-setup
prefix Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants. string ✓ 0-org-setup
clusters Clusters configuration. Refer to the gke-cluster module for type details. map(object({…})) {}
deletion_protection Prevent Terraform from destroying data resources. bool false
fleet_config Fleet configuration. object({…}) null
fleet_configmanagement_templates Sets of fleet configurations that can be applied to member clusters, in config name => {options} format. map(object({…})) {}
folder_ids Folder name => id mappings. map(string) {} 0-org-setup
host_project_ids Shared VPC host project name => id mappings. map(string) {} 2-networking
iam Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. map(list(string)) {}
iam_by_principals Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the iam variable. map(list(string)) {}
nodepools Nodepools configuration. Refer to the gke-nodepool module for type details. map(map(object({…}))) {}
stage_config FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management. object({…}) {…}
subnet_self_links Subnet VPC name => { name => self link } mappings. map(map(string)) {} 2-networking
vpc_config VPC-level configuration for project and clusters. object({…}) {…}
vpc_self_links Shared VPC name => self link mappings. map(string) {} 2-networking

Outputs

name description sensitive consumers
cluster_ids Cluster ids.
clusters Cluster resources. ✓
project_id GKE project id.