Remove GKE and GCVE stages (#3850)
This commit is contained in:
Julio Castillo
@@ -1,4 +0,0 @@
|
||||
FAST_STAGE_DESCRIPTION="GCVE (dev)"
|
||||
FAST_STAGE_LEVEL=3
|
||||
FAST_STAGE_NAME=gcve-dev
|
||||
FAST_STAGE_DEPS="0-globals 0-org-setup 2-networking"
|
||||
@@ -1,195 +0,0 @@
|
||||
# GCVE Private Cloud Minimal
|
||||
|
||||
This stage implements a simple architecture that integrates Google VMware Engine in a FAST organization.
|
||||
|
||||
The setup configured here is for a single environment in a single region, and is provided as a starting point for the more complex patterns [described below in this document](#architectural-patterns) which can be easily implemented by extending this stage, and/or duplicating it across environments. Some configuration examples are provided in the [GCVE module](../../../modules/gcve-private-cloud/).
|
||||
|
||||
<!-- BEGIN TOC -->
|
||||
- [Stage configuration](#stage-configuration)
|
||||
- [Project-level IAM](#project-level-iam)
|
||||
- [Networking](#networking)
|
||||
- [Architectural patterns](#architectural-patterns)
|
||||
- [Single-region shared GCVE deployment](#single-region-shared-gcve-deployment)
|
||||
- [Single-region per-environment GCVE deployment](#single-region-per-environment-gcve-deployment)
|
||||
- [Multi-regional deployments](#multi-regional-deployments)
|
||||
- [How to run this stage](#how-to-run-this-stage)
|
||||
- [FAST prerequisites](#fast-prerequisites)
|
||||
- [Provider and Terraform variables](#provider-and-terraform-variables)
|
||||
- [Impersonating the automation service account](#impersonating-the-automation-service-account)
|
||||
- [Variable configuration](#variable-configuration)
|
||||
- [Running the stage](#running-the-stage)
|
||||
- [Files](#files)
|
||||
- [Variables](#variables)
|
||||
- [Outputs](#outputs)
|
||||
<!-- END TOC -->
|
||||
|
||||
## Stage configuration
|
||||
|
||||
### Project-level IAM
|
||||
|
||||
Project-level IAM is controlled via the `iam` and `iam_by_principals` variables, which allow controlling authoritative bindings on the project.
|
||||
|
||||
To manage GCVE assign the `roles/vmwareengine.vmwareengineAdmin` and `roles/vmwareengine.vmwareengineViewer` roles to suitable groups via either of the above variables.
|
||||
|
||||
### Networking
|
||||
|
||||
Any of the FAST networking stages can be used to provide prerequisites for this stage. The development spoke VPC is used by default to attach the GCVE Private Cloud. To adapt this stage to production (or to a custom VPC) simply change the configuration of the GCVE module in the `main.tf` file.
|
||||
|
||||
Peerings can be configured to additional VPCs via the `network_peerings` variable, provided the service account running this stage has suitable permissions on the VPCs. When running FAST, network projects matching this stage's environment already have the suitable IAM binding via the custom `gcveNetworkAdmin` role defined in the bootstrap stage. For custom setups outside of FAST, the [VMware Engine Admin role](https://cloud.google.com/iam/docs/understanding-roles#vmwareengine-roles) can be used.
|
||||
|
||||
## Architectural patterns
|
||||
|
||||
The patterns shown here can be achieved by combining this stage with the relevant networking stage, and configuring network peerings to achieve the desired connectivity layout. Different patterns can of course be implemented by modifying the default configuration.
|
||||
|
||||
### Single-region shared GCVE deployment
|
||||
|
||||
This approach creates one GCVE deployment in a single region connected to every environment. When using a networking stage with a dedicated landing VPC as in the first two diagrams, an additional peering is created there to allow connections to the Private Cloud from on premises.
|
||||
|
||||
<p align="center">
|
||||
<img src="./diagrams/diagram-single-net-a.png" alt="Single region shared GCVE deployment with hub and spoke.">
|
||||
<br>
|
||||
With hub and spoke networking stage.
|
||||
</p>
|
||||
<p>
|
||||
<img src="./diagrams/diagram-single-net-c.png" alt="Single region shared GCVE deployment with separate network environments.">
|
||||
<br>
|
||||
With separate environments networking stage.
|
||||
</p>
|
||||
|
||||
### Single-region per-environment GCVE deployment
|
||||
|
||||
This approach creates one GCVE deployment per environment in a single region. As in the approach above, when using a networking stage with a dedicated landing VPC as in the first two diagrams, additional peerings are created there to allow connections to the Private Cloud from on premises.
|
||||
|
||||
<p align="center">
|
||||
<img src="./diagrams/diagram-multi-net-a.png" alt="Single region split GCVE deployment with hub and spoke.">
|
||||
<br>
|
||||
With hub and spoke networking stage.
|
||||
</p>
|
||||
<p align="center">
|
||||
<img src="./diagrams/diagram-multi-net-c.png" alt="Single region split GCVE deployment with separate network environments.">
|
||||
<br>
|
||||
With separate environments networking stage.
|
||||
</p>
|
||||
|
||||
### Multi-regional deployments
|
||||
|
||||
A design for a multi-regional deployment with the NVA FAST networking stage is shown below.
|
||||
|
||||
<p align="center">
|
||||
<img src="./diagrams/diagram-multi-net-b.png" alt="Multiregion shared GCVE deployment with NVA.">
|
||||
</p>
|
||||
|
||||
## How to run this stage
|
||||
|
||||
If this stage is deployed within a FAST-based GCP organization, we recommend executing it after foundational FAST `stage-2` components like `networking` and `security`. This is the recommended flow as specific data platform features in this stage might depend on configurations from these earlier stages. Although this stage can be run independently, instructions for such a standalone setup are beyond the scope of this document.
|
||||
|
||||
### FAST prerequisites
|
||||
|
||||
This stage needs specific automation resources, and permissions granted on those that allow control of selective IAM roles on specific networking and security resources.
|
||||
|
||||
Network permissions are needed to associate data domain or product projects to Shared VPC hosts and grant network permissions to data platform managed service accounts. They are mandatory when deploying Composer.
|
||||
|
||||
Security permissions are only needed when using CMEK encryption, to grant the relevant IAM roles to data platform service agents on the encryption keys used.
|
||||
|
||||
The ["Classic FAST" dataset](../0-org-setup/README.md#classic-fast-dataset) in the bootstrap stage contains the configuration for a development Data Platform that can be easily adapted to serve for this stage.
|
||||
|
||||
### Provider and Terraform variables
|
||||
|
||||
As all other FAST stages, the [mechanism used to pass variable values and pre-built provider files from one stage to the next](../0-org-setup/README.md#output-files-and-cross-stage-variables) is also leveraged here.
|
||||
|
||||
The commands to link or copy the provider and terraform variable files can be easily derived from the `fast-links.sh` script in the FAST stages folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following examples demonstrate both cases, and the resulting commands that then need to be copy/pasted and run.
|
||||
|
||||
```bash
|
||||
../fast-links.sh ~/fast-config
|
||||
|
||||
# File linking commands for GCVE (dev) stage
|
||||
|
||||
# provider file
|
||||
ln -s ~/fast-config/fast-test-00/providers/3-gcve-dev-providers.tf ./
|
||||
|
||||
# input files from other stages
|
||||
ln -s ~/fast-config/fast-test-00/tfvars/0-globals.auto.tfvars.json ./
|
||||
ln -s ~/fast-config/fast-test-00/tfvars/0-org-setup.auto.tfvars.json ./
|
||||
ln -s ~/fast-config/fast-test-00/tfvars/2-networking.auto.tfvars.json ./
|
||||
|
||||
# conventional place for stage tfvars (manually created)
|
||||
ln -s ~/fast-config/fast-test-00/3-gcve-dev.auto.tfvars ./
|
||||
```
|
||||
|
||||
```bash
|
||||
../fast-links.sh gs://xxx-prod-iac-core-outputs-0
|
||||
|
||||
# File linking commands for GCVE (dev) stage
|
||||
|
||||
# provider file
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/providers/3-gcve-dev-providers.tf ./
|
||||
|
||||
# input files from other stages
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-globals.auto.tfvars.json ./
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-org-setup.auto.tfvars.json ./
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/2-networking.auto.tfvars.json ./
|
||||
|
||||
# conventional place for stage tfvars (manually created)
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/3-gcve-dev.auto.tfvars ./
|
||||
```
|
||||
|
||||
### Impersonating the automation service account
|
||||
|
||||
The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The `gcp-devops` and `organization-admins` groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.
|
||||
|
||||
### Variable configuration
|
||||
|
||||
Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:
|
||||
|
||||
- variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the `0-globals.auto.tfvars.json` file linked or copied above
|
||||
- variables which refer to resources managed by previous stage, which are prepopulated here via the `*.auto.tfvars.json` files linked or copied above
|
||||
- and finally variables that optionally control this stage's behaviour and customizations, and can to be set in a custom `terraform.tfvars` file
|
||||
|
||||
The full list can be found in the [Variables](#variables) table at the bottom of this document.
|
||||
|
||||
### Running the stage
|
||||
|
||||
Once provider and variable values are in place and the correct user is configured, the stage can be run:
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform apply
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 exclude:3-gcve-dev-providers.tf -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Files
|
||||
|
||||
| name | description | modules | resources |
|
||||
|---|---|---|---|
|
||||
| [gcve-pc.tf](./gcve-pc.tf) | GCVE private cloud resources. | <code>gcve-private-cloud</code> | <code>google_vmwareengine_network_peering</code> |
|
||||
| [main.tf](./main.tf) | Locals and project-level resources. | <code>project</code> | |
|
||||
| [outputs.tf](./outputs.tf) | Output variables. | | <code>google_storage_bucket_object</code> |
|
||||
| [variables-fast.tf](./variables-fast.tf) | FAST stage interface. | | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [automation](variables-fast.tf#L19) | Automation resources created by the bootstrap stage. | <code>object({…})</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [billing_account](variables-fast.tf#L28) | Billing account id. If billing account is not part of the same org set `is_org_level` to false. | <code>object({…})</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [environments](variables-fast.tf#L36) | Long environment names. | <code>object({…})</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [prefix](variables-fast.tf#L53) | Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants. | <code>string</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [folder_ids](variables-fast.tf#L46) | Folders used by FAST stages in folders/nnnnnnnnnnn format. | <code>map(string)</code> | | <code>{}</code> | <code>0-org-setup</code> |
|
||||
| [iam](variables.tf#L17) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam_by_principals](variables.tf#L24) | Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [network_peerings](variables.tf#L31) | The network peerings between users' VPCs and the VMware Engine networks. Key is used for the peering name suffix. Network is expanded for FAST defined networks. | <code>map(object({…}))</code> | | <code>{…}</code> | |
|
||||
| [private_cloud_configs](variables.tf#L54) | The VMware private cloud configurations. Key is used for the private cloud name suffix. | <code>map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [stage_config](variables.tf#L76) | FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management. | <code>object({…})</code> | | <code>{…}</code> | |
|
||||
| [vpc_self_links](variables-fast.tf#L63) | FAST host VPC self links. | <code>map(string)</code> | | <code>{}</code> | <code>2-networking</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [network](outputs.tf#L17) | VMware engine network. | | |
|
||||
| [network_peerings](outputs.tf#L21) | The peerings created towards the user VPC or other VMware engine networks. | | |
|
||||
| [private_clouds](outputs.tf#L26) | VMware engine private cloud resources. | | |
|
||||
| [project_id](outputs.tf#L31) | GCVE project id. | | |
|
||||
<!-- END TFDOC -->
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 119 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 193 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 109 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 92 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 105 KiB |
@@ -1,15 +0,0 @@
|
||||
# Copyright 2025 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# FAST release: v54.3.0
|
||||
@@ -1,58 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GCVE private cloud resources.
|
||||
|
||||
locals {
|
||||
network_peerings = {
|
||||
for k, v in var.network_peerings : k => merge(v, {
|
||||
# interpolate FAST VPC ids if available
|
||||
peer_network = lookup(var.vpc_self_links, v.peer_network, v.peer_network)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
module "gcve-pc" {
|
||||
source = "../../../modules/gcve-private-cloud"
|
||||
prefix = var.prefix
|
||||
project_id = module.gcve-project-0.id
|
||||
vmw_network_config = {
|
||||
create = true
|
||||
name = "default"
|
||||
}
|
||||
vmw_network_peerings = local.network_peerings
|
||||
vmw_private_cloud_configs = var.private_cloud_configs
|
||||
}
|
||||
|
||||
# optional reverse peering configuration from the peer network projects
|
||||
|
||||
resource "google_vmwareengine_network_peering" "vmw_engine_network_peerings" {
|
||||
for_each = {
|
||||
for k, v in local.network_peerings : k => v if v.configure_peer_network
|
||||
}
|
||||
project = regex(
|
||||
"projects/([^/]+)/", each.value.peer_network
|
||||
)[0]
|
||||
name = "${var.prefix}-${each.key}"
|
||||
description = each.value.description
|
||||
peer_network = each.value.peer_network
|
||||
peer_network_type = "STANDARD"
|
||||
vmware_engine_network = module.gcve-pc.network_id
|
||||
export_custom_routes = each.value.routes_config.import
|
||||
export_custom_routes_with_public_ip = each.value.routes_config.public_import
|
||||
import_custom_routes = each.value.routes_config.export
|
||||
import_custom_routes_with_public_ip = each.value.routes_config.public_export
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Locals and project-level resources.
|
||||
|
||||
locals {
|
||||
folder_id = var.folder_ids[var.stage_config.name]
|
||||
}
|
||||
|
||||
module "gcve-project-0" {
|
||||
source = "../../../modules/project"
|
||||
billing_account = var.billing_account.id
|
||||
name = "dev-gcve-core-0"
|
||||
parent = local.folder_id
|
||||
prefix = var.prefix
|
||||
iam = var.iam
|
||||
iam_by_principals = var.iam_by_principals
|
||||
labels = {
|
||||
environment = lower(
|
||||
var.environments[var.stage_config.environment].name
|
||||
)
|
||||
}
|
||||
services = [
|
||||
"compute.googleapis.com",
|
||||
"logging.googleapis.com",
|
||||
"monitoring.googleapis.com",
|
||||
"vmwareengine.googleapis.com"
|
||||
]
|
||||
}
|
||||
@@ -1,43 +0,0 @@
|
||||
# Copyright 2024 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# tfdoc:file:description Output variables.
|
||||
|
||||
output "network" {
|
||||
description = "VMware engine network."
|
||||
value = module.gcve-pc.network_id
|
||||
}
|
||||
output "network_peerings" {
|
||||
description = "The peerings created towards the user VPC or other VMware engine networks."
|
||||
value = module.gcve-pc.network_peerings
|
||||
}
|
||||
|
||||
output "private_clouds" {
|
||||
description = "VMware engine private cloud resources."
|
||||
value = module.gcve-pc.private_clouds
|
||||
}
|
||||
|
||||
output "project_id" {
|
||||
description = "GCVE project id."
|
||||
value = module.gcve-project-0.project_id
|
||||
depends_on = [module.gcve-pc]
|
||||
}
|
||||
|
||||
resource "google_storage_bucket_object" "version" {
|
||||
count = fileexists("fast_version.txt") ? 1 : 0
|
||||
bucket = var.automation.outputs_bucket
|
||||
name = "versions/3-${var.stage_config.name}-version.txt"
|
||||
source = "fast_version.txt"
|
||||
source_md5hash = filemd5("fast_version.txt")
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description FAST stage interface.
|
||||
|
||||
variable "automation" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Automation resources created by the bootstrap stage."
|
||||
type = object({
|
||||
outputs_bucket = string
|
||||
})
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "billing_account" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Billing account id. If billing account is not part of the same org set `is_org_level` to false."
|
||||
type = object({
|
||||
id = string
|
||||
})
|
||||
}
|
||||
|
||||
variable "environments" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Long environment names."
|
||||
type = object({
|
||||
dev = object({
|
||||
name = string
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
variable "folder_ids" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Folders used by FAST stages in folders/nnnnnnnnnnn format."
|
||||
type = map(string)
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants."
|
||||
type = string
|
||||
validation {
|
||||
condition = try(length(var.prefix), 0) < 12
|
||||
error_message = "Use a maximum of 9 chars for organizations, and 11 chars for tenants."
|
||||
}
|
||||
}
|
||||
|
||||
variable "vpc_self_links" {
|
||||
# tfdoc:variable:source 2-networking
|
||||
description = "FAST host VPC self links."
|
||||
type = map(string)
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
@@ -1,86 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam_by_principals" {
|
||||
description = "Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "network_peerings" {
|
||||
description = "The network peerings between users' VPCs and the VMware Engine networks. Key is used for the peering name suffix. Network is expanded for FAST defined networks."
|
||||
type = map(object({
|
||||
peer_network = string
|
||||
configure_peer_network = optional(bool, false)
|
||||
description = optional(string, "Managed by Terraform.")
|
||||
peer_to_vmware_engine_network = optional(bool, false)
|
||||
routes_config = optional(object({
|
||||
export = optional(bool, false)
|
||||
import = optional(bool, false)
|
||||
public_export = optional(bool, false)
|
||||
public_import = optional(bool, false)
|
||||
}), {})
|
||||
}))
|
||||
nullable = false
|
||||
default = {
|
||||
dev-spoke-0 = {
|
||||
peer_network = "dev"
|
||||
configure_peer_network = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
variable "private_cloud_configs" {
|
||||
description = "The VMware private cloud configurations. Key is used for the private cloud name suffix."
|
||||
type = map(object({
|
||||
cidr = string
|
||||
zone = string
|
||||
additional_cluster_configs = optional(map(object({
|
||||
custom_core_count = optional(number)
|
||||
node_count = optional(number, 3)
|
||||
node_type_id = optional(string, "standard-72")
|
||||
})), {})
|
||||
management_cluster_config = optional(object({
|
||||
custom_core_count = optional(number)
|
||||
name = optional(string, "mgmt-cluster")
|
||||
node_count = optional(number, 3)
|
||||
node_type_id = optional(string, "standard-72")
|
||||
}), {})
|
||||
description = optional(string, "Managed by Terraform.")
|
||||
}))
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "stage_config" {
|
||||
description = "FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management."
|
||||
type = object({
|
||||
environment = string
|
||||
name = string
|
||||
})
|
||||
default = {
|
||||
environment = "dev"
|
||||
name = "gcve/dev"
|
||||
}
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
FAST_STAGE_DESCRIPTION="GKE (dev)"
|
||||
FAST_STAGE_LEVEL=3
|
||||
FAST_STAGE_NAME=gke-dev
|
||||
FAST_STAGE_DEPS="0-globals 0-org-setup 2-networking"
|
||||
@@ -1,209 +0,0 @@
|
||||
# GKE Multitenant
|
||||
|
||||
This stage allows creation and management of a fleet of GKE multitenant clusters for a single environment, optionally leveraging GKE Hub to configure additional features.
|
||||
|
||||
The following diagram illustrates the high-level design of created resources, which can be adapted to specific requirements via variables:
|
||||
|
||||
<p align="center">
|
||||
<img src="diagram.png" alt="GKE multitenant">
|
||||
</p>
|
||||
|
||||
<!-- BEGIN TOC -->
|
||||
- [Design overview and choices](#design-overview-and-choices)
|
||||
- [How to run this stage](#how-to-run-this-stage)
|
||||
- [FAST prerequisites](#fast-prerequisites)
|
||||
- [Customizations](#customizations)
|
||||
- [Clusters and node pools](#clusters-and-node-pools)
|
||||
- [Fleet management](#fleet-management)
|
||||
- [Files](#files)
|
||||
- [Variables](#variables)
|
||||
- [Outputs](#outputs)
|
||||
<!-- END TOC -->
|
||||
|
||||
## Design overview and choices
|
||||
|
||||
The general idea behind this stage is to deploy a single project hosting multiple clusters leveraging several useful GKE features like Config Sync, which lend themselves well to a multitenant approach to GKE.
|
||||
|
||||
Some high level choices applied here:
|
||||
|
||||
- all clusters are created as [private clusters](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters) which then need to be [VPC-native](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips).
|
||||
- Logging and monitoring uses Cloud Operations for system components and user workloads.
|
||||
- [GKE metering](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering) is enabled by default and stored in a BigQuery dataset created within the project.
|
||||
- [GKE Fleet](https://cloud.google.com/kubernetes-engine/docs/fleets-overview) can be optionally with support for the following features:
|
||||
- [Fleet workload identity](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs/use-workload-identity)
|
||||
- [Config Management](https://cloud.google.com/anthos-config-management/docs/overview)
|
||||
- [Service Mesh](https://cloud.google.com/service-mesh/docs/overview)
|
||||
- [Identity Service](https://cloud.google.com/anthos/identity/setup/fleet)
|
||||
- [Multi-cluster services](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-services)
|
||||
- [Multi-cluster ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress).
|
||||
- Support for [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview) and [Hierarchy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/hierarchy-controller) when using Config Management.
|
||||
- [Groups for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac) can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
|
||||
- Support for [application layer secret encryption](https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets).
|
||||
- Some features are enabled by default in all clusters:
|
||||
- [Intranode visibility](https://cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility)
|
||||
- [Dataplane v2](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2)
|
||||
- [Shielded GKE nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes)
|
||||
- [Workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
|
||||
- [Node local DNS cache](https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache)
|
||||
- [Use of the GCE persistent disk CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
|
||||
- Node [auto-upgrade](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) and [auto-repair](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair) for all node pools
|
||||
|
||||
## How to run this stage
|
||||
|
||||
If this stage is deployed within a FAST-based GCP organization, we recommend executing it after foundational FAST `stage-2` components like `networking` and `security`. This is the recommended flow as specific data platform features in this stage might depend on configurations from these earlier stages. Although this stage can be run independently, instructions for such a standalone setup are beyond the scope of this document.
|
||||
|
||||
### FAST prerequisites
|
||||
|
||||
This stage needs specific automation resources, and permissions granted on those that allow control of selective IAM roles on specific networking and security resources.
|
||||
|
||||
Network permissions are needed to associate data domain or product projects to Shared VPC hosts and grant network permissions to data platform managed service accounts. They are mandatory when deploying Composer.
|
||||
|
||||
Security permissions are only needed when using CMEK encryption, to grant the relevant IAM roles to data platform service agents on the encryption keys used.
|
||||
|
||||
The ["Classic FAST" dataset](../0-org-setup/README.md#classic-fast-dataset) in the bootstrap stage contains the configuration for a development Data Platform that can be easily adapted to serve for this stage.
|
||||
|
||||
## Customizations
|
||||
|
||||
This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Variables allow management of clusters, nodepools, and fleet registration and configurations.
|
||||
|
||||
### Clusters and node pools
|
||||
|
||||
This is an example of declaring a private cluster with one nodepool via `tfvars` file:
|
||||
|
||||
```hcl
|
||||
clusters = {
|
||||
test-00 = {
|
||||
description = "Cluster test 0"
|
||||
location = "europe-west8"
|
||||
private_cluster_config = {
|
||||
enable_private_endpoint = true
|
||||
master_global_access = true
|
||||
}
|
||||
vpc_config = {
|
||||
subnetwork = "projects/ldj-dev-net-spoke-0/regions/europe-west8/subnetworks/gke"
|
||||
master_ipv4_cidr_block = "172.16.20.0/28"
|
||||
master_authorized_ranges = {
|
||||
private = "10.0.0.0/8"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
test-00 = {
|
||||
00 = {
|
||||
node_count = { initial = 1 }
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest skip
|
||||
```
|
||||
|
||||
And here another example of declaring a private cluster following hardening guidelines with one nodepool via `tfvars` file:
|
||||
```hcl
|
||||
clusters = {
|
||||
test-00 = {
|
||||
description = "Hardened Cluster test 0"
|
||||
location = "europe-west1"
|
||||
private_cluster_config = {
|
||||
enable_private_endpoint = true
|
||||
master_global_access = true
|
||||
}
|
||||
access_config = {
|
||||
dns_access = true
|
||||
ip_access = {
|
||||
disable_public_endpoint = true
|
||||
}
|
||||
private_nodes = true
|
||||
}
|
||||
enable_features = {
|
||||
binary_authorization = true
|
||||
groups_for_rbac = "gke-security-groups@example.com"
|
||||
intranode_visibility = true
|
||||
rbac_binding_config = {
|
||||
enable_insecure_binding_system_unauthenticated : false
|
||||
enable_insecure_binding_system_authenticated : false
|
||||
}
|
||||
shielded_nodes = true
|
||||
upgrade_notifications = {
|
||||
event_types = ["SECURITY_BULLETIN_EVENT", "UPGRADE_AVAILABLE_EVENT", "UPGRADE_INFO_EVENT", "UPGRADE_EVENT"]
|
||||
}
|
||||
workload_identity = true
|
||||
}
|
||||
vpc_config = {
|
||||
subnetwork = "projects/ldj-dev-net-spoke-0/regions/europe-west8/subnetworks/gke"
|
||||
master_ipv4_cidr_block = "172.16.20.0/28"
|
||||
master_authorized_ranges = {
|
||||
private = "10.0.0.0/8"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
nodepools = {
|
||||
test-00 = {
|
||||
00 = {
|
||||
node_count = { initial = 1 }
|
||||
node_config = {
|
||||
sandbox_config_gvisor = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest skip
|
||||
```
|
||||
|
||||
|
||||
If clusters share similar configurations, those can be centralized via `locals` blocks in this stage's `main.tf` file, and merged in with clusters via a simple `for_each` loop.
|
||||
|
||||
### Fleet management
|
||||
|
||||
Fleet management is entirely optional, and uses two separate variables:
|
||||
|
||||
- `fleet_config`: specifies the [GKE fleet](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs/fleet-concepts#fleet-enabled-components) features to activate
|
||||
- `fleet_configmanagement_templates`: defines configuration templates for specific sets of features ([Config Management](https://cloud.google.com/anthos-config-management/docs/how-to/install-anthos-config-management) currently)
|
||||
|
||||
Clusters can then be configured for fleet registration and one of the config management templates attached via the cluster-level `fleet_config` attribute.
|
||||
|
||||
<!-- TFDOC OPTS files:1 show_extra:1 exclude:3-gke-dev-providers.tf -->
|
||||
<!-- BEGIN TFDOC -->
|
||||
## Files
|
||||
|
||||
| name | description | modules | resources |
|
||||
|---|---|---|---|
|
||||
| [gke-clusters.tf](./gke-clusters.tf) | GKE clusters. | <code>gke-cluster-standard</code> · <code>gke-nodepool</code> | |
|
||||
| [gke-hub.tf](./gke-hub.tf) | GKE hub configuration. | <code>gke-hub</code> | |
|
||||
| [main.tf](./main.tf) | Project and usage dataset. | <code>bigquery-dataset</code> · <code>iam-service-account</code> · <code>project</code> | |
|
||||
| [outputs.tf](./outputs.tf) | Module outputs. | | <code>google_storage_bucket_object</code> |
|
||||
| [variables-fast.tf](./variables-fast.tf) | None | | |
|
||||
| [variables-fleet.tf](./variables-fleet.tf) | GKE fleet configurations. | | |
|
||||
| [variables.tf](./variables.tf) | Module variables. | | |
|
||||
|
||||
## Variables
|
||||
|
||||
| name | description | type | required | default | producer |
|
||||
|---|---|:---:|:---:|:---:|:---:|
|
||||
| [automation](variables-fast.tf#L17) | Automation resources created by the bootstrap stage. | <code>object({…})</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [billing_account](variables-fast.tf#L26) | Billing account id. If billing account is not part of the same org set `is_org_level` to false. | <code>object({…})</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [environments](variables-fast.tf#L34) | Long environment names. | <code>object({…})</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [prefix](variables-fast.tf#L60) | Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants. | <code>string</code> | ✓ | | <code>0-org-setup</code> |
|
||||
| [clusters](variables.tf#L17) | Clusters configuration. Refer to the gke-cluster module for type details. | <code>map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [deletion_protection](variables.tf#L106) | Prevent Terraform from destroying data resources. | <code>bool</code> | | <code>false</code> | |
|
||||
| [fleet_config](variables-fleet.tf#L19) | Fleet configuration. | <code>object({…})</code> | | <code>null</code> | |
|
||||
| [fleet_configmanagement_templates](variables-fleet.tf#L35) | Sets of fleet configurations that can be applied to member clusters, in config name => {options} format. | <code>map(object({…}))</code> | | <code>{}</code> | |
|
||||
| [folder_ids](variables-fast.tf#L44) | Folder name => id mappings. | <code>map(string)</code> | | <code>{}</code> | <code>0-org-setup</code> |
|
||||
| [host_project_ids](variables-fast.tf#L52) | Shared VPC host project name => id mappings. | <code>map(string)</code> | | <code>{}</code> | <code>2-networking</code> |
|
||||
| [iam](variables.tf#L113) | Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [iam_by_principals](variables.tf#L120) | Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable. | <code>map(list(string))</code> | | <code>{}</code> | |
|
||||
| [nodepools](variables.tf#L127) | Nodepools configuration. Refer to the gke-nodepool module for type details. | <code>map(map(object({…})))</code> | | <code>{}</code> | |
|
||||
| [stage_config](variables.tf#L176) | FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management. | <code>object({…})</code> | | <code>{…}</code> | |
|
||||
| [subnet_self_links](variables-fast.tf#L70) | Subnet VPC name => { name => self link } mappings. | <code>map(map(string))</code> | | <code>{}</code> | <code>2-networking</code> |
|
||||
| [vpc_config](variables.tf#L188) | VPC-level configuration for project and clusters. | <code>object({…})</code> | | <code>{…}</code> | |
|
||||
| [vpc_self_links](variables-fast.tf#L78) | Shared VPC name => self link mappings. | <code>map(string)</code> | | <code>{}</code> | <code>2-networking</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
| name | description | sensitive | consumers |
|
||||
|---|---|:---:|---|
|
||||
| [cluster_ids](outputs.tf#L15) | Cluster ids. | | |
|
||||
| [clusters](outputs.tf#L22) | Cluster resources. | ✓ | |
|
||||
| [project_id](outputs.tf#L28) | GKE project id. | | |
|
||||
<!-- END TFDOC -->
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 43 KiB |
@@ -1,15 +0,0 @@
|
||||
# Copyright 2025 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# FAST release: v54.3.0
|
||||
@@ -1,105 +0,0 @@
|
||||
/**
|
||||
* Copyright 2023 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE clusters.
|
||||
|
||||
locals {
|
||||
nodepools = merge([
|
||||
for cluster, nodepools in var.nodepools : {
|
||||
for nodepool, config in nodepools :
|
||||
"${cluster}/${nodepool}" => merge(config, {
|
||||
name = nodepool
|
||||
cluster = cluster
|
||||
})
|
||||
}
|
||||
]...)
|
||||
subnet_self_links = try(
|
||||
var.subnet_self_links[var.vpc_config.vpc_self_link], {}
|
||||
)
|
||||
vpc_self_link = lookup(
|
||||
var.vpc_self_links,
|
||||
var.vpc_config.vpc_self_link,
|
||||
var.vpc_config.vpc_self_link
|
||||
)
|
||||
}
|
||||
|
||||
module "gke-cluster" {
|
||||
source = "../../../modules/gke-cluster-standard"
|
||||
for_each = var.clusters
|
||||
name = each.key
|
||||
project_id = module.gke-project-0.project_id
|
||||
access_config = each.value.access_config
|
||||
cluster_autoscaling = each.value.cluster_autoscaling
|
||||
description = each.value.description
|
||||
enable_features = each.value.enable_features
|
||||
enable_addons = each.value.enable_addons
|
||||
issue_client_certificate = each.value.issue_client_certificate
|
||||
labels = each.value.labels
|
||||
location = each.value.location
|
||||
logging_config = each.value.logging_config
|
||||
maintenance_config = each.value.maintenance_config
|
||||
max_pods_per_node = each.value.max_pods_per_node
|
||||
min_master_version = each.value.min_master_version
|
||||
monitoring_config = each.value.monitoring_config
|
||||
node_locations = each.value.node_locations
|
||||
release_channel = each.value.release_channel
|
||||
vpc_config = merge(each.value.vpc_config, {
|
||||
network = try(
|
||||
var.vpc_self_links[each.value.vpc_config.network],
|
||||
each.value.vpc_config.network,
|
||||
local.vpc_self_link
|
||||
)
|
||||
subnetwork = try(
|
||||
local.subnet_self_links[each.value.vpc_config.subnetwork],
|
||||
each.value.vpc_config.subnetwork,
|
||||
null
|
||||
)
|
||||
})
|
||||
deletion_protection = var.deletion_protection
|
||||
node_config = merge(coalesce(each.value.node_config, {}), {
|
||||
service_account = (
|
||||
each.value.service_account == null
|
||||
? module.gke-nodes-service-account.email
|
||||
: each.value.service_account
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
module "gke-nodepool" {
|
||||
source = "../../../modules/gke-nodepool"
|
||||
for_each = local.nodepools
|
||||
name = each.value.name
|
||||
project_id = module.gke-project-0.project_id
|
||||
cluster_name = module.gke-cluster[each.value.cluster].name
|
||||
location = module.gke-cluster[each.value.cluster].location
|
||||
gke_version = each.value.gke_version
|
||||
k8s_labels = each.value.k8s_labels
|
||||
max_pods_per_node = each.value.max_pods_per_node
|
||||
node_config = each.value.node_config
|
||||
node_count = each.value.node_count
|
||||
node_locations = each.value.node_locations
|
||||
nodepool_config = each.value.nodepool_config
|
||||
network_config = each.value.network_config
|
||||
reservation_affinity = each.value.reservation_affinity
|
||||
service_account = (
|
||||
each.value.service_account == null
|
||||
? { email = module.gke-nodes-service-account.email }
|
||||
: each.value.service_account
|
||||
)
|
||||
sole_tenant_nodegroup = each.value.sole_tenant_nodegroup
|
||||
tags = each.value.tags
|
||||
taints = each.value.taints
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
/**
|
||||
* Copyright 2022 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE hub configuration.
|
||||
|
||||
locals {
|
||||
fleet_clusters = var.fleet_config == null ? {} : {
|
||||
for k, v in var.clusters : k => v.fleet_config
|
||||
if v.fleet_config.register == true
|
||||
}
|
||||
fleet_mcs_enabled = (
|
||||
try(
|
||||
var.fleet_config.enable_features.multiclusterservicediscovery, false
|
||||
) == true
|
||||
)
|
||||
}
|
||||
|
||||
module "gke-hub" {
|
||||
source = "../../../modules/gke-hub"
|
||||
count = var.fleet_config != null ? 1 : 0
|
||||
project_id = module.gke-project-0.project_id
|
||||
clusters = {
|
||||
for k, v in local.fleet_clusters : k => {
|
||||
id = module.gke-cluster[k].id
|
||||
configmanagement = v.configmanagement_template
|
||||
policycontroller = null # Can be extended if needed
|
||||
servicemesh = null # Can be extended if needed
|
||||
workload_identity = var.fleet_config.use_workload_identity
|
||||
}
|
||||
}
|
||||
features = var.fleet_config.enable_features
|
||||
configmanagement_templates = var.fleet_configmanagement_templates
|
||||
depends_on = [
|
||||
module.gke-nodepool
|
||||
]
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description Project and usage dataset.
|
||||
|
||||
locals {
|
||||
folder_id = var.folder_ids[var.stage_config.name]
|
||||
gke_nodes_sa_roles = [
|
||||
"autoscaling.metricsWriter",
|
||||
"logging.logWriter",
|
||||
"monitoring.viewer",
|
||||
"monitoring.metricWriter",
|
||||
"stackdriver.resourceMetadata.writer"
|
||||
]
|
||||
project_name = "${var.stage_config.environment}-gke-core-0"
|
||||
_cmek_keys_container = toset(compact(flatten([
|
||||
[for k, v in var.clusters : try(v.node_config.boot_disk_kms_key, null)],
|
||||
[
|
||||
for k, v in var.nodepools : [
|
||||
for nk, nv in v : try(nv.node_config.boot_disk_kms_key, null)
|
||||
]
|
||||
]
|
||||
])))
|
||||
_cmek_keys_pubsub = toset(compact(flatten([
|
||||
[for k, v in var.clusters : try(v.enable_features.upgrade_notifications.kms_key_name, null)],
|
||||
])))
|
||||
service_encryption_key_ids = {
|
||||
for k, v in {
|
||||
"container.googleapis.com" = local._cmek_keys_container
|
||||
"pubsub.googleapis.com" = local._cmek_keys_pubsub
|
||||
} : k => v if length(v) > 0
|
||||
}
|
||||
}
|
||||
|
||||
module "gke-project-0" {
|
||||
source = "../../../modules/project"
|
||||
billing_account = var.billing_account.id
|
||||
name = local.project_name
|
||||
parent = local.folder_id
|
||||
prefix = var.prefix
|
||||
iam = merge(var.iam, {
|
||||
"roles/gkehub.serviceAgent" = [
|
||||
module.gke-project-0.service_agents.fleet.iam_email
|
||||
] }
|
||||
)
|
||||
iam_by_principals = var.iam_by_principals
|
||||
iam_bindings_additive = {
|
||||
for r in local.gke_nodes_sa_roles : "gke-nodes-sa-${r}" => {
|
||||
member = module.gke-nodes-service-account.iam_email
|
||||
role = "roles/${r}"
|
||||
}
|
||||
}
|
||||
labels = {
|
||||
environment = lower(var.environments[var.stage_config.environment].name)
|
||||
}
|
||||
org_policies = {
|
||||
// GKE cluster require serial port logging for low level troubleshooting
|
||||
"compute.managed.disableSerialPortLogging" = {
|
||||
rules = [{ enforce = false }]
|
||||
}
|
||||
}
|
||||
services = [
|
||||
"anthos.googleapis.com",
|
||||
"anthosconfigmanagement.googleapis.com",
|
||||
"cloudresourcemanager.googleapis.com",
|
||||
"container.googleapis.com",
|
||||
"compute.googleapis.com",
|
||||
"dns.googleapis.com",
|
||||
"gkeconnect.googleapis.com",
|
||||
"gkehub.googleapis.com",
|
||||
"iam.googleapis.com",
|
||||
"logging.googleapis.com",
|
||||
"monitoring.googleapis.com",
|
||||
"pubsub.googleapis.com",
|
||||
"multiclusteringress.googleapis.com",
|
||||
"multiclusterservicediscovery.googleapis.com",
|
||||
"orgpolicy.googleapis.com",
|
||||
"trafficdirector.googleapis.com"
|
||||
]
|
||||
service_encryption_key_ids = local.service_encryption_key_ids
|
||||
shared_vpc_service_config = {
|
||||
attach = true
|
||||
host_project = lookup(
|
||||
var.host_project_ids,
|
||||
var.vpc_config.host_project_id,
|
||||
var.vpc_config.host_project_id
|
||||
)
|
||||
service_agent_iam = merge({
|
||||
"roles/compute.networkUser" = [
|
||||
"cloudservices", "container-engine"
|
||||
]
|
||||
"roles/container.hostServiceAgentUser" = [
|
||||
"container-engine"
|
||||
]
|
||||
},
|
||||
!local.fleet_mcs_enabled ? {} : {
|
||||
"roles/multiclusterservicediscovery.serviceAgent" = ["mcsd"]
|
||||
"roles/compute.networkViewer" = [
|
||||
"serviceAccount:${var.prefix}-${local.project_name}.svc.id.goog[gke-mcs/gke-mcs-importer]"
|
||||
]
|
||||
})
|
||||
}
|
||||
# specify project-level org policies here if you need them
|
||||
}
|
||||
|
||||
module "gke-dataset-resource-usage" {
|
||||
source = "../../../modules/bigquery-dataset"
|
||||
project_id = module.gke-project-0.project_id
|
||||
id = "gke_resource_usage"
|
||||
friendly_name = "GKE resource usage."
|
||||
}
|
||||
|
||||
module "gke-nodes-service-account" {
|
||||
source = "../../../modules/iam-service-account"
|
||||
project_id = module.gke-project-0.project_id
|
||||
name = "gke-node-default"
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
# Copyright 2023 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
output "cluster_ids" {
|
||||
description = "Cluster ids."
|
||||
value = {
|
||||
for k, v in module.gke-cluster : k => v.id
|
||||
}
|
||||
}
|
||||
|
||||
output "clusters" {
|
||||
description = "Cluster resources."
|
||||
sensitive = true
|
||||
value = module.gke-cluster
|
||||
}
|
||||
|
||||
output "project_id" {
|
||||
description = "GKE project id."
|
||||
value = module.gke-project-0.project_id
|
||||
}
|
||||
|
||||
resource "google_storage_bucket_object" "version" {
|
||||
count = fileexists("fast_version.txt") ? 1 : 0
|
||||
bucket = var.automation.outputs_bucket
|
||||
name = "versions/3-${var.stage_config.name}-version.txt"
|
||||
source = "fast_version.txt"
|
||||
source_md5hash = filemd5("fast_version.txt")
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "automation" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Automation resources created by the bootstrap stage."
|
||||
type = object({
|
||||
outputs_bucket = string
|
||||
})
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "billing_account" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Billing account id. If billing account is not part of the same org set `is_org_level` to false."
|
||||
type = object({
|
||||
id = string
|
||||
})
|
||||
}
|
||||
|
||||
variable "environments" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Long environment names."
|
||||
type = object({
|
||||
dev = object({
|
||||
name = string
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
variable "folder_ids" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Folder name => id mappings."
|
||||
type = map(string)
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "host_project_ids" {
|
||||
# tfdoc:variable:source 2-networking
|
||||
description = "Shared VPC host project name => id mappings."
|
||||
type = map(string)
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "prefix" {
|
||||
# tfdoc:variable:source 0-org-setup
|
||||
description = "Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants."
|
||||
type = string
|
||||
validation {
|
||||
condition = try(length(var.prefix), 0) < 12
|
||||
error_message = "Use a maximum of 9 chars for organizations, and 11 chars for tenants."
|
||||
}
|
||||
}
|
||||
|
||||
variable "subnet_self_links" {
|
||||
# tfdoc:variable:source 2-networking
|
||||
description = "Subnet VPC name => { name => self link } mappings."
|
||||
type = map(map(string))
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
|
||||
variable "vpc_self_links" {
|
||||
# tfdoc:variable:source 2-networking
|
||||
description = "Shared VPC name => self link mappings."
|
||||
type = map(string)
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
@@ -1,68 +0,0 @@
|
||||
/**
|
||||
* Copyright 2024 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
# tfdoc:file:description GKE fleet configurations.
|
||||
|
||||
variable "fleet_config" {
|
||||
description = "Fleet configuration."
|
||||
type = object({
|
||||
enable_features = optional(object({
|
||||
appdevexperience = optional(bool, false)
|
||||
configmanagement = optional(bool, false)
|
||||
identityservice = optional(bool, false)
|
||||
multiclusteringress = optional(string, null)
|
||||
multiclusterservicediscovery = optional(bool, false)
|
||||
servicemesh = optional(bool, false)
|
||||
}), {})
|
||||
use_workload_identity = optional(bool, false)
|
||||
})
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "fleet_configmanagement_templates" {
|
||||
description = "Sets of fleet configurations that can be applied to member clusters, in config name => {options} format."
|
||||
type = map(object({
|
||||
binauthz = optional(bool)
|
||||
version = optional(string)
|
||||
config_sync = object({
|
||||
git = optional(object({
|
||||
sync_repo = string
|
||||
policy_dir = string
|
||||
gcp_service_account_email = optional(string)
|
||||
https_proxy = optional(string)
|
||||
secret_type = optional(string, "none")
|
||||
sync_branch = optional(string)
|
||||
sync_rev = optional(string)
|
||||
sync_wait_secs = optional(number)
|
||||
}))
|
||||
prevent_drift = optional(bool)
|
||||
source_format = optional(string, "hierarchy")
|
||||
})
|
||||
hierarchy_controller = optional(object({
|
||||
enable_hierarchical_resource_quota = optional(bool)
|
||||
enable_pod_tree_labels = optional(bool)
|
||||
}))
|
||||
policy_controller = object({
|
||||
audit_interval_seconds = optional(number)
|
||||
exemptable_namespaces = optional(list(string))
|
||||
log_denies_enabled = optional(bool)
|
||||
referential_rules_enabled = optional(bool)
|
||||
template_library_installed = optional(bool)
|
||||
})
|
||||
}))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
@@ -1,199 +0,0 @@
|
||||
/**
|
||||
* Copyright 2025 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
variable "clusters" {
|
||||
description = "Clusters configuration. Refer to the gke-cluster module for type details."
|
||||
type = map(object({
|
||||
access_config = optional(object({
|
||||
dns_access = optional(object({
|
||||
allow_external_traffic = optional(bool, true)
|
||||
enable_k8s_tokens = optional(bool)
|
||||
enable_k8s_certs = optional(bool)
|
||||
}), {})
|
||||
ip_access = optional(object({
|
||||
authorized_ranges = optional(map(string), {})
|
||||
disable_public_endpoint = optional(bool, true)
|
||||
gcp_public_cidrs_access_enabled = optional(bool, false)
|
||||
private_endpoint_config = optional(object({
|
||||
endpoint_subnetwork = optional(string)
|
||||
global_access = optional(bool, true)
|
||||
}), {})
|
||||
}))
|
||||
private_nodes = optional(bool, true)
|
||||
}), {})
|
||||
cluster_autoscaling = optional(any)
|
||||
description = optional(string)
|
||||
enable_addons = optional(any, {
|
||||
horizontal_pod_autoscaling = true, http_load_balancing = true
|
||||
})
|
||||
enable_features = optional(any, {
|
||||
shielded_nodes = true
|
||||
workload_identity = true
|
||||
})
|
||||
fleet_config = optional(object({
|
||||
register = optional(bool, true)
|
||||
configmanagement_template = optional(string)
|
||||
}), {})
|
||||
issue_client_certificate = optional(bool, false)
|
||||
labels = optional(map(string))
|
||||
location = string
|
||||
logging_config = optional(object({
|
||||
enable_system_logs = optional(bool, true)
|
||||
enable_workloads_logs = optional(bool, true)
|
||||
enable_api_server_logs = optional(bool, false)
|
||||
enable_scheduler_logs = optional(bool, false)
|
||||
enable_controller_manager_logs = optional(bool, false)
|
||||
}), {})
|
||||
maintenance_config = optional(any, {
|
||||
daily_window_start_time = "03:00"
|
||||
recurring_window = null
|
||||
maintenance_exclusion = []
|
||||
})
|
||||
max_pods_per_node = optional(number, 110)
|
||||
min_master_version = optional(string)
|
||||
monitoring_config = optional(object({
|
||||
enable_system_metrics = optional(bool, true)
|
||||
# (Optional) control plane metrics
|
||||
enable_api_server_metrics = optional(bool, false)
|
||||
enable_controller_manager_metrics = optional(bool, false)
|
||||
enable_scheduler_metrics = optional(bool, false)
|
||||
# (Optional) kube state metrics
|
||||
enable_daemonset_metrics = optional(bool, false)
|
||||
enable_deployment_metrics = optional(bool, false)
|
||||
enable_hpa_metrics = optional(bool, false)
|
||||
enable_pod_metrics = optional(bool, false)
|
||||
enable_statefulset_metrics = optional(bool, false)
|
||||
enable_storage_metrics = optional(bool, false)
|
||||
# Google Cloud Managed Service for Prometheus
|
||||
enable_managed_prometheus = optional(bool, true)
|
||||
}), {})
|
||||
node_locations = optional(list(string))
|
||||
release_channel = optional(string)
|
||||
service_account = optional(string)
|
||||
vpc_config = object({
|
||||
subnetwork = string
|
||||
network = optional(string)
|
||||
secondary_range_blocks = optional(object({
|
||||
pods = string
|
||||
services = string
|
||||
}))
|
||||
secondary_range_names = optional(object({
|
||||
pods = string
|
||||
services = string
|
||||
}), { pods = "pods", services = "services" })
|
||||
})
|
||||
node_config = optional(object({
|
||||
boot_disk_kms_key = optional(string)
|
||||
}))
|
||||
}))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
description = "Prevent Terraform from destroying data resources."
|
||||
type = bool
|
||||
default = false
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam" {
|
||||
description = "Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "iam_by_principals" {
|
||||
description = "Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the `iam` variable."
|
||||
type = map(list(string))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "nodepools" {
|
||||
description = "Nodepools configuration. Refer to the gke-nodepool module for type details."
|
||||
type = map(map(object({
|
||||
gke_version = optional(string)
|
||||
k8s_labels = optional(map(string), {})
|
||||
max_pods_per_node = optional(number)
|
||||
name = optional(string)
|
||||
node_config = optional(any, {
|
||||
disk_type = "pd-balanced"
|
||||
shielded_instance_config = {
|
||||
enable_integrity_monitoring = true
|
||||
enable_secure_boot = true
|
||||
}
|
||||
})
|
||||
node_count = optional(map(number), {
|
||||
initial = 1
|
||||
})
|
||||
node_locations = optional(list(string))
|
||||
nodepool_config = optional(any)
|
||||
network_config = optional(object({
|
||||
enable_private_nodes = optional(bool, true)
|
||||
pod_range = optional(object({
|
||||
cidr = optional(string)
|
||||
create = optional(bool, false)
|
||||
name = optional(string)
|
||||
}), {})
|
||||
additional_node_network_configs = optional(list(object({
|
||||
network = string
|
||||
subnetwork = string
|
||||
})), [])
|
||||
additional_pod_network_configs = optional(list(object({
|
||||
subnetwork = string
|
||||
secondary_pod_range = string
|
||||
max_pods_per_node = string
|
||||
})), [])
|
||||
}))
|
||||
reservation_affinity = optional(any)
|
||||
service_account = optional(any)
|
||||
sole_tenant_nodegroup = optional(string)
|
||||
tags = optional(list(string))
|
||||
taints = optional(map(object({
|
||||
value = string
|
||||
effect = string
|
||||
})))
|
||||
})))
|
||||
default = {}
|
||||
nullable = false
|
||||
}
|
||||
|
||||
variable "stage_config" {
|
||||
description = "FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management."
|
||||
type = object({
|
||||
environment = string
|
||||
name = string
|
||||
})
|
||||
default = {
|
||||
environment = "dev"
|
||||
name = "gke-dev"
|
||||
}
|
||||
}
|
||||
|
||||
variable "vpc_config" {
|
||||
description = "VPC-level configuration for project and clusters."
|
||||
type = object({
|
||||
host_project_id = string
|
||||
vpc_self_link = string
|
||||
})
|
||||
nullable = false
|
||||
default = {
|
||||
host_project_id = "dev-spoke-0"
|
||||
vpc_self_link = "dev-spoke-0"
|
||||
}
|
||||
}
|
||||
@@ -4,21 +4,6 @@ If you want to destroy a previous FAST deployment in your organization, follow t
|
||||
|
||||
Destruction must be done in reverse order, from stage 3 to stage 0
|
||||
|
||||
## Stage 3 (GKE)
|
||||
|
||||
Terraform refuses to delete non-empty GCS buckets and BigQuery datasets, so they need to be removed manually from the state.
|
||||
|
||||
```bash
|
||||
cd $FAST_PWD/3-gke-multitenant/dev/
|
||||
|
||||
# remove BQ dataset manually
|
||||
for x in $(terraform state list | grep google_bigquery_dataset); do
|
||||
terraform state rm "$x";
|
||||
done
|
||||
|
||||
terraform destroy
|
||||
```
|
||||
|
||||
## Stage 3 (Data Platform)
|
||||
|
||||
Terraform refuses to delete non-empty GCS buckets and BigQuery datasets, so they need to be removed manually from the state.
|
||||
|
||||
@@ -45,8 +45,6 @@ To destroy a previous FAST deployment follow the instructions detailed in [clean
|
||||
## Environment-level resources (3)
|
||||
|
||||
- [Data Platform](./3-data-platform-dev/)
|
||||
- [GKE Multitenant](./3-gke-dev/)
|
||||
- [Google Cloud VMware Engine](./3-gcve-dev/)
|
||||
|
||||
|
||||
## Importing existing setup into FAST
|
||||
|
||||
Reference in New Issue
Block a user