End-to-end tests for terraform modules (#1751)

Add end-to-end tests (apply, plan, destroy) for examples.

When run, `tests/examples_e2e`:
1. Create an environment for tests to run (folder, project vpc network) 
2. For each marked example (with `e2e` tftest directive), run apply, plan, destroy
3. Verify:
* no failure in apply
* empty plan after apply
* no failure during destroy
4. When all tests are done, destroy test environment

More details in `tests/examples_e2e/README.md`

tests/examples_e2e/README.md

@@ -0,0 +1,88 @@
+# Prerequisites
+Prepare following information:
+* billing account id
+* your organization id
+* parent folder under which resources will be created
+  * (you may want to disable / restore to default some organization policies under this folder) 
+* decide in which region you want to deploy (choose one, that has wide service coverage)
+* prepare a prefix, suffix and a timestamp for you (this is to provide project and other resources name uniqueness)
+* prepare service account that has necessary permissions (able to assign billing account to project, resource creation etc)
+
+# How does it work
+Each test case is provided by additional environment defined in [variables.tf](./variables.tf). This simplifies writing the examples as this follows the same structure as for non-end-to-end tests, and allows multiple, independent and concurrent runs of tests.
+
+The test environment can be provisioned automatically during the test run (which now takes ~2 minutes) and destroyed and the end, when of the tests (Option 1 below), which is targeting automated runs in CI/CD pipeline, or can be provisioned manually to reduce test time, which might be typical use case for tests run locally.
+
+# Option 1 - automatically provision and de-provision testing infrastructure
+
+## Create `e2e.tfvars` file
+```hcl
+billing_account = "123456-123456-123456"  # billing account id to associate projects
+organization_id = "1234567890" # your organization id
+parent          = "folders/1234567890"  # folder under which test resources will be created
+prefix          = "your-unique-prefix"  # unique prefix for projects
+region          = "europe-west4"  # region to use
+
+# tftest skip
+```
+And set environment variable pointing to the file:
+```bash
+export TFTEST_E2E_SETUP_TFVARS_PATH=<path to e2e.tfvars file>
+```
+
+Or set above variables in environment:
+```bash
+export TF_VAR_billing_account="123456-123456-123456"  # billing account id to associate projects
+export TF_VAR_organization_id="1234567890" # your organization id
+export TF_VAR_parent="folders/1234567890"  # folder under which test resources will be created
+export TF_VAR_prefix="your-unique-prefix"  # unique prefix for projects
+export TF_VAR_region="europe-west4"  # region to use
+```
+
+To use Service Account Impersonation, use provider environment variable
+```bash
+export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=<username>@<project-id>.iam.gserviceaccount.com
+```
+
+You can keep the prefix the same for all the tests run, the tests will add necessary suffix for subsequent runs, and in case tests are run in parallel, use separate suffix for the workers.
+# Run the tests
+```bash
+pytest tests/examples_e2e
+```
+
+# Option 2 - Provision manually test environment and use it for tests
+## Provision manually test environment
+In `tests/examples_e2e/setup_module` create `terraform.tfvars` with following values:
+```hcl
+billing_account = "123456-123456-123456"  # billing account id to associate projects
+organization_id = "1234567890"  # your organization id
+parent          = "folders/1234567890"  # folder under which test resources will be created
+prefix          = "your-unique-prefix"  # unique prefix for projects
+region          = "europe-west4"  # region to use
+suffix          = "1" # suffix, keep 1 for now
+timestamp       = "1696444185" # generate your own timestamp - will be used as a part of prefix
+# tftest skip
+```
+
+If you use service account impersonation, set `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT`
+```bash
+export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=<username>@<project-id>.iam.gserviceaccount.com
+```
+
+Provision the environment using terraform
+```bash
+(cd tests/examples_e2e/setup_module/ && terraform init && terraform apply)
+```
+
+This will generate also `tests/examples_e2e/setup_module/e2e_tests.tfvars` for you, which can be used by tests.
+
+## Setup your environment
+```bash
+export TFTEST_E2E_TFVARS_PATH=`pwd`/tests/examples_e2e/setup_module/e2e_tests.tfvars  # generated above
+```
+
+## Run tests
+Run tests using:
+```bash
+pytest tests/examples_e2e
+```

tests/examples_e2e/__init__.py

@@ -0,0 +1,13 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

tests/examples_e2e/conftest.py

@@ -0,0 +1,21 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Pytest configuration for testing code examples."""
+
+from ..examples.conftest import pytest_generate_tests as _examples_generate_test
+
+
+def pytest_generate_tests(metafunc):
+  """Find all README.md files and collect code examples tagged for testing."""
+  _examples_generate_test(metafunc, "examples_e2e", lambda x: 'e2e' in x)

tests/examples_e2e/setup_module/e2e_tests.tfvars.tftpl

@@ -0,0 +1,50 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+bucket= "${bucket}"
+billing_account_id = "${billing_account_id}"
+kms_key= {
+    self_link = "kms_key_self_link"
+}
+organization_id = "${organization_id}"
+folder_id = "${folder_id}"
+prefix = "${prefix}"
+project_id = "${project_id}"
+region = "${region}"
+service_account = {
+    id        = "${service_account.id}"
+    email     = "${service_account.email}"
+    iam_email = "${service_account.iam_email}"
+  }
+subnet = {
+    name      = "${subnet.name}"
+    region    = "${subnet.region}"
+    cidr      = "${subnet.ip_cidr_range}"
+    self_link = "${subnet.self_link}"
+  }
+vpc = {
+    name      = "${vpc.name}"
+    self_link = "${vpc.self_link}"
+    id        = "${vpc.id}"
+  }
+
+# vpc1 =  {
+#    name      = "vpc_name"
+#    self_link = "projects/xxx/global/networks/bbb"
+#  }
+#vpc2 = {
+#    name      = "vpc2_name"
+#    self_link = "projects/xxx/global/networks/ccc"
+#  }
+zone = "${region}-a"

tests/examples_e2e/setup_module/main.tf

@@ -0,0 +1,107 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+locals {
+  prefix = "${var.prefix}-${var.timestamp}-${var.suffix}"
+  services = [
+    # trimmed down list of services, to be extended as needed
+    "cloudbuild.googleapis.com",
+    "cloudfunctions.googleapis.com",
+    "cloudresourcemanager.googleapis.com",
+    "compute.googleapis.com",
+    "iam.googleapis.com",
+    "run.googleapis.com",
+    "serviceusage.googleapis.com",
+    "stackdriver.googleapis.com",
+    "storage-component.googleapis.com",
+    "storage.googleapis.com",
+  ]
+}
+
+resource "google_folder" "folder" {
+  display_name = "E2E Tests ${var.timestamp}-${var.suffix}"
+  parent       = var.parent
+}
+
+resource "google_project" "project" {
+  name            = "${local.prefix}-prj"
+  billing_account = var.billing_account
+  folder_id       = google_folder.folder.id
+  project_id      = "${local.prefix}-prj"
+}
+
+resource "google_project_service" "project_service" {
+  for_each                   = toset(local.services)
+  service                    = each.value
+  project                    = google_project.project.project_id
+  disable_dependent_services = true
+}
+
+resource "google_storage_bucket" "bucket" {
+  location      = var.region
+  name          = "${local.prefix}-bucket"
+  project       = google_project.project.project_id
+  force_destroy = true
+  depends_on    = [google_project_service.project_service]
+}
+
+resource "google_compute_network" "network" {
+  name                    = "e2e-test"
+  project                 = google_project.project.project_id
+  auto_create_subnetworks = false
+  depends_on              = [google_project_service.project_service]
+}
+
+resource "google_compute_subnetwork" "subnetwork" {
+  ip_cidr_range = "10.0.16.0/24"
+  name          = "e2e-test-1"
+  network       = google_compute_network.network.name
+  project       = google_project.project.project_id
+  region        = var.region
+}
+
+resource "google_service_account" "service_account" {
+  account_id = "e2e-service-account"
+  project    = google_project.project.project_id
+  depends_on = [google_project_service.project_service]
+}
+
+resource "local_file" "terraform_tfvars" {
+  filename = "e2e_tests.tfvars"
+  content = templatefile("e2e_tests.tfvars.tftpl", {
+    bucket             = google_storage_bucket.bucket.name
+    billing_account_id = var.billing_account
+    organization_id    = var.organization_id
+    folder_id          = google_folder.folder.folder_id
+    prefix             = local.prefix
+    project_id         = google_project.project.project_id
+    region             = var.region
+    service_account = {
+      id        = google_service_account.service_account.id
+      email     = google_service_account.service_account.email
+      iam_email = "serviceAccount:${google_service_account.service_account.email}"
+    }
+    subnet = {
+      name          = google_compute_subnetwork.subnetwork.name
+      region        = google_compute_subnetwork.subnetwork.region
+      ip_cidr_range = google_compute_subnetwork.subnetwork.ip_cidr_range
+      self_link     = google_compute_subnetwork.subnetwork.self_link
+    }
+    vpc = {
+      name      = google_compute_network.network.name
+      self_link = google_compute_network.network.self_link
+      id        = google_compute_network.network.id
+    }
+  })
+}

tests/examples_e2e/setup_module/variables.tf

@@ -0,0 +1,35 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "billing_account" {
+  type = string
+}
+variable "organization_id" {
+  type = string
+}
+variable "parent" {
+  type = string
+}
+variable "prefix" {
+  type = string
+}
+variable "region" {
+  type = string
+}
+variable "suffix" {
+  type = string
+}
+variable "timestamp" {
+  type = string
+}

tests/examples_e2e/test_plan.py

@@ -0,0 +1,35 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import re
+
+from pathlib import Path
+
+BASE_PATH = Path(__file__).parent
+COUNT_TEST_RE = re.compile(r'# tftest +modules=(\d+) +resources=(\d+)' +
+                           r'(?: +files=([\w@,_-]+))?' +
+                           r'(?: +inventory=([\w\-.]+))?')
+
+
+def test_example(e2e_validator, tmp_path, examples_e2e, e2e_tfvars_path):
+  (tmp_path / 'fabric').symlink_to(BASE_PATH.parents[1])
+  (tmp_path / 'variables.tf').symlink_to(BASE_PATH / 'variables.tf')
+  (tmp_path / 'main.tf').write_text(examples_e2e.code)
+  assets_path = BASE_PATH.parent / str(examples_e2e.module).replace(
+      '-', '_') / 'assets'
+  if assets_path.exists():
+    (tmp_path / 'assets').symlink_to(assets_path)
+  (tmp_path / 'terraform.tfvars').symlink_to(e2e_tfvars_path)
+
+  e2e_validator(module_path=tmp_path, extra_files=[],
+                tf_var_files=[(tmp_path / 'terraform.tfvars')])

tests/examples_e2e/variables.tf

@@ -0,0 +1,92 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# common variables used for examples
+
+variable "bucket" {
+  default = "bucket"
+}
+
+variable "billing_account_id" {
+  default = "123456-123456-123456"
+}
+
+variable "kms_key" {
+  default = {
+    self_link = "kms_key_self_link"
+  }
+}
+
+variable "organization_id" {
+  default = "organizations/1122334455"
+}
+
+variable "folder_id" {
+  default = "folders/1122334455"
+}
+
+variable "prefix" {
+  default = "test"
+}
+
+variable "project_id" {
+  default = "project-id"
+}
+
+variable "region" {
+  default = "region"
+}
+
+variable "service_account" {
+  default = {
+    id        = "service_account_id"
+    email     = "service_account_email"
+    iam_email = "service_account_iam_email"
+  }
+}
+
+variable "subnet" {
+  default = {
+    name      = "subnet_name"
+    region    = "subnet_region"
+    cidr      = "subnet_cidr"
+    self_link = "subnet_self_link"
+  }
+}
+
+variable "vpc" {
+  default = {
+    name      = "vpc_name"
+    self_link = "projects/xxx/global/networks/aaa"
+    id        = "projects/xxx/global/networks/aaa"
+  }
+}
+
+variable "vpc1" {
+  default = {
+    name      = "vpc_name"
+    self_link = "projects/xxx/global/networks/bbb"
+  }
+}
+
+variable "vpc2" {
+  default = {
+    name      = "vpc2_name"
+    self_link = "projects/xxx/global/networks/ccc"
+  }
+}
+
+variable "zone" {
+  default = "zone"
+}