kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Use TFTEST_E2E_ instead of TF_VAR variables

Browse Source 
Use of TF_VAR variables modified results of `tests/examples` and
required setting different environment to run `tests/examples` and
`tests/examples_e2e` tests. No both can be run using the same
environment.
This commit is contained in:
Wiktor Niesiobędzki
2023-11-27 21:26:06 +00:00
parent 6112d06706
commit c5c127b9df
4 changed files with 185 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
CONTRIBUTING.md
View File

@@ -1070,6 +1070,116 @@ outputs:
You can now use this output to create the inventory file for your test. As mentioned before, please only use those values relevant to your test scenario.
### Running end-to-end tests
You can use end-to-end tests to verify your code against GCP API. These tests verify that `terraform apply` succeeds, `terraform plan` is empty afterwards and that `terraform destroy` raises no error.
#### Prerequisites
Prepare following information:
* billing account id
* organization id
* parent folder under which resources will be created
* (you may want to disable / restore to default some organization policies under this folder)
* decide in which region you want to deploy (choose one, that has wide service coverage)
* (optional) prepare service account that has necessary permissions (able to assign billing account to project, resource creation etc)
* prepare a prefix (this is to provide project and other global resources name uniqueness)
#### How does it work
Each test case is provided by additional environment defined in [variables.tf](../examples/variables.tf). This simplifies writing the examples as this follows the same structure as for non-end-to-end tests, and allows multiple, independent and concurrent runs of tests.
The test environment can be provisioned automatically during the test run (which takes ~2 minutes) and destroyed at the end, when of the tests (option 1 below), which is targeting automated runs in CI/CD pipeline, or can be provisioned manually (option 2 below) to reduce test time, which might be typical use case for tests run locally.
#### Option 1 - automatically provision and de-provision testing infrastructure
Set variables in environment:
```bash
export TFTEST_E2E_billing_account="123456-123456-123456" # billing account id to associate projects
export TFTEST_E2E_group_email="group@example.org" # existing group within organization
export TFTEST_E2E_organization_id="1234567890" # your organization id
export TFTEST_E2E_parent="folders/1234567890" # folder under which test resources will be created
export TFTEST_E2E_prefix="your-unique-prefix" # unique prefix for projects, no longer than 7 characters
export TFTEST_E2E_region="europe-west4" # region to use
```
To use Service Account Impersonation, use provider environment variable
```bash
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=<username>@<project-id>.iam.gserviceaccount.com
```
You can keep the prefix the same for all the tests run, the tests will add necessary suffix for subsequent runs, and in case tests are run in parallel, use separate suffix for the workers.
# Run the tests
```bash
pytest tests/examples_e2e
```
#### Option 2 - Provision manually test environment and use it for tests
##### Provision manually test environment
In `tests/examples_e2e/setup_module` create `terraform.tfvars` with following values:
```hcl
billing_account = "123456-123456-123456" # billing account id to associate projects
group_email = "group@example.org" # existing group within organization
organization_id = "1234567890" # your organization id
parent = "folders/1234567890" # folder under which test resources will be created
prefix = "your-unique-prefix" # unique prefix for projects
region = "europe-west4" # region to use
suffix = "1" # suffix, keep 1 for now
timestamp = "1696444185" # generate your own timestamp - will be used as a part of prefix
# tftest skip
```
If you use service account impersonation, set `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT`
```bash
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=<username>@<project-id>.iam.gserviceaccount.com
```
Provision the environment using terraform
```bash
(cd tests/examples_e2e/setup_module/ && terraform init && terraform apply)
```
This will generate also `tests/examples_e2e/setup_module/e2e_tests.tfvars` for you, which can be used by tests.
##### Setup your environment
```bash
export TFTEST_E2E_TFVARS_PATH=`pwd`/tests/examples_e2e/setup_module/e2e_tests.tfvars # generated above
export TF_VAR_prefix="your-unique-prefix" # unique prefix for projects, no longer than 7 characters
```
#### Run tests
Run tests using:
```bash
pytest tests/examples_e2e
```
#### Creating sandbox environment for examples
When developing it is convenient to have a module that represents chosen example, so you can inspect the environment after running apply and quickly verify fixes. Shell script [create_e2e_sandbox.sh](tools/create_e2e_sandbox.sh) will create such environment for you.
Prepare the environment variables as defined in Option 1 above and run:
```bash
$ tools/create_e2e_sandbox.sh <directory>
```
The script will create in `<directory>` following structure:
```
<directory>
├── default-versions.tf
├── e2e_tests.auto.tfvars -> infra/e2e_tests.tfvars
├── fabric -> <cloud-foundation-fabric root>
├── infra
│ ├── e2e_tests.tfvars
│ ├── e2e_tests.tfvars.tftpl
│ ├── main.tf
│ ├── randomizer.auto.tfvars
│ ├── terraform.tfvars
│ └── variables.tf
├── main.tf
└── variables.tf
```
The `infra` directory contains the sandbox infrastructure as well as all environment variables dumped into `terraform.tfvars` file. The script runs `terraform init` and `terraform apply -auto-approve` in this folder.
The `<direcotry>` has empty `main.tf` where you can paste any example, and it will get all necessary variables from `e2e_tests.auto.tfvars` file.
If there are any changes to the test sandbox, you can rerun the script and only changes will be applied to the project.
### Writing tests in Python (legacy approach)
Where possible, we recommend using the testing methods described in the previous sections. However, if you need it, you can still write tests using Python directly.

93
tests/examples_e2e/README.md
View File

@@ -1,93 +0,0 @@
# Prerequisites
Prepare following information:
* billing account id
* your organization id
* parent folder under which resources will be created
* (you may want to disable / restore to default some organization policies under this folder)
* decide in which region you want to deploy (choose one, that has wide service coverage)
* prepare a prefix, suffix and a timestamp for you (this is to provide project and other resources name uniqueness)
* prepare service account that has necessary permissions (able to assign billing account to project, resource creation etc)
# How does it work
Each test case is provided by additional environment defined in [variables.tf](../examples/variables.tf). This simplifies writing the examples as this follows the same structure as for non-end-to-end tests, and allows multiple, independent and concurrent runs of tests.
The test environment can be provisioned automatically during the test run (which now takes ~2 minutes) and destroyed and the end, when of the tests (Option 1 below), which is targeting automated runs in CI/CD pipeline, or can be provisioned manually to reduce test time, which might be typical use case for tests run locally.
# Option 1 - automatically provision and de-provision testing infrastructure
## Create `e2e.tfvars` file
```hcl
billing_account = "123456-123456-123456" # billing account id to associate projects
group_email = "group@example.org" # existing group within organization
organization_id = "1234567890" # your organization id
parent = "folders/1234567890" # folder under which test resources will be created
prefix = "your-unique-prefix" # unique prefix for projects
region = "europe-west4" # region to use
# tftest skip
```
And set environment variable pointing to the file:
```bash
export TF_VAR_prefix="your-unique-prefix" # unique prefix for projects, no longer than 7 characters
export TFTEST_E2E_SETUP_TFVARS_PATH=<path to e2e.tfvars file>
```
Or set above variables in environment:
```bash
export TF_VAR_billing_account="123456-123456-123456" # billing account id to associate projects
export TF_VAR_group_email="group@example.org" # existing group within organization
export TF_VAR_organization_id="1234567890" # your organization id
export TF_VAR_parent="folders/1234567890" # folder under which test resources will be created
export TF_VAR_prefix="your-unique-prefix" # unique prefix for projects, no longer than 7 characters
export TF_VAR_region="europe-west4" # region to use
```
To use Service Account Impersonation, use provider environment variable
```bash
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=<username>@<project-id>.iam.gserviceaccount.com
```
You can keep the prefix the same for all the tests run, the tests will add necessary suffix for subsequent runs, and in case tests are run in parallel, use separate suffix for the workers.
# Run the tests
```bash
pytest tests/examples_e2e
```
# Option 2 - Provision manually test environment and use it for tests
## Provision manually test environment
In `tests/examples_e2e/setup_module` create `terraform.tfvars` with following values:
```hcl
billing_account = "123456-123456-123456" # billing account id to associate projects
group_email = "group@example.org" # existing group within organization
organization_id = "1234567890" # your organization id
parent = "folders/1234567890" # folder under which test resources will be created
prefix = "your-unique-prefix" # unique prefix for projects
region = "europe-west4" # region to use
suffix = "1" # suffix, keep 1 for now
timestamp = "1696444185" # generate your own timestamp - will be used as a part of prefix
# tftest skip
```
If you use service account impersonation, set `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT`
```bash
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=<username>@<project-id>.iam.gserviceaccount.com
```
Provision the environment using terraform
```bash
(cd tests/examples_e2e/setup_module/ && terraform init && terraform apply)
```
This will generate also `tests/examples_e2e/setup_module/e2e_tests.tfvars` for you, which can be used by tests.
## Setup your environment
```bash
export TFTEST_E2E_TFVARS_PATH=`pwd`/tests/examples_e2e/setup_module/e2e_tests.tfvars # generated above
export TF_VAR_prefix="your-unique-prefix" # unique prefix for projects, no longer than 7 characters
```
## Run tests
Run tests using:
```bash
pytest tests/examples_e2e
```

24
tests/fixtures.py
View File

@@ -281,6 +281,12 @@ def plan_validator_fixture(request):
return inner
def get_tfvars_for_e2e():
_variables = ["billing_account", "group_email", "organization_id", "parent", "prefix", "region"]
tf_vars = {k: os.environ.get(f"TFTEST_E2E_{k}") for k in _variables}
return tf_vars
def e2e_validator(module_path, extra_files, tf_var_files, basedir=None):
"""Function running apply, plan and destroy to verify the case end to end
@@ -298,12 +304,14 @@ def e2e_validator(module_path, extra_files, tf_var_files, basedir=None):
tf.setup(extra_files=extra_files, upgrade=True)
tf_var_files = [(basedir / x).resolve() for x in tf_var_files or []]
# we need only prefix variable to run the example test, all the other are passed in terraform.tfvars file
prefix = get_tfvars_for_e2e()["prefix"]
# to allow different tests to create projects (or other globally unique resources) with the same name
# bump prefix forward on each test execution
prefix = f'{os.environ.get("TF_VAR_prefix")}-{int(time.time())}{os.environ.get("PYTEST_XDIST_WORKER", "0")[-2:]}'
tf_vars = {"prefix": f'{prefix}-{int(time.time())}{os.environ.get("PYTEST_XDIST_WORKER", "0")[-2:]}'}
try:
apply = tf.apply(tf_var_file=tf_var_files, tf_vars={"prefix": prefix})
plan = tf.plan(output=True, tf_var_file=tf_var_files, tf_vars={"prefix": prefix})
apply = tf.apply(tf_var_file=tf_var_files, tf_vars=tf_vars)
plan = tf.plan(output=True, tf_var_file=tf_var_files, tf_vars=tf_vars)
changes = {}
for resource_name, value in plan.resource_changes.items():
if value.get('change', {}).get('actions') != ['no-op']:
@@ -327,7 +335,7 @@ def e2e_validator(module_path, extra_files, tf_var_files, basedir=None):
# If above did not fail, this should not either, but left as a safety check
assert changes == {}, f'Plan not empty for following resources: {", ".join(changes.keys())}'
finally:
destroy = tf.destroy(tf_var_file=tf_var_files, tf_vars={'prefix': prefix})
destroy = tf.destroy(tf_var_file=tf_var_files, tf_vars=tf_vars)
@pytest.fixture(name='e2e_validator')
@@ -373,10 +381,10 @@ def e2e_tfvars_path():
binary = os.environ.get('TERRAFORM', 'terraform')
tf = tftest.TerraformTest(test_path, binary=binary)
tf_vars_file = None
tf_vars = {
'suffix': os.environ.get("PYTEST_XDIST_WORKER", "0")[-2:], # take at most 2 last chars for suffix
'timestamp': str(int(time.time()))
}
tf_vars = get_tfvars_for_e2e()
tf_vars['suffix'] = os.environ.get("PYTEST_XDIST_WORKER", "0")[-2:] # take at most 2 last chars for suffix
tf_vars['timestamp'] = str(int(time.time()))
if 'TFTEST_E2E_SETUP_TFVARS_PATH' in os.environ:
tf_vars_file = os.environ["TFTEST_E2E_SETUP_TFVARS_PATH"]
tf.setup(upgrade=True)

59
tools/create_e2e_sandbox.sh Executable file
View File

@@ -0,0 +1,59 @@
#!/bin/bash
# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# create_e2e_sandbox.sh <directory>
#
# creates end-to-end tests sandbox in given directory, in result you get:
# 1. <directory>/infra with terraform module applied with all prerequisites provisioned
# 2. <directory> with empty main.tf file, where you can paste any example and manually run any terraform command
#
# You need to have all the environment variables set up only during running of this script. Their values will be
# preserved in terraform.tfvars file, so if you want to destroy / make changes to the infra module, all should work
#
# You can rerun this script on existing directory, this will update all the (terraform) files, but will reuse the existing
# project name and folder name to reduce time to update
#
set -e
DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
DEST=$1
INFRA="${DEST}/infra"
TIMESTAMP=$(date +%s)
mkdir -p "${DEST}" "${INFRA}"
ln -sfT "${DIR}" "${DEST}/fabric"
SETUP_MODULE="${DIR}/tests/examples_e2e/setup_module"
cp "${SETUP_MODULE}/main.tf" "${SETUP_MODULE}/variables.tf" "${SETUP_MODULE}/e2e_tests.tfvars.tftpl" "${INFRA}"
cp "${DIR}/tests/examples/variables.tf" "${DEST}"
if [ ! -f "${INFRA}/randomizer.auto.tfvars" ] ; then
echo "suffix=0" > "${INFRA}/randomizer.auto.tfvars"
echo "timestamp=${TIMESTAMP}" >> "${INFRA}/randomizer.auto.tfvars"
fi
# TODO correct environment variable prefix
export | sed -e 's/^declare -x //' | grep '^TFTEST_E2E_' | sed -e 's/^TFTEST_E2E_//' > "${INFRA}/terraform.tfvars"
(
cd "${INFRA}"
terraform init
terraform apply -auto-approve
ln -sfT "${INFRA}/e2e_tests.tfvars" "${DEST}/e2e_tests.auto.tfvars"
)
tocuh "${DEST}/main.tf"