kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7,009 Commits 1 Branch 1 Tag
34d99733ba15b70144ea8ebd1adb4490b1811e51
Commit Graph

44 Commits

Author SHA1 Message Date
Ludovico Magnocavallo
34d99733ba Fix test race condition by ignoring pytest-* directories (#3954) 2026-05-10 17:33:30 +02:00
Ludovico Magnocavallo
1fd1c9ae98 Fix random test failures by ignoring .git in copytree (#3949) 2026-05-08 20:57:40 +02:00
Julio Castillo
ff5cd25869 Ignore md5_hash in plan_summary 2026-04-27 09:50:15 +02:00
Wiktor Niesiobędzki
608a1eff2a Strip values when comparing strings 2025-10-24 13:11:17 +02:00
Ludovico Magnocavallo
36648b6b63 FAST light implementation (#3255) 
* data wip

* wip data

* update org schema, add note on expansion

* all schemas, workload notes

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* Update WORKLOG.md

* wip

* data wip

* wip

* wip

* wip

* wip

* org module IAM context (using lookup)

* new-style context expansion in project IAM

* remove spurious file

* project module contexts

* finalize context replacement format for project module

* revert org module changes

* fix tag id interpolation in project

* fix tag id interpolation in project

* organization module context

* organization context test

* context expansion for folder tag bindings

* test context expansion for tag bindings

* service account module context

* simplify context local

* context for iam service account

* nuke blueprints

* remove links to blueprints

* vpc sc context in project module

* Add context to GCS module

* Add inline deps to plan_summary script

* Make context a top-level variable for folder, organization, sa

* Add add context top-level to VPC-SC

* move context out of factories_config variable

* tfdoc

* fix merge

* fix merge

* fix examples

* net-vpc module context

* add parent ids to folder context

* rename folder parent context

* fix folder parent check

* new project factory stub

* wip

* wip

* refactor defaults

* project iam

* bueckts and service accounts

* start adding context replacements

* better test data

* automation resources for folders and projects

* automation

* add support for project id interpolation

* first tested apply

* improve IAM description in gcs module

* add context to billing account module

* add notification channels to billing account module context

* add billing budgets to new pf

* schemas and defaults

* bootstrap wip

* bootstrap wip

* bootstrap wip

* pf outputs

* pf fixes

* fix pf sample data

* bootstrap lite fixes

* add locations to organization module contexts

* bootstrap lite fixes

* org fixes, billing accounts

* fix default project parent

* bootstrap lite wip

* add locations to gcs module context

* add context support to logging bucket module

* add context to pubsub module

* split out iam variables in gcs module

* fix logging bucket context test

* bootstrap log sink destinations

* streamline logging-bucket module variables

* fix logging bucket context test

* align logging bucket module interface in fast bootstrap

* add support for project-level log buckets to project factory

* support full context expansion in organization module log sinks

* log buckets in fast-lite bootstrap

* make og sink type optional in organization module

* log sinks in fast-lite bootstrap

* set tag values in factory context

* bootstrap lite data

* output files schema

* billing account schema

* output files

* output providers

* gcs output files

* boilerplate

* tflint

* check documentation

* check docs

* fix project module parent variable validation

* fix log bucket examples

* allow null parent in project module

* silence folder test errors

* fix billing account sink example

* fix project example

* fix billing account module

* fix folder tests

* fix FAST

* fix fast

* tfvars outputs

* wif

* cicd service accounts

* cicd

* allow defaults in context, minimal org policies

* support gcs managed folders in project factory and bootstrap lite

* support prefix in provider output files

* rename bootstrap stage

* gitignore

* gitignore

* security folder, billing IAM

* wip tfvars

* fix typo

* security IAM

* control tag iam/context via variables in organization module

* split tag creation from tag IAM to avoid circular refs

* port organization module tag changes to project module

* implement new-style context expansion in vpc-sc module

* fix fast vpc-sc tests

* boilerplate

* vpc sc stage

* schemas

* fast-lite compatibility for vpc sc stage

* make log project number optional in vpc-sc stage

* networking

* networking

* networking

* networking

* rename and move new stage under fast

* clone pf tests

* use context replacement for internal notification channels in billing account module

* support service agents in project module iam context replacements

* support service agents in project module iam context replacements

* add support for kms keys to project module context

* experimental pf example test and fixes

* fix schemas

* fix tests

* tfdoc

* tfdoc

* pf config

* experimental pf

* remove redundant dot from gcs managed folder IAM keys

* bootstrap experimental test

* project factory exp stage test

* skip tflint for bootstrap experimental test

* tflint

* fix gcs test

* documentation work

* documentation work

* Update README.md

* tfdoc

* tfdoc

* readme

* tfdoc

* readme

* readme

* readme

* readme

* support universe in pf exp projects

* missing universe service agents

* org policies import, non-admin billing IAM

* todo

* fix test

* custom constraints

* fast classic dataset

* fix test data

* context replacements in billing module log sinks

* fix typo

* add support for billing log sinks

* update docs

* readme

* cicd fix and test

---------

Co-authored-by: Julio Castillo <jccb@google.com>
 2025-09-02 08:38:57 +02:00
Ludovico Magnocavallo
ef45aa704a Rename workflows config variable introduced in #3198 (#3199) 
* rename workflows_config.tf_var_files in resman stage

* fix test result

* fix cloud deploy module versions file
 2025-06-28 08:57:55 +00:00
Ludovico Magnocavallo
cb7bed50e3 New FAST data platform (#3066) 
* copy from broken dp dev branch

* remove extra excalidraw file

* fix networking yaml

* tfdoc

* tfdoc

* nuke old data platform

* fix tests

* tests

* tflint

* high level diagram

* make location optional in composer schema

* add composer outputs

* docs

* remove schema docs

* tfdoc

* update service agent encryption composer def for composer 3

* encryption keys

* typo

* typo

* fix security IAM

* inventory

* tflint

* Fix roles and diagram.

* Fix tflint

* Fix test DP.

* Fix test

* Diagrams excalidraw gz

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
 2025-05-03 21:21:38 +00:00
Wiktor Niesiobędzki
c90005553a Add tests for data_defaults / data_overrides 2025-04-06 20:17:22 +02:00
Ludovico Magnocavallo
c8e4179f2b Add custom routes for directpath to net-vpc module (#2966) 
* add custom routes for directpath to net-vpc module

* blueprint tests

* blueprint tests

* blueprint tests

* fast tests

* tfdoc

* module examples
 2025-03-19 10:22:47 +00:00
Wiktor Niesiobędzki
12979e8f50 Introduce test isolation and fix missing GCS service account 2025-03-01 14:45:16 +01:00
Wiktor Niesiobędzki
fd07db5000 Add tftests for project factory 2025-02-19 16:47:15 +01:00
Ludovico Magnocavallo
9148e76087 Add support for project-level log sinks to FAST stage 0 (#2893) 
* Fix ipv6 and align loadbalancer address types

* Release v37.3.0

* add support for project-level log sinks to stage 0

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
 2025-02-14 11:58:18 +01:00
Ludovico Magnocavallo
95ec5ee3b5 Flexible stage 2s in FAST resource manager (#2840) 
* wip

* WIP

* wip

* wip

* apply untested

* tests

* support tag expansion for tenant-level installations in IAM conditions

* fix stage config output

* inventories

* remove dev files

* tfdoc

* enable org policies for stage folders

* resman README

* tfdoc

* stage 3 documentation

* inventory

* support extra_dirs in testing franework

* remove org policy files from stage 1

* Add principal interpolation to iam_by_principals (#2847)

* Add principal interpolation to iam_by_principals

* Fix tests

* relax schemas

* relax schemas

---------

Co-authored-by: Julio Castillo <jccb@google.com>
 2025-01-29 12:16:35 +00:00
Ludovico Magnocavallo
27f1cc2b79 Implement FAST stage add-ons, refactor netsec as add-on (#2800) 
* security fixes

* change netsec to be a virtual stage in resman

* remove netsec bits from security stage, leave CAs in place

* netsec - security profile groups

* export regions to networking tfvars

* netsec - trust stores

* netsec refactor, untested

* netsec plan working

* netsec apply

* netsec apply errors

* netsec diagram

* update diagram

* move addon stages to addons folder

* remove top-level assets folder

* deprecate and remove fast plugins

* addon tests

* dynamic addon providers and cicd, untested

* stage 1 addons in stage 0, refactor stage 0 cicd

* addons and cicd refactor in stage 0 with tests

* refactor stage 0 cicd

* readd removed block

* small bootstrap cicd fixes

* refactor stage 1 cicd

* resman tests

* remove plugins from networking tests

* fix fast tests

* ngfw addon outputs

* try to fix unrelated tflint error in bootstrap

* remove common tfvars from bootstrap tests to fix linter errors

* tfdoc

* minimal readmes and links fixes

* tfdoc

* trim down test inventories

* fix plan test

* tfdoc

* allow configuring output files names

* fix tls inspection after adding count to project module

* comment fixes

* tfdoc
 2025-01-09 18:14:11 +00:00
Ludovico Magnocavallo
345716e576 VPC-SC as separate FAST stage 1 (#2460) 
* initial commit

* README

* boilerplate

* tflint

* tfdoc

* fix security stage tests

* vpc-sc stage tests

* tflint

* fix resman stage test inventories

* security README

* stage-level README

* Update README.md

* flexible perimeter variable

* remove diagram

* change default to dry run

* default to dry run
 2024-08-02 18:04:36 +02:00
Luca Prete
140c4e6446 Add --extra-files option to plan_summary.py cmd (#2452) 2024-07-30 11:29:19 +02:00
dibaskar-google
3d668170e5 Secret manager e2etests (#2280) 
secret-manager e2e tests
 2024-05-28 09:28:08 +02:00
Wiktor Niesiobędzki
6a3c7fe444 CloudSQL PSC Endpoints support (#2242) 
* Add PSC endpoints consumers to net-address
* Cloud SQL E2E tests
 2024-05-12 12:00:39 +02:00
Ludovico Magnocavallo
fbc7e891db Net LB App Internal Cross-Region recipe (#2214) 
* reorder tfdoc methods

* add support for recipes to tfdoc

* fix repo url in tfdoc

* update module README

* validated untested recipe

* validated untested refactored recipe

* add optional proxy subnet creation, outputs, fixes

* tested

* tfdoc fix

* fix README

* exclude examples from test collector
 2024-04-14 19:38:05 +03:00
bluPhy
c420f17636 Updating cloud-run-v2 terraform and some typos (#2201) 
* Updating cloud-run-v2 terraform and some typos

Updating modules/cloud-run-v2/main.tf to determine the tcp_socket value

Fixin typos in:
tests/fixtures.py
modules/workstation-cluster/README.md
modules/net-lb-int/README.md
modules/dataplex/README.md
modules/dataform-repository/README.md
modules/__docs/20230816-iam-refactor.md
CONTRIBUTING.md
blueprints/third-party-solutions/f5-bigip/f5-bigip-ha-active/startup-script.tpl
blueprints/third-party-solutions/f5-bigip/f5-bigip-ha-active/README.md

* Update README.md

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Ludovico Magnocavallo <ludo@qix.it>
 2024-04-07 09:49:07 +00:00
Wiktor Niesiobędzki
bee3072568 Add support for Cloud Run v2 jobs (#1954) 
Add support for Cloud Run v2 jobs

* create a separate file for service creation (service.tf) and job
  (job.tf) - for easy comparison
* add E2E tests where possibile
* remove default value for input variable `region`
* fix subnet range VPC Access Connector example
* add creation of service account for audit logs call (trigger requires
  service account)
* use provided trigger service account email in
  `local.trigger_sa_email`, so explicitly provided SA is passed to
  trigger
* set default value for vpc_connector_create.throughput.max, to match
  what is set by GCP API, as provider uses wrong default of 300 which
  results in perma-diff
* create inventory fiels for all examples

Global changes
* (tests) add input variable `project_number`, to allow assigning IAM permissions to Service Accounts in fixtures
* (tests) fix not outputting the path, when object is not found in inventory
* (tests) fix `create_e2e_sandbox.sh` - now it properly finds root of the repo

Secret Manager
* added `version_versions` output, to allow specifying versions in other modules. `versions` is sensitive and it makes it unsuitable for `for_each` values

New test fixtures
* `pubsub.tf` - creating one topic
* `secret-credential.tf` - creating Secret Manager `credential` secret
* `shared-vpc.tf` - creating two projects (host and service), and vpc in host project
* `vpc-connector.tf` - creating VPC Access Connector instance
 2024-02-18 14:57:34 +01:00
Ludovico Magnocavallo
a8c84357f4 Integrate checklist data in FAST (#1969) 
* add locals for additive and authoritative org iam roles

* first shot at IAM and logging location

* tfdoc

* use locals for locations

* fix file parsing, resman stubs

* initial resman implementation

* remove unneeded code

* fix data file

* replace dumb yamldecode

* fix wrong type in organization additive bindings try

* simplify logging local

* Use check asserts for version and org id

* Checks on checklist for resman

* refactor checks, ignore checklist files on wrong org id

* stage 0 tests

* fix checklist checks

* stage 1 tests

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
 2024-01-18 05:45:29 +01:00
Wiktor Niesiobędzki
a34cdd5597 Better error reporting when missing setup for E2E tests (#1985) 2024-01-17 23:34:20 +03:00
Julio Castillo
fde7b76036 Allow per-module terraform fixtures (#1914) 
* Allow terraform fixtures for examples

* Allow defining multiple fixtures, and named fixtures under tests/fixtures/

* Enable e2e for wiktorn

* Fix prepare_files call for e2e

* Move fixture to separate file, fix test

* Revert shallow-copying symlinks, performane penalty - 20%

* Update tfdoc.py to list used fixtures

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
 2023-12-29 09:43:44 +00:00
Ludovico Magnocavallo
9d6e61428b (WIP) Read-only service accounts for automation and CI/CD (#1899) 
* add design doc for the new CI/CD sa

* describe the actual implementation

* specify which files will need to be changed

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Update 0-cicd-plan-sa.md

* Fix typo

* stage 0 read-only service accounts

* stage 0 IAM map

* linting

* cicd read-only service accounts

* tweak workflow templates

* roles and github workflow fixes

* tfdoc

* Ad-hoc custom role factory for FAST bootstrap

* use factory variable for custom roles data path

* custom roles factory in org/project modules

* tfdoc

* rename custom roles factory variable, fix gitlab template

* gitlab workflow fixes

* fix merge

* output plan results on failed assertion

* update stage 0 expected values

* data platform branch

* gke

* networking

* security

* project factory

* outputs

* workflow templates

* resman apply fixes

* tfdoc

* fix stage 1 test fixture

* fix gh workflow

* read-only resman sa roles

* fix test

* read-only resman sa roles

* read-only resman sa roles

* read-only resman sa roles

* read-only resman sa roles

* fix test variables

* rename wif principal attribute names

* rename wif principal variables

* multitenant stages

---------

Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
 2023-12-27 11:33:16 +00:00
Wiktor Niesiobędzki
c5c127b9df Use TFTEST_E2E_ instead of TF_VAR variables 
Use of TF_VAR variables modified results of `tests/examples` and
required setting different environment to run `tests/examples` and
`tests/examples_e2e` tests. No both can be run using the same
environment.
 2023-11-30 07:43:46 +00:00
Wiktor Niesiobędzki
86cf5dfc84 Fix reporting non-empty plan after apply 2023-11-17 11:47:44 +00:00
Wiktor Niesiobędzki
fe485414e6 Add end-to-end tests for project module (#1823) 
* Add end-to-end tests for project module
* Add inventory to data tests
* Add files to end-to-end test cases
* Review fixes - use named groups

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
 2023-11-03 18:04:19 +01:00
Wiktor Niesiobędzki
d07daf966a End-to-end tests for terraform modules (#1751) 
Add end-to-end tests (apply, plan, destroy) for examples.

When run, `tests/examples_e2e`:
1. Create an environment for tests to run (folder, project vpc network) 
2. For each marked example (with `e2e` tftest directive), run apply, plan, destroy
3. Verify:
* no failure in apply
* empty plan after apply
* no failure during destroy
4. When all tests are done, destroy test environment

More details in `tests/examples_e2e/README.md`
 2023-10-20 09:59:52 +02:00
Wiktor Niesiobędzki
71def9e6dd Allow partial specification for deeply nested inventories 
For objects that are not simple attributes of the resource (i.e.
objects, lists) allow specifing subset of values that needs to be
verified.

In case of lists, count of list objects must match, but if values are
objects / dicts, they can be specified as empty dicts which should pass
all the checks, as no keys are specified.
 2023-09-26 11:22:35 +02:00
Julio Castillo
804e7c961e Silence FAST tests warnings 
- Fix pytest PytestUnraisableExceptionWarning
- Remove incorrect print
- Use tfvars for some examples in READMEs
 2023-08-28 18:40:41 +02:00
Julio Castillo
4d6574e0a1 Print inventory path when a test fails 2023-08-11 12:07:07 +02:00
Julio Castillo
b097d297ff Extend tests to use lockfile if available 
This commints allows the test framework to use a lockfile generated
within the tools/lockfile directory. This allows reusing the provider
cache and also running tests in parallel.
 2023-04-25 15:12:11 +02:00
Julio Castillo
cb13d481d6 Migrate cloud-function tests 2023-04-21 17:51:20 +02:00
Ludovico Magnocavallo
3d41d01efc FAST plugin system (#1266) 
* plugin folder, gitignore, serverless connector example

* add support to fast plugin variables and outputs to tfdoc

* rename folder, READMEs

* add variable description

* show diffs

* check documentation, use multiple files

* debug check doc

* try a different glob

* debug tfdoc names

* more debug

* and even more debug

* fix gitignore

* fix links

* support extra files in tests

* fix fixture, switch stage 2 peering to new tests

* tfdoc

* Allow globs in extra files

---------

Co-authored-by: Julio Castillo <jccb@google.com>
 2023-03-24 12:28:32 +00:00
Ludovico Magnocavallo
5453c585e0 FAST multitenant bootstrap and resource management, rename org-level FAST stages (#1052) 
* rename stages

* remove support for external org billing, rename output files

* resman: make groups optional, align on new billing account variable

* bootstrap: multitenant outputs

* tenant bootstrap stage, untested

* fix folder name

* fix stage 0 output names

* optional creation for tag keys in organization module

* single tenant bootstrap minus tag

* rename output files, add tenant tag key

* fix organization module tag values output

* test skipping creation for tags in organization module

* single tenant bootstrap plan working

* multitenant bootstrap

* tfdoc

* fix check links error messages

* fix links

* tfdoc

* fix links

* rename fast tests, fix bootstrap tests

* multitenant stages have their own folder, simplify stage numbering

* stage renumbering

* wip

* rename tests

* exclude fast providers in fixture

* stage 0 tests

* stage 1 tests

* network stages tests

* stage tests

* tfdoc

* fix links

* tfdoc

* multitenant tests

* remove local files

* stage links command

* fix links script, TODO

* wip

* wip single tenant bootstrap

* working tenant bootstrap

* update gitignore

* remove local files

* tfdoc

* remove local files

* allow tests for tenant bootstrap stage

* tenant bootstrap proxies stage 1 tfvars

* stage 2 and 3 service accounts and IAM in tenant bootstrap

* wip

* wip

* wip

* drop multitenant bootstrap

* tfdoc

* add missing stage 2 SAs, fix org-level IAM condition

* wip

* wip

* optional tag value creation in organization module

* stage 1 working

* linting

* linting

* READMEs

* wip

* Make stage-links script work in old macos bash

* stage links command help

* fix output file names

* diagrams

* fix svg

* stage 0 skeleton and diagram

* test svg

* test svg

* test diagram

* diagram

* readme

* fix stage links script

* stage 0 readme

* README changes

* stage readmes

* fix outputs order

* fix link

* fix tests

* stage 1 test

* skip stage example

* boilerplate

* fix tftest skip

* default bootstrap stage log sinks to log buckets

* add logging to tenant bootstrap

* move iam variables out of tenant config

* fix cicd, reintroduce missing variable

* use optional in stage 1 cicd variable

* rename extras stage

* rename and move identity providers local, use optional for cicd variable

* tfdoc

* add support for wif pool and providers, ci/cd

* tfdoc

* fix links

* better handling of modules repository

* add missing role on logging project

* fix cicd pools in locals, test cicd

* fix workflow extension

* fix module source replacement

* allow tenant bootstrap cicd sa to impersonate resman sa

* tenant workflow templates fix for no providers file

* fix output files, push github workflow template to new repository

* remove try from outpout files

* align stage 1 cicd internals to stage 0

* tfdoc

* tests

* fix tests

* tests

* improve variable descriptions

* use optional in fast features

* actually create tenant log sinks, and allow the resman sa to do it

* test

* tests

* aaaand tests again

* fast features tenant override

* fast features tenant override

* fix wording

* add missing comment

* configure pf service accounts

* add missing comment

* tfdoc

* tests

* IAM docs

* update copyright

---------

Co-authored-by: Julio Castillo <jccb@google.com>
 2023-02-04 15:00:45 +01:00
Julio Castillo
aba14fe574 Allow passing an inventory to examples test directive 2022-12-18 14:00:19 +01:00
Julio Castillo
1981647595 Custom context manager to create temp directory 2022-12-06 15:09:08 +01:00
Julio Castillo
a017fef121 Removed refresh=True from plan() call 2022-12-06 14:44:18 +01:00
Julio Castillo
f8d5f43c05 Use ignore instead of copy+delete in plan_summary() 2022-12-06 10:08:36 +01:00
Julio Castillo
be0e807435 Bring back tests key in test yaml spec 2022-12-06 00:06:29 +01:00
Julio Castillo
284f8ff106 Basic error handling 2022-12-06 00:06:29 +01:00
Julio Castillo
589f7a5c2f Simplify yaml test spec 2022-12-06 00:06:29 +01:00
Julio Castillo
0a6285fd4e Reorder code 2022-12-06 00:06:29 +01:00