Commit Graph

992 Commits

Author SHA1 Message Date
Julio Castillo
993bef71aa Introduce mandatory OWNERS file for blueprint maintainership (#2131)
* Delete deprecated/broken blueprints

* Adding OWNERS to all blueprints

* Fix links

* Update OWNERS

---------

Co-authored-by: javiergp <javiergp@users.noreply.github.com>
2024-03-08 09:40:46 +01:00
Ben Swenka
dcd54246cc Updated diagram to better reflect code naming. (#2133)
* Updated diagram to better reflect code naming.
2024-03-06 19:23:36 +00:00
Julio Castillo
dfed7cc5bd Rename modules/cloudsql-instance deletion protection variables (#2135)
* Rename Cloud SQL deletion protection variables

* Fix tests
2024-03-06 10:44:54 +00:00
Ludovico Magnocavallo
e12cf83188 add links to factories doc (#2134) 2024-03-06 08:25:43 +01:00
simonebruzzechesse
a5f8c501a0 Fix phpipam blueprint (#2119)
* fix phpipam blueprint
2024-02-29 09:33:07 +01:00
Julio Castillo
30496068e2 Implement GKE patterns naming conventions (#2120)
* GKE Pattern Naming

* Add MySQL pattern estimated cost

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2024-02-29 06:57:22 +00:00
simonebruzzechesse
a34d93fb43 Gitlab blueprint (#2110)
* add gitlab blueprint
* add TODO.md

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-27 18:36:46 +01:00
Ludovico Magnocavallo
6941313c7d Factories refactor (#1843)
* factories refactor doc

* Adds file schema and filesystem organization

* Update 20231106-factories.md

* move factories out of blueprints and create new factories  README

* align factory in billing-account module

* align factory in dataplex-datascan module

* align factory in billing-account module

* align factory in net-firewall-policy module

* align factory in dns-response-policy module

* align factory in net-vpc-firewall module

* align factory in net-vpc module

* align factory variable names in FAST

* remove decentralized firewall blueprint

* bump terraform version

* bump module versions

* update top-level READMEs

* move project factory to modules

* fix variable names and tests

* tfdoc

* remove changelog link

* add project factory to top-level README

* fix cludrun eventarc diff

* fix README

* fix cludrun eventarc diff

---------

Co-authored-by: Simone Ruffilli <sruffilli@google.com>
2024-02-26 10:16:52 +00:00
Ludovico Magnocavallo
220ab76e40 enable shielded nodes by default on GKE mt blueprint and FAST stage (#2105) 2024-02-22 07:35:27 +00:00
Julio Castillo
8102803d2e Update tutorial.md 2024-02-21 15:01:23 +01:00
Ludovico Magnocavallo
b408494a74 Update main.tf (#2082) 2024-02-16 15:15:21 +01:00
Wiktor Niesiobędzki
e02d871d58 fix handling default values 2024-02-15 21:57:47 +01:00
Wiktor Niesiobędzki
b1956403d0 Fix missing region 2024-02-15 21:57:47 +01:00
Wiktor Niesiobędzki
19be54a72f Use Fabric modules when possibile 2024-02-15 21:57:47 +01:00
Ben Swenka
d158aecba1 Clarifying that the default path --> Producer A 2024-02-15 18:27:26 +01:00
Ben Swenka
4513d7de0a Updated path matchers to be more user friendly, added better test examples in README.md 2024-02-15 18:27:26 +01:00
simonebruzzechesse
d11c380aec Format python files in blueprints (#2079)
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
2024-02-15 09:37:49 +01:00
Ben Swenka
87a350db93 Bswenka/psc glb and armor 2 producers (#2071)
* Enhanced this blueprint to add a second producer, and modularized the producer.

* Fixed terraform formatting

* Updating README.md with tfdoc

* Fixed test case conditions & module variable passing
2024-02-14 16:40:51 +01:00
Wiktor Niesiobędzki
b902b1dab9 Fix non-empty plan after apply 2024-02-13 07:40:31 +01:00
Ludovico Magnocavallo
71a64487d5 Extend FAST to support different principal types (#2064)
* add doc draft

* typos

* typo

* typo

* typos

* rewording

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* move iam variables to a separate file

* move billing-account module to iam_principals

* move data-catalog-policy-tag module to iam_principals

* move dataplex-datascan module to iam_principals

* move dataproc module to iam_principals

* move folder module to iam_principals

* copyright

* move organization module to iam_principals

* move project module to iam_principals

* move source-repository module to iam_principals

* update blueprints for iam_principals interface

* FAST bootstrap

* module READMEs fixes

* FAST bootstrap

* FAST networking stages

* FAST security stage

* FAST gke stage

* FAST multitenant bootstrap stage

* FAST multitenant resman stage

* tfdoc

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* fix module test

* Update 0-domainless-iam.md

* Update 0-domainless-iam.md

* Rename iam_principals to iam_by_principals

* Update IAM template to include iam_by_principals

* Update Resman README

* Fix ADR link format

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-12 14:35:30 +01:00
Ben Swenka
89779c0d0e Bswenka/psc glb and armor subnet fix (#2058)
* Updated ilb-subnetwork issue

* Fixed Terraform formatting issue

* Fixing merge issue

* Fixing merge issue in producer.tf

* Fixed linting issues

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 10:41:14 +00:00
Wiktor Niesiobędzki
597579fa2b HA MySQL cluster deployment on GKE (#2061)
* MySQL pattern on GKE

* Use terraform managed password

* Use hardcoded network references

* Explain why Cloud NAT

* Rename versions_override.tf

* Fix subnet reference

* Fix password

* Fix MysQL connect commands

* Remove self-link

* Update README.md

* Add TOC and Variables table

* Fix outputs

* Fix linter

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-02-09 11:23:35 +01:00
Julio Castillo
c42c4c141f GKE stateful blueprints (#2059)
* var definitions

* skeleton, untested

* fix errors, test with existing cluster

* test vpc creation, todo notes

* initial variables for AR and image

* initial variables for AR and image

* Add support for remote repositories to artifact-registry

* Add support for virtual repositories to artifact-registry

* Add support for extra config options to artifact-registry

* artifact registry module: add validation and precondition, fix tests

* ar module id/name

* registry

* service accoutn and roles

* fetch pods, remove image prefix

* small changes

* use additive IAM at project level

* use additive IAM at project level

* configmaps

* manifests

* fix statefulset manifest

* service manifest

* fix configmap mode

* add todo

* job (broken)

* job

* wait on manifest, endpoints datasource

* fix job

* Fix local

* sa

* Update README.md

* Restructure gke bp

* refactor tree and infra variables

* no create test

* simplify cluster SA

* test cluster and vpc creation

* project creation fixes

* use iam_members variable

* nits

* readme with examples

* readme with examples

* outputs

* variables, provider configuration

* variables, manifests

* start cluster job

* fix redis cluster creation

Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com>

* Revert changes in autopilot cluster

* Default templates path, use namespace for node names

* Update readmes

* Fix IAM bindings

* Make STABLE the default release channel

* Use Cloud DNS as default DNS provider

* Allow optional Cloud NAT creation

* Allow backup agent and proxy only subnet

* Work around terraform not short-circuiting logical operators

* Rename create variables to be more consistent with other blueprints

* Add basic features

* Update variable names

* Initial kafka JS

* Move providers to a new file

* Kafka / Strimzi

* First possibily working version for MySQL (with a lot of todo's left)

* Explicitly use proxy repo + some other fixes

* Strimzi draft

* Refactor variables, use CluterIP as pointer for mysql-router for bootstraping

* Validate number of replicas, autoscale required number of running nodes to n/2+1

* Use seaprate service for bootstrap, do not recreate all resources on change of replicas count as the config is preserved in PV

* Test dual chart kafka

* Update chart for kafka

* Expose basic kafka configuration options

* Remove unused manifest

* Added batch blueprint

* Added README

* switch to kubectl_manifest

* Add README and support for static IP address

* Move namespace creation to helm

* Interpolate kafka variables

* Rename kafka-strimzi to kafka

* Added TUTORIAL for cloudshell for batch blueprint

* deleted tutorial

* Remove commented replace trigger

* Move to helm chart

* WIP of Cloud Shell tutorial for MySQL

* Rename folders

* Fix rename

* Update paths

* Unify styles

* Update paths

* Add Readme links

* Update mysql tutorial

* Fix path according to self-link

* Use relative path to cwd

* Fix service_account variable location

* Fix tfvars creation

* Restore some fixes for helm deployment

* Add cluster deletion_prevention

* Fixes for tutorial

* Update cluster docs

* Fixes to batch tutorial

* Bare bones readme for batch

* Update batch readme

* README fixes

* Fix README title for redis

* Fix Typos

* Make it easy to pass variables from autopilot-cluster to other modules

* Add connectivity test and bastion host

* updates to readme, and gpu fix

* Add versions.tf and README updates

* Fix typo

* Kafka and Redis README updates

* Update versions.tf

* Fixes

* Add boilerplate

* Fix linting

* Move mysql to separate branch

* Update cloud shell links

* Fix broken link

---------

Co-authored-by: Ludo <ludomagno@google.com>
Co-authored-by: Daniel Marzini <44803752+danielmarzini@users.noreply.github.com>
Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
Co-authored-by: Miren Esnaola <mirene@google.com>
2024-02-08 18:28:41 +00:00
Wiktor Niesiobędzki
29232c10c4 Fix failing tests after Terraform upgrade 2024-02-07 17:25:11 +01:00
Ludovico Magnocavallo
d127c25ad0 Shielded nodes and custom service account in FAST GKE stage and blueprint (CSPR-related) (#2036)
* default to shielded nodes in FAST gke stage

* use custom service account in GKE multitenant blueprint
2024-02-01 15:16:00 +00:00
Ludovico Magnocavallo
3b4b355800 Ensure data platform service accounts meet FAST requirements (#2016)
* use 9-chars prefix in data platform example test

* use 9-chars prefix in data platform example test
2024-01-28 14:00:32 +01:00
Wiktor Niesiobędzki
526185fd1f Remove default region for Cloud Function and Cloud Run (#2004)
Remove default region for Cloud Function and Cloud Run
2024-01-24 10:23:40 +00:00
Ludovico Magnocavallo
11d7edac64 Add example to FAST GKE stage, streamline GKE Hub module variables and usage (#1977)
* implement optionals in gke-hub module

* simplify gke hub module call in mc mesh blueprint

* simplify gke hub module call and variables in multitenant blueprint

* gke hub inventory

* provide cluster and fleet examples in stage
2024-01-20 10:06:38 +00:00
lcaggio
208902c8da Fix Data platform foundation (#1992)
* FAST + Minimal DP

* Fix tests

---------

Co-authored-by: Julio Castillo <jccb@google.com>
2024-01-20 08:49:46 +01:00
Aurélien Legrand
eee6aaff2c Network dashboard - fixing 2 bugs: overriden variable and page token … (#1976)
* Network dashboard - fixing 2 bugs: overriden variable and page token not handled properly

* changing pageToken in the regexp
2024-01-15 14:28:17 +01:00
Julio Diez
e623fa4a66 Merge branch 'master' into jd/serverless-program 2024-01-05 21:49:12 +01:00
Julio Diez
8435ad85f3 Address some style comments 2024-01-05 21:48:02 +01:00
Wiktor Niesiobędzki
6c258e6562 Fix description 2023-12-29 12:09:16 +01:00
Wiktor Niesiobędzki
1eea077460 Add service account email to outputs to manage its permissions 2023-12-29 12:09:16 +01:00
Wiktor Niesiobędzki
724dfda09c Provider doesn't set defaults on config
Can't provide just one size (like `web_server` or `triggerrer`) because
of no defaults are taken:
module.composer.google_composer_environment.env: Modifying... [id=***]
╷
│ Error: googleapi: Error 400: Found 6 problems:
│       1) You have to specify Scheduler CPUs not lower than 0.5.
│       2) You have to specify number of schedulers larger than 0.
│       3) You have to specify Web Server CPUs not lower than 0.5.
│       4) You have to specify Worker CPUs not lower than 0.5.
│       5) You have to specify minimum number of workers larger than 0.
│       6) Triggerer memory must be between 1.00GB and 6.50GB for given vCpu

So provide the defaults as set workloads_config == null
2023-12-29 12:09:16 +01:00
Julio Diez
6bad85d758 Update related serverless READMEs 2023-12-29 11:14:16 +01:00
Julio Diez
3685eb8493 Fix tftest values 2023-12-28 20:30:10 +01:00
Julio Diez
bae9845f4b Update README 2023-12-28 20:06:11 +01:00
Julio Diez
2e500c69a7 Avoid permadiff in launch_stage 2023-12-28 17:57:15 +01:00
Julio Diez
2ca24d320e Use of new module cloud-run-v2 2023-12-28 17:30:41 +01:00
Julio Diez
8889c18690 Merge branch 'master' into jd/serverless-program 2023-12-27 12:31:09 +01:00
Simone Ruffilli
87548f9739 Networking Sandbox Blueprint (#1939)
This blueprint creates a networking playground showing a number of different VPC connectivity options:

Hub and spoke via HA VPN
Hub and spoke via VPC peering
Interconnecting two networks via a network virtual appliance (aka NVA)
On top of that, this blueprint implements Policy Based Routing (aka PBR) to show how to force all traffic within a VPC to be funneled through an internal network passthrough load balancer, to implement an Intrusion Prevention System (IPS). PBR is enabled in the hub VPC, matching all traffic originating from within that VPC.
2023-12-21 17:50:38 +01:00
simonebruzzechesse
717f7ecad1 Blueprints naming convention update (#1942)
* update network-dashboard and quota-monitoring naming convention
2023-12-21 17:02:25 +01:00
Simone Ruffilli
0255c80e90 Move squid to __need_fixing (#1936)
* Moved `modules/cloud-config-container/squid` to __need_fixing
* Moved `blueprints/networking/filtering-proxy{,-psc}` to __need_fixing
2023-12-19 14:27:37 +00:00
Ludovico Magnocavallo
21562eefbe don't fail quota fetch on deleted project (#1931) 2023-12-15 20:20:49 +01:00
simonebruzzechesse
c50b732c79 Allow granting network user role on host project from project module and factory (#1930)
* Update shared vpc config for project factory and project module for more granular Shared VPC configuration

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
2023-12-15 14:39:21 +01:00
maunope
1dc6965694 Update quota monitor blueprint to support project discovery (#1924)
* fist test

* dev complete

* update tf with permissions, enabled APIs and discovery root management

* updated readme

* moved projects discovery to a separate method

* reviewed Mauri's changes

* add missing lines from last change

* - fixed discovery page size to 100
- removed last_asset_page_reached var from discover_projects
- added cast to list for projects var in _main, to make the script work both using CLI and pub/sub

* fixed discovery_root default value to work when no value is passed

* fixed tfdoc

* fixed tftest resources #

---------

Co-authored-by: Ludo <ludomagno@google.com>
2023-12-12 19:17:01 +01:00
Ludovico Magnocavallo
bba814c091 Custom role factories for organization and project modules (#1912)
* backport custom role factories

* backport from fast ci/cd branch

* indent

* tfdoc

* fix module tests
2023-12-11 14:16:39 +00:00
Wiktor Niesiobędzki
886734e1e9 Add trigger configuration for Composer (#1916) 2023-12-11 12:54:49 +01:00
Ludovico Magnocavallo
f548b65b1c Add support for subnet-level service network user grants to project module, improve docs (#1907)
* improve project factory example

* light refactor of project modules shared vpc internals and docs

* add support for subnet-level grants on host project
2023-12-07 09:07:48 +00:00