kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,282 Commits 1 Branch 1 Tag
f02aa1f63f9364cefd6990d5683c370f0519decb
Commit Graph

40 Commits

Author SHA1 Message Date
Ludovico Magnocavallo
cb7bed50e3 New FAST data platform (#3066) 
* copy from broken dp dev branch

* remove extra excalidraw file

* fix networking yaml

* tfdoc

* tfdoc

* nuke old data platform

* fix tests

* tests

* tflint

* high level diagram

* make location optional in composer schema

* add composer outputs

* docs

* remove schema docs

* tfdoc

* update service agent encryption composer def for composer 3

* encryption keys

* typo

* typo

* fix security IAM

* inventory

* tflint

* Fix roles and diagram.

* Fix tflint

* Fix test DP.

* Fix test

* Diagrams excalidraw gz

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
 2025-05-03 21:21:38 +00:00
Simone Ruffilli
3c91acd181 Replace all instances of stackdriver.googleapis.com with log+mon (#3022) 
Replace all instances of stackdriver.googleapis.com with logging.googleapis.com  and monitoring.googleapis.com.
Fixes #2932
 2025-04-11 12:04:50 +00:00
Ludovico Magnocavallo
c8e4179f2b Add custom routes for directpath to net-vpc module (#2966) 
* add custom routes for directpath to net-vpc module

* blueprint tests

* blueprint tests

* blueprint tests

* fast tests

* tfdoc

* module examples
 2025-03-19 10:22:47 +00:00
Ludovico Magnocavallo
1a4b298cc9 Project factory additions, project module reuse implementation (#2899) 
* add support for buckets

* add project-level interpolation for own SAs

* docs

* project reuse changes

* fix example

* tfdoc

* update check documentation tool

* fast tests

* blueprints

* typo
 2025-02-15 20:37:45 +01:00
Ludovico Magnocavallo
50ac3a5013 Refactor of FAST resource management and subsequent stages (#2648) 
* untested

* pllan testing

* fix stage 2s

* move providers to their own file

* single-environment stage 3

* fixes and moved blocks

* stage3 factory

* doc

* review comments

* review comments

* tfdoc

* fasts tage 1 tests

* netsec as stage 2

* fix backported roles

* fix backported roles

* tfdoc

* fixes

* fix tag value roles in stage 1

* remove checklist, fix stage 1 tests

* inventory

* Small bugfix

* refactor context tag values

* fix previous merge

* fix previous merge

* fix previous merge

* support short names for top level automation resources, change top level context variable

* fix new top level context

* roll back merge changes to stage 0 outputs

* roll back more merge changes

* linting errors

* tfdoc

* fix tests, roll back merge in tenants stage

* tfdoc

* fix inventory

* optional stage 2 env folders and tag bindings

* tflint

* damn tflint

* damn tflint

* tfdoc

* fix networking tests

* tflint

* fix test inventories

* tfdoc

* use coalesce for project parents

* fix billing role conditions

* fix billing role conditions

* security stage tested (ngw resources need fixing/porting)

* boilerplate

* fix inventory

* stage envs and stage linking script

* initial work on resman docs, update diagram, improve teams folder

* resman README

* fix stage 2 IAM delegation

* remove checklist from bootstrap

* stage 1 tests

* stage 0 1 and 2 tests

* tflint

* tflint

* tfdoc

* GCVE stage refactor (untested)

* GCVE stage refactor (untested)

* GCVE stage 3

* gcve tests

* tflint

* tfdoc

* fix links

* module tests

* stages README

* move network security to stage 2

* network security tests

* replace stage links in README files

* minimal netsec stage refactor

* use factory for iac org policies, add configurable drs org policy for iac

* test mt stage

* tfdoc

* fix cicd workflows

* fix cicd workflows

* gke-dev stage

* tflint

* remove data platform stage

* exclude provider files via tfdoc opts

* remove data platform tests and links

* fix merge

* fix resman inventory

* boilerplate

* inventory

---------

Co-authored-by: Simone Ruffilli <sruffilli@google.com>
 2024-10-31 16:55:54 +01:00
Julio Castillo
c0bf32e797 Refactor service agent management (#2423) 
* Service agents script

* Service agents update

* WIP

* Update script and terraform

* Fix tests

* Fix linter

* Update docs

* Bring back pf example inventory

* Fix tests

* Fix more tests

* Fix tests

* Use dataclasses for build_service_agents.py

* Remove unneeded field() from build_service_agents

* Re-enable CMEK depends_on in project outputs

* Update tools/requirements.txt

* Enable storage in GCS example projects

* Fix tests

* Add CMEK Service Agents dependencies for services

* Fix typos and data platform cmek

* More typos
 2024-07-23 22:05:38 +02:00
Julio Castillo
3af7e257d2 Add tflint to pipelines (#2220) 
* Fix terraform_deprecated_index

https://github.com/terraform-linters/tflint-ruleset-terraform/blob/v0.5.0/docs/rules/terraform_deprecated_index.md

* Fix terraform_deprecated_interpolation

Reference: https://github.com/terraform-linters/tflint-ruleset-terraform/blob/v0.5.0/docs/rules/terraform_deprecated_interpolation.md

* Fix more indexing

* Remove unused variable

* Enable TFLint for modules

* Add tflint config file

* Fix chdir

* Lint modules

* TFLint fixes

* TFLint

* Fixes binauthz README

* Fixes DNS response policy tests. Restores MIG outputs.

* Fixes other DNS response policy tests.

* Update tests for fast 2-e

* Moar fixed tests

---------

Co-authored-by: Simone Ruffilli <sruffilli@google.com>
 2024-04-17 10:23:48 +02:00
Wiktor Niesiobędzki
67382f53b7 Fix data-platform-minimal blueprint 2024-03-11 11:05:33 +01:00
Julio Castillo
993bef71aa Introduce mandatory OWNERS file for blueprint maintainership (#2131) 
* Delete deprecated/broken blueprints

* Adding OWNERS to all blueprints

* Fix links

* Update OWNERS

---------

Co-authored-by: javiergp <javiergp@users.noreply.github.com>
 2024-03-08 09:40:46 +01:00
simonebruzzechesse
d11c380aec Format python files in blueprints (#2079) 
* format python files in blueprints
* update check on blueprints python code
* update python linter in CI workflow
 2024-02-15 09:37:49 +01:00
Wiktor Niesiobędzki
9ff67fdaf0 Add DLP Service Agent role 2023-11-20 15:34:27 +01:00
ddaluka
543ea6e7f3 Fix/dlpagent (#1868) 
Create DLP Service Account on service activation.
 2023-11-20 14:11:01 +01:00
lcaggio
a0ae43fc6f [Minimal Data Platform] Fix Landing and curated IAM (#1832) 
* Fix IAM on Minimal DP

* fix
 2023-11-01 17:53:06 +01:00
Simone Ruffilli
6d89b88149 versions.tf maintenance + copyright notice bump (#1782) 
* Bump copyright notice to 2023

* Delete versions.tf on blueprints

* Pin provider to major version 5

* Remove comment

* Fix lint

* fix bq-ml blueprint readme

---------

Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
Co-authored-by: Julio Castillo <jccb@google.com>
 2023-10-20 18:17:47 +02:00
Julio Castillo
dfc5023e0b Make deletion protection consistent across all modules (#1735) 
* Expose deletion_protection in GKE modules

* Make deletion protection consistent across all modules

* Add deletion_protection option to blueprints

* Fix blueprints tests

* Fix types

* Update READMEs

* Fix dp readme

* Fix cmek blueprint default deletion_protection

* Fix blueprints tests
 2023-10-05 17:31:07 +02:00
lcaggio
de883957fe Minimal Data Platform - Fix (#1730) 
* Fix

* Fix README

* Fix data platform minimal tests

---------

Co-authored-by: Julio Castillo <jccb@google.com>
 2023-10-04 12:15:51 +02:00
lcaggio
79723f9ce1 Add lineage on Minimal Data Platform blueprint (#1679) 
Add lineage on Minimal Data Platform blueprint
 2023-09-14 17:52:19 +02:00
Pierre Formont
0e7cfc8b29 add AIRFLOW_VAR_ prefix to environment variables in data-platform blueprints (#1651) 
* add AIRFLOW_VAR_ prefix to env vars in minimal blueprint

* update DAGs for env vars in minimal blueprint

* add AIRFLOW_VAR_ prefix to env vars in foundation blueprint

* update DAGs for env vars in foundation blueprint

* apply tf linting

---------

Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com>
Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
 2023-09-08 07:38:28 +00:00
Ludovico Magnocavallo
819894d2ba IAM interface refactor (#1595) 
* IAM modules refactor proposal

* policy

* subheading

* Update 20230816-iam-refactor.md

* log Julio's +1

* data-catalog-policy-tag

* dataproc

* dataproc

* folder

* folder

* folder

* folder

* project

* better filtering in test examples

* project

* folder

* folder

* organization

* fix variable descriptions

* kms

* net-vpc

* dataplex-datascan

* modules/iam-service-account

* modules/source-repository/

* blueprints/cloud-operations/vm-migration/

* blueprints/third-party-solutions/wordpress

* dataplex-datascan

* blueprints/cloud-operations/workload-identity-federation

* blueprints/data-solutions/cloudsql-multiregion/

* blueprints/data-solutions/composer-2

* Update 20230816-iam-refactor.md

* Update 20230816-iam-refactor.md

* capture discussion in architectural doc

* update variable names and refactor proposal

* project

* blueprints first round

* folder

* organization

* data-catalog-policy-tag

* re-enable folder inventory

* project module style fix

* dataproc

* source-repository

* source-repository tests

* dataplex-datascan

* dataplex-datascan tests

* net-vpc

* net-vpc test examples

* iam-service-account

* iam-service-account test examples

* kms

* boilerplate

* tfdoc

* fix module tests

* more blueprint fixes

* fix typo in data blueprints

* incomplete refactor of data platform foundations

* tfdoc

* data platform foundation

* refactor data platform foundation iam locals

* remove redundant example test

* shielded folder fix

* fix typo

* project factory

* project factory outputs

* tfdoc

* test workflow: less verbose tests, fix tf version

* re-enable -vv, shorter traceback, fix action version

* ignore github extension warning, re-enable action version

* fast bootstrap IAM, untested

* bootstrap stage IAM fixes

* stage 0 tests

* fast stage 1

* tenant stage 1

* minor changes to fast stage 0 and 1

* fast security stage

* fast mt stage 0

* fast mt stage 0

* fast pf
 2023-08-20 09:44:20 +02:00
Ludovico Magnocavallo
9c75aa469c More module descriptions (#1572) 
* bigquery dataset

* data catalog policy tag

* net-address

* fix data catalog callers

* bigquery dataset views

* fix data catalog callers

* logging bucket

* net vpn ha
 2023-08-06 09:25:45 +00:00
Wiktor Niesiobędzki
0b224a7f55 Return only bucket name of composer, not full url to dags folder 2023-07-10 11:20:51 +02:00
Alejandro Leal
43b3490ef1 Updating a few files for typos 
fast/stages/3-data-platform/dev/README.md
fast/stages/3-data-platform/dev/outputs.tf
CHANGELOG.md
blueprints/data-solutions/data-platform-minimal/README.md
blueprints/data-solutions/data-platform-minimal/outputs.tf
blueprints/data-solutions/data-platform-foundations/README.md
 2023-06-29 21:47:17 -04:00
Keith Harvey
b203198441 Merge branch 'master' into kthhrv-patch-1 2023-06-29 10:10:14 +01:00
Keith Harvey
912497a158 use key/value and update docs 2023-06-29 09:57:23 +01:00
Keith Harvey
f15e2f3509 default to empty map 2023-06-29 09:55:10 +01:00
Keith Harvey
e3e05540de change var name, drop default, update docs 2023-06-29 09:45:29 +01:00
lcaggio
026071209c Minimal Data Platform - Shared VPC (#1475) 
* Fix

* Fix dataproc vpc links

* Add missing networkUser role.

* Fix README.
 2023-06-28 21:58:03 +02:00
Keith Harvey
52ad1fee34 more linting 2023-06-28 14:07:36 +01:00
Keith Harvey
f9742c5405 lint 2023-06-28 14:06:05 +01:00
Keith Harvey
9e8323bd7d Update 02-composer.tf 2023-06-28 13:56:35 +01:00
Keith Harvey
b30efa8bfe Update variables.tf 2023-06-28 13:56:00 +01:00
Keith Harvey
7cd9083fd7 data-platform-minimal - support web_server_network_access_control 2023-06-28 13:28:28 +01:00
lcaggio
099ad03910 Improve Minimal Data Platform Blueprint (#1473) 
* Add SA to access to Curated resources

* Add BQ dataset in the landing project

* Provide example to move data from landing to curated using BQ engine

* Improve diagram
 2023-06-28 09:05:48 +02:00
lcaggio
261ad646a8 Improve Minimal Data Platform blueprint (#1451) 2023-06-20 16:47:15 +00:00
Keith Harvey
dc964411e0 data-platform-minimal - 02-processing.tf typo (#1454) 
error is 
```
│ Error: Unsupported attribute
│ 
│   on ../../../../blueprints/data-solutions/data-platform-minimal/02-processing.tf line 46, in locals:
│   46:     ? var.network_config.subnet_self_links.processingestration
│     ├────────────────
│     │ var.network_config.subnet_self_links is object with 2 attributes
│ 
│ This object does not have an attribute named "processingestration".
╵
```

I'm guessing this should be `processing_transformation` the other option is `processing_composer`
 2023-06-20 15:26:10 +02:00
Keith Harvey
a37b99501e fix typo data-platform-minimal (#1453) 
Fix `container-engine` service agent typo.
 2023-06-20 13:12:00 +02:00
bluPhy
26e5662e84 Some fixes 2023-06-16 00:01:07 +00:00
Julio Castillo
cecbd2072c Fix modules and blueprints tests 2023-05-26 16:38:41 +02:00
lcaggio
40656a23de Minimal Data Platform - Make components optional (#1380) 
make some components optional: Composer and Data Proc history server.
 2023-05-16 14:08:03 +02:00
lcaggio
3cc6c71e96 Add Minimal Data Platform blueprint (#1362) 
Minimal Data Platform blueprint
 2023-05-08 10:25:06 +02:00