bf64a3dfda
* Import Fast from dev repository. > > Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * Import Fast from dev repository. > > Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * merge tools changes * Import Fast from dev repository. > > Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * add bolierplate to validate_schema Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com> * stage 02-security * Import Fast from dev repository. Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * Copy FAST top level README * Copy FAST top level README * TODO list * TODO list * fix linting action to account for fast * remove providers file * add missing boilerplate * update factory README * align examples tfdoc * fast readmes tfdoc * disable markdown link check * really disable markdown link check * update TODO * switch to local module refs in stage0 * replace module refs in 02-sec * Import Fast from dev repository. > > Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * merge tools changes * Import Fast from dev repository. > > Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * add bolierplate to validate_schema Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com> * Import Fast from dev repository. > > Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * stage 02-security * Import Fast from dev repository. Co-authored-by: Julio Castillo <jccb@google.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> * Copy FAST top level README * Copy FAST top level README * TODO list * TODO list * fix linting action to account for fast * remove providers file * add missing boilerplate * update factory README * align examples tfdoc * fast readmes tfdoc * disable markdown link check * really disable markdown link check * update TODO * switch to local module refs in stage0 * replace module refs in 02-sec * Move first draft to fast branch * Fix roles and variables. Add e2e DAG example! * Fix example * Fix KMS * First draft: README * Update README * Add DLP, update README * Update Readme * README * Add todos * Merge master * Merge master * Merge master * Fix and test KMS, Fix and test existing prj (it works also with single prj), Update README * Fix READM and Demo * add on TF files * Remove block comments * simplify service_encryption_keys logic * fix README * Fix TODOs * fix tfdoc description * fix demo README * fix sample files * rename tf files * Fix outputs file name, fix README, remove dependeces on composer resource * Add test. * Fix README. * Initial README update * README review * Fix issues & readme * Fix README * Fix README * Fix test error * Fix test error * Add datacatalog * Fix test, for real? :-) * fix readme * support policy_boolean * split Cloud NAT flag * Fix README. * Fix Shared VPC, first try :-) * Fix tests and resource name * fix tests * fix tests * README refactor * Fix secondary range logic * First commit * Replace existing data platform * Fix secondary range logic * Fix README * Replace DP example tests with the new one. * Fix test module location. * Fix test module location, for real. * Support DataPlatform project in VPC-SC * Fix VPC-SC * Add TODO, VPC-SC * Possible improvement to handle VPC-SC perimeter projects with folder as variable * Add TODO * Fix module path * Initial fix for KMS * Add PubSub encryption * Fix secondary range logic * First commit * Support DataPlatform project in VPC-SC * Fix VPC-SC * Add TODO, VPC-SC * Possible improvement to handle VPC-SC perimeter projects with folder as variable * Add TODO * Fix module path * Initial fix for KMS * Update READMEs * Update README * Fix composer roles and README. * Fix test. * Fixes. * Add DLP documentation link. * Temp commit with errors * Refactor variables * Fix secondary range logic * First commit * Support DataPlatform project in VPC-SC * Fix VPC-SC * Add TODO, VPC-SC * Possible improvement to handle VPC-SC perimeter projects with folder as variable * Add TODO * Fix module path * Initial fix for KMS * rebase * rebase * rebase * Rebase * rebase * Update READMEs * Fixes. * Fix new variables * Fix misconfiguration and tests. * Fix secondary range logic * First commit * Support DataPlatform project in VPC-SC * Fix VPC-SC * Add TODO, VPC-SC * Possible improvement to handle VPC-SC perimeter projects with folder as variable * Add TODO * Fix module path * Initial fix for KMS * rebase * rebase * rebase * Rebase * rebase * Update READMEs * Fixes. * Rebase - Fix secondary range logic * Rebase - First commit * Support DataPlatform project in VPC-SC * Fix VPC-SC * Possible improvement to handle VPC-SC perimeter projects with folder as variable * Initial fix for KMS * Fix secondary range logic * First commit * Support DataPlatform project in VPC-SC * Fix VPC-SC * Fix module path * Initial fix for KMS * Update READMEs * Fixes. * Fix new variables * Revert VPC-SC logic * Fix variable typos * README fixes * Fix Project Name logic * Fix Linting * READEME * update READEME * update READEME * update README * mandatory project creation, refactor * formatting * add TODO for service accounts descriptive name * use project module to assign shared vpc roles * Fix shared-vpc-project module * Fix vpc name and tests * README * update to newer version Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com> Co-authored-by: Simone Ruffilli <sruffilli@google.com> Co-authored-by: Julio Castillo <juliocc@users.noreply.github.com> Co-authored-by: Julio Castillo <jccb@google.com>
GCP Data Services examples
The examples in this folder implement typical data service topologies and end-to-end scenarios, that allow testing specific features like Cloud KMS to encrypt your data, or VPC-SC to mitigate data exfiltration.
They are meant to be used as minimal but complete starting points to create actual infrastructure, and as playgrounds to experiment with specific Google Cloud features.
Examples
GCE and GCS CMEK via centralized Cloud KMS
This example implements CMEK for GCS and GCE, via keys hosted in KMS running in a centralized project. The example shows the basic resources and permissions for the typical use case of application projects implementing encryption at rest via a centrally managed KMS service.
Cloud Storage to Bigquery with Cloud Dataflow with least privileges
This example implements resources required to run GCS to BigQuery Dataflow pipelines. The solution rely on a set of Services account created with the least privileges principle.
Data Platform Foundations
This example implements a robust and flexible Data Foundation on GCP that provides opinionated defaults, allowing customers to build and scale out additional data pipelines quickly and reliably.