From f3f1ce9051ec6e5f10a996caf65af8220329df8f Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Tue, 30 Jun 2020 20:41:47 +0200 Subject: [PATCH] Update README.md --- .../gcs-to-bq-with-dataflow/README.md | 62 +++++++++---------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/data-solutions/gcs-to-bq-with-dataflow/README.md b/data-solutions/gcs-to-bq-with-dataflow/README.md index b4b1a9de1..23a4470d7 100644 --- a/data-solutions/gcs-to-bq-with-dataflow/README.md +++ b/data-solutions/gcs-to-bq-with-dataflow/README.md @@ -3,9 +3,9 @@ This example creates the infrastructure needed to run a [Cloud Dataflow](https://cloud.google.com/dataflow) pipeline to import data from [GCS](https://cloud.google.com/storage) to [Bigquery](https://cloud.google.com/bigquery). The solution will use: - - internal IP for GCE and Dataflow instances - - CMEK encription for GCS bucket, GCE instances, DataFlow instances and BigQuery Tables - - Cloud NAT to let resource comunicate to internet to run updates and packages installation + - internal IPs for GCE and Dataflow instances + - CMEK encription for GCS bucket, GCE instances, DataFlow instances and BigQuery tables + - Cloud NAT to let resources comunicate to the Internet, run system updates, and install packages The example is designed to match real-world use cases with a minimum amount of resources. It can be used as a starting point for more complex scenarios. @@ -42,33 +42,6 @@ This sample creates several distinct groups of resources: - One dataset encrypted with a CMEK Cryptokey hosted in Cloud KMS - Two tables encrypted with a CMEK Cryptokey hosted in Cloud KMS - -## Variables - -| name | description | type | required | default | -|---|---|:---: |:---:|:---:| -| billing_account | Billing account id used as default for new projects. | string | ✓ | | -| project_kms_name | Name for the new KMS Project. | string | ✓ | | -| project_service_name | Name for the new Service Project. | string | ✓ | | -| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | string | ✓ | | -| *location* | The location where resources will be deployed. | string | | europe | -| *region* | The region where resources will be deployed. | string | | europe-west1 | -| *ssh_source_ranges* | IP CIDR ranges that will be allowed to connect via SSH to the onprem instance. | list(string) | | ["0.0.0.0/0"] | -| *vpc_ip_cidr_range* | Ip range used in the subnet deployef in the Service Project. | string | | 10.0.0.0/20 | -| *vpc_name* | Name of the VPC created in the Service Project. | string | | local | -| *vpc_subnet_name* | Name of the subnet created in the Service Project. | string | | subnet | -| *zone* | The zone where resources will be deployed. | string | | europe-west1-b | - -## Outputs - -| name | description | sensitive | -|---|---|:---:| -| bq_tables | Bigquery Tables. | | -| buckets | GCS Bucket Cloud KMS crypto keys. | | -| projects | Project ids. | | -| vm | GCE VMs. | | - - ## Test your environment with Cloud Dataflow You can now connect to the GCE instance with the following command: @@ -135,4 +108,31 @@ gs://my-bucket/person.csv \ schema_bq_import.json ``` -You can check data imported into Google BigQuery from the Google Cloud Console UI. \ No newline at end of file +You can check data imported into Google BigQuery from the Google Cloud Console UI. + + +## Variables + +| name | description | type | required | default | +|---|---|:---: |:---:|:---:| +| billing_account | Billing account id used as default for new projects. | string | ✓ | | +| project_kms_name | Name for the new KMS Project. | string | ✓ | | +| project_service_name | Name for the new Service Project. | string | ✓ | | +| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | string | ✓ | | +| *location* | The location where resources will be deployed. | string | | europe | +| *region* | The region where resources will be deployed. | string | | europe-west1 | +| *ssh_source_ranges* | IP CIDR ranges that will be allowed to connect via SSH to the onprem instance. | list(string) | | ["0.0.0.0/0"] | +| *vpc_ip_cidr_range* | Ip range used in the subnet deployef in the Service Project. | string | | 10.0.0.0/20 | +| *vpc_name* | Name of the VPC created in the Service Project. | string | | local | +| *vpc_subnet_name* | Name of the subnet created in the Service Project. | string | | subnet | +| *zone* | The zone where resources will be deployed. | string | | europe-west1-b | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| bq_tables | Bigquery Tables. | | +| buckets | GCS Bucket Cloud KMS crypto keys. | | +| projects | Project ids. | | +| vm | GCE VMs. | | +