diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d5e45b7e..eaaf05b0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ All notable changes to this project will be documented in this file. ## [Unreleased] + - end to end example: `Scheduled Cloud Asset Inventory Export to Bigquery` ## [3.4.0] - 2020-09-24 diff --git a/README.md b/README.md index 811bc8440..d204b6c99 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,7 @@ Currently available examples: - **foundations** - [single level hierarchy](./foundations/environments/) (environments), [multiple level hierarchy](./foundations/business-units/) (business units + environments) - **networking** - [hub and spoke via peering](./networking/hub-and-spoke-peering/), [hub and spoke via VPN](./networking/hub-and-spoke-vpn/), [DNS and Google Private Access for on-premises](./networking/onprem-google-access-dns/), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [ILB as next hop](./networking/ilb-next-hop) - **data solutions** - [GCE/GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms/), [Cloud Storage to Bigquery with Cloud Dataflow](./data-solutions/gcs-to-bq-with-dataflow/) -- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring) - +- **cloud operations** - [Resource tracking and remediation via Cloud Asset feeds](.//cloud-operations/asset-inventory-feed-remediation), [Granular Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Compute Engine quota monitoring](./cloud-operations/quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq) For more information see the README files in the [foundations](./foundations/), [networking](./networking/), [data solutions](./data-solutions/) and [cloud operations](./cloud-operations/) folders. ## Modules diff --git a/cloud-operations/README.md b/cloud-operations/README.md index e1fa1382d..afe4f15c1 100644 --- a/cloud-operations/README.md +++ b/cloud-operations/README.md @@ -10,6 +10,12 @@ The example's feed tracks changes to Google Compute instances, and the Cloud Fun
+## Scheduled Cloud Asset Inventory Export to Bigquery + + This [example](./scheduled-asset-inventory-export-bq) shows how to leverage the [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature, to keep track of your organization's assets over time storing information in Bigquery. Data stored in Bigquery can then be used for different purposes like dashboarding or analysis. + +
+ ## Granular Cloud DNS IAM via Service Directory This [example](./dns-fine-grained-iam) shows how to leverage [Service Directory](https://cloud.google.com/blog/products/networking/introducing-service-directory) and Cloud DNS Service Directory private zones, to implement fine-grained IAM controls on DNS. The example creates a Service Directory namespace, a Cloud DNS private zone that uses it as its authoritative source, service accounts with different levels of permissions, and VMs to test them. diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/README.md b/cloud-operations/scheduled-asset-inventory-export-bq/README.md new file mode 100644 index 000000000..40fe75a5c --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/README.md @@ -0,0 +1,62 @@ +# Scheduled Cloud Asset Inventory Export to Bigquery + +This example shows how to leverage [Cloud Asset Inventory Exporting to Bigquery](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery) feature to keep track of your project wide assets over time storing information in Bigquery. + +The data stored in Bigquery can then be used for different purposes: + +- dashboarding +- analysis + +The example uses export resources at the project level for ease of testing, in actual use a few changes are needed to operate at the resource hierarchy level: + +- the export should be set at the folder or organization level +- the `roles/cloudasset.viewer` on the service account should be set at the folder or organization level + +The resources created in this example are shown in the high level diagram below: + + + +## Prerequisites + +Ensure that you grant your account one of the following roles on your project, folder, or organization: + +- Cloud Asset Viewer role (`roles/cloudasset.viewer`) +- Owner primitive role (`roles/owner`) + +## Running the example + +Clone this repository, specify your variables in a `terraform.tvars` and then go through the following steps to create resources: + +- `terraform init` +- `terraform apply` + +Once done testing, you can clean up resources by running `terraform destroy`. To persist state, check out the `backend.tf.sample` file. + +## Testing the example + +Once resources are created, you can run queries on the data you exported on Bigquery. [Here](https://cloud.google.com/asset-inventory/docs/exporting-to-bigquery#querying_an_asset_snapshot) you can find some example of queries you can run. + +You can also create a dashboard connecting [Datalab](https://datastudio.google.com/) or any other BI tools of your choice to your Bigquery datase. + + +## Variables + +| name | description | type | required | default | +|---|---|:---: |:---:|:---:| +| billing_account | Billing account id used as default for new projects. | string | ✓ | | +| cai_config | Cloud Asset inventory export config. | object({...}) | ✓ | | +| project_id | Project id that references existing project. | string | ✓ | | +| root_node | The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id. | string | ✓ | | +| *bundle_path* | Path used to write the intermediate Cloud Function code bundle. | string | | ./bundle.zip | +| *location* | Appe Engine location used in the example. | string | | europe-west | +| *name* | Arbitrary string used to name created resources. | string | | asset-inventory | +| *project_create* | Create project instead ofusing an existing one. | bool | | true | +| *region* | Compute region used in the example. | string | | europe-west1 | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| bq-dataset | Bigquery instance details. | | +| cloud-function | Cloud Function instance details. | | + diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/backend.tf.sample b/cloud-operations/scheduled-asset-inventory-export-bq/backend.tf.sample new file mode 100644 index 000000000..61572d61a --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/backend.tf.sample @@ -0,0 +1,23 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# set a valid bucket below and rename this file to backend.tf + +terraform { + backend "gcs" { + bucket = "" + prefix = "fabric/operations/inventory" + } +} + diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/cf/main.py b/cloud-operations/scheduled-asset-inventory-export-bq/cf/main.py new file mode 100755 index 000000000..80d629f33 --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/cf/main.py @@ -0,0 +1,112 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +'''Cloud Function module to export data for a given day. + +This module is designed to be plugged in a Cloud Function, attached to Cloud +Scheduler trigger to create a Cloud Asset Inventory Export to BigQuery. + +''' + +import base64 +import datetime +import json +import logging +import os +import warnings + +import click + +from google.api_core.exceptions import GoogleAPIError +from google.cloud import asset_v1 + +import googleapiclient.discovery +import googleapiclient.errors + + +def _configure_logging(verbose=True): + '''Basic logging configuration. + Args: + verbose: enable verbose logging + ''' + level = logging.DEBUG if verbose else logging.INFO + logging.basicConfig(level=level) + warnings.filterwarnings('ignore', r'.*end user credentials.*', UserWarning) + + +@click.command() +@click.option('--project', required=True, help='Project ID') +@click.option('--bq-project', required=True, help='Bigquery project to use.') +@click.option('--bq-dataset', required=True, help='Bigquery dataset to use.') +@click.option('--bq-table', required=True, help='Bigquery table name to use.') +@click.option('--read-time', required=False, help=( + 'Day to take an asset snapshot in \'YYYYMMDD\' format, uses current day ' + ' as default. Export will run at midnight of the specified day.')) +@click.option('--verbose', is_flag=True, help='Verbose output') +def main_cli(project=None, bq_project=None, bq_dataset=None, bq_table=None, + read_time=None, verbose=False): + '''Trigger Cloud Asset inventory export to Bigquery. Data will be stored in + the dataset specified on a dated table with the name specified. + ''' + try: + _main(project, bq_project, bq_dataset, bq_table, read_time, verbose) + except RuntimeError: + logging.exception('exception raised') + + +def main(event, context): + 'Cloud Function entry point.' + try: + data = json.loads(base64.b64decode(event['data']).decode('utf-8')) + print(data) + _main(**data) + # uncomment once https://issuetracker.google.com/issues/155215191 is fixed + # except RuntimeError: + # raise + except Exception: + logging.exception('exception in cloud function entry point') + + +def _main(project=None, bq_project=None, bq_dataset=None, bq_table=None, read_time=None, verbose=False): + 'Module entry point used by cli and cloud function wrappers.' + + _configure_logging(verbose) + if not read_time: + read_time = datetime.datetime.now() + client = asset_v1.AssetServiceClient() + parent = 'projects/%s' % project + content_type = asset_v1.ContentType.RESOURCE + output_config = asset_v1.OutputConfig() + output_config.bigquery_destination.dataset = 'projects/%s/datasets/%s' % ( + bq_project, bq_dataset) + output_config.bigquery_destination.table = '%s_%s' % ( + bq_table, read_time.strftime('%Y%m%d')) + output_config.bigquery_destination.force = True + try: + response = client.export_assets( + request={ + 'parent': parent, + 'read_time': read_time, + 'content_type': content_type, + 'output_config': output_config + } + ) + except (GoogleAPIError, googleapiclient.errors.HttpError) as e: + logging.debug('API Error: %s', e, exc_info=True) + raise RuntimeError( + 'Error fetching Asset Inventory entries (project: %s)' % parent, e) + + +if __name__ == '__main__': + main_cli() diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/cf/requirements.txt b/cloud-operations/scheduled-asset-inventory-export-bq/cf/requirements.txt new file mode 100644 index 000000000..6e893cc33 --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/cf/requirements.txt @@ -0,0 +1,4 @@ +Click>=7.0 +google-api-python-client>=1.10.1 +google-cloud-monitoring>=1.1.0 +google-cloud-asset \ No newline at end of file diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/diagram.png b/cloud-operations/scheduled-asset-inventory-export-bq/diagram.png new file mode 100644 index 000000000..b91591042 Binary files /dev/null and b/cloud-operations/scheduled-asset-inventory-export-bq/diagram.png differ diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/main.tf b/cloud-operations/scheduled-asset-inventory-export-bq/main.tf new file mode 100644 index 000000000..f1f07aea7 --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/main.tf @@ -0,0 +1,134 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +############################################################################### +# Projects # +############################################################################### +module "project" { + source = "../../modules/project" + name = var.project_id + parent = var.root_node + billing_account = var.billing_account + project_create = var.project_create + services = [ + "bigquery.googleapis.com", + "cloudasset.googleapis.com", + "compute.googleapis.com", + "cloudfunctions.googleapis.com", + "cloudbuild.googleapis.com", + "cloudscheduler.googleapis.com", + "pubsub.googleapis.com" + ] +} + +module "service-account" { + source = "../../modules/iam-service-accounts" + project_id = module.project.project_id + names = ["${var.name}-cf"] + iam_project_roles = { + (var.project_id) = ["roles/cloudasset.viewer"] + } +} + +############################################################################### +# Pub/Sub # +############################################################################### +module "pubsub" { + source = "../../modules/pubsub" + project_id = module.project.project_id + name = var.name + subscriptions = { + "${var.name}-default" = null + } + # the Cloud Scheduler robot service account already has pubsub.topics.publish + # at the project level via roles/cloudscheduler.serviceAgent +} + +############################################################################### +# Cloud Function # +############################################################################### +module "cf" { + source = "../../modules/cloud-function" + project_id = module.project.project_id + name = var.name + bucket_name = "${var.name}-${random_pet.random.id}" + bucket_config = { + location = var.region + lifecycle_delete_age = null + } + bundle_config = { + source_dir = "cf" + output_path = var.bundle_path + } + service_account = module.service-account.email + trigger_config = { + event = "google.pubsub.topic.publish" + resource = module.pubsub.topic.id + retry = null + } +} + +resource "random_pet" "random" { + length = 1 +} + +############################################################################### +# Cloud Scheduler # +############################################################################### +resource "google_app_engine_application" "app" { + project = module.project.project_id + location_id = var.location +} + +resource "google_cloud_scheduler_job" "job" { + project = google_app_engine_application.app.project + region = var.region + name = "test-job" + description = "test http job" + schedule = "* 9 * * 1" + time_zone = "Etc/UTC" + + pubsub_target { + attributes = {} + topic_name = module.pubsub.topic.id + data = base64encode(jsonencode({ + project = module.project.project_id + bq_project = module.project.project_id + bq_dataset = var.cai_config.bq_dataset + bq_table = var.cai_config.bq_table + })) + } +} + +############################################################################### +# Bigquery # +############################################################################### +module "bq" { + source = "../../modules/bigquery-dataset" + project_id = module.project.project_id + id = var.cai_config.bq_dataset + access_roles = { + owner = { role = "OWNER", type = "user_by_email" } + } + access_identities = { + owner = module.service-account.email + } + options = { + default_table_expiration_ms = null + default_partition_expiration_ms = null + delete_contents_on_destroy = true + } +} diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/outputs.tf b/cloud-operations/scheduled-asset-inventory-export-bq/outputs.tf new file mode 100644 index 000000000..fa07eea03 --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/outputs.tf @@ -0,0 +1,25 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "bq-dataset" { + description = "Bigquery instance details." + value = module.bq.dataset +} + +output "cloud-function" { + description = "Cloud Function instance details." + value = module.cf.function +} diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf b/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf new file mode 100644 index 000000000..80e5be855 --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/variables.tf @@ -0,0 +1,69 @@ +/** + * Copyright 2020 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "billing_account" { + description = "Billing account id used as default for new projects." + type = string +} + +variable "bundle_path" { + description = "Path used to write the intermediate Cloud Function code bundle." + type = string + default = "./bundle.zip" +} + +variable "cai_config" { + description = "Cloud Asset inventory export config." + type = object({ + bq_dataset = string + bq_table = string + }) +} + +variable "location" { + description = "Appe Engine location used in the example." + type = string + default = "europe-west" +} + + +variable "name" { + description = "Arbitrary string used to name created resources." + type = string + default = "asset-inventory" +} + +variable "project_create" { + description = "Create project instead ofusing an existing one." + type = bool + default = true +} + +variable "project_id" { + description = "Project id that references existing project." + type = string +} + +variable "region" { + description = "Compute region used in the example." + type = string + default = "europe-west1" +} + +variable "root_node" { + description = "The resource name of the parent Folder or Organization. Must be of the form folders/folder_id or organizations/org_id." + type = string +} diff --git a/cloud-operations/scheduled-asset-inventory-export-bq/versions.tf b/cloud-operations/scheduled-asset-inventory-export-bq/versions.tf new file mode 100644 index 000000000..057095c0f --- /dev/null +++ b/cloud-operations/scheduled-asset-inventory-export-bq/versions.tf @@ -0,0 +1,17 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +terraform { + required_version = ">= 0.12.6" +}