hunfabric/modules/pubsub

Google Cloud Pub/Sub Module

This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.

• Simple topic with IAM
• Topic with schema
• Subscriptions
• Push subscriptions
• BigQuery subscriptions
• BigQuery Subscription with service account email
• Cloud Storage subscriptions
• Subscriptions with IAM
• Variables
• Outputs
• Fixtures

Simple topic with IAM

IAM is managed via several variables that implement different features and levels of control:

• iam and iam_by_principals configure authoritative bindings that manage individual roles exclusively, and are internally merged
• iam_bindings configure authoritative bindings with optional support for conditions, and are not internally merged with the previous two variables
• iam_bindings_additive configure additive bindings via individual role/member pairs with optional support conditions

The authoritative and additive approaches can be used together, provided different roles are managed by each. Some care must also be taken with the iam_by_principals variable to ensure that variable keys are static values, so that Terraform is able to compute the dependency graph.

Refer to the project module for examples of the IAM interface. IAM also supports variable interpolation for both roles and principals and for the foreign resources where the service account is the principal, via the respective attributes in the var.context variable. Basic usage is shown in the example below.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  context = {
    iam_principals = {
      mygroup = "group:${var.group_email}"
    }
  }
  iam = {
    "roles/pubsub.viewer"     = ["$iam_principals:mygroup"]
    "roles/pubsub.subscriber" = ["serviceAccount:${var.service_account.email}"]
  }
}
# tftest modules=1 resources=3 inventory=simple.yaml e2e

Topic with schema

module "topic_with_schema" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  schema = {
    msg_encoding = "JSON"
    schema_type  = "AVRO"
    definition = jsonencode({
      "type" = "record",
      "name" = "Avro",
      "fields" = [{
        "name" = "StringField",
        "type" = "string"
        },
        {
          "name" = "FloatField",
          "type" = "float"
        },
        {
          "name" = "BooleanField",
          "type" = "boolean"
        },
      ]
    })
  }
}
# tftest modules=1 resources=2 inventory=schema.yaml e2e

Subscriptions

Subscriptions are defined with the subscriptions variable, allowing optional configuration of per-subscription defaults. Push subscriptions need extra configuration, shown in the following example.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  labels     = { test = "default" }
  subscriptions = {
    test-pull = {}
    test-pull-override = {
      labels                = { test = "override" }
      retain_acked_messages = true
    }
  }
}
# tftest modules=1 resources=3 inventory=subscriptions.yaml e2e

Push subscriptions

Push subscriptions need extra configuration in the push_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-push = {
      push = {
        endpoint = "https://example.com/foo"
      }
    }
  }
}
# tftest modules=1 resources=2 inventory=push-subscription.yaml e2e

BigQuery subscriptions

BigQuery subscriptions need extra configuration in the bigquery_subscription_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-bigquery = {
      bigquery = {
        table               = "${module.bigquery-dataset.tables["my_table"].project}:${module.bigquery-dataset.tables["my_table"].dataset_id}.${module.bigquery-dataset.tables["my_table"].table_id}"
        use_topic_schema    = true
        write_metadata      = false
        drop_unknown_fields = true
      }
    }
  }
}
# tftest modules=2 resources=5 fixtures=fixtures/bigquery-dataset.tf inventory=bigquery-subscription.yaml e2e

BigQuery Subscription with service account email

BigQuery subscription example configuration with service account email.

module "iam-service-account" {
  source     = "./fabric/modules/iam-service-account"
  project_id = var.project_id
  name       = "fixture-service-account"
  iam_project_roles = {
    "${var.project_id}" = [
      "roles/bigquery.dataEditor",
    ]
  }
}

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-bigquery-with-service-account = {
      bigquery = {
        table                 = "${module.bigquery-dataset.tables["my_table"].project}:${module.bigquery-dataset.tables["my_table"].dataset_id}.${module.bigquery-dataset.tables["my_table"].table_id}"
        use_table_schema      = true
        write_metadata        = false
        service_account_email = module.iam-service-account.email
      }
    }
  }
  depends_on = [
    module.iam-service-account # wait for IAM grants to finish
  ]
}
# tftest fixtures=fixtures/bigquery-dataset.tf inventory=bigquery-subscription-with-service-account.yaml e2e

Cloud Storage subscriptions

Cloud Storage subscriptions need extra configuration in the cloud_storage_subscription_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-cloudstorage = {
      cloud_storage = {
        bucket          = module.gcs.name
        filename_prefix = var.prefix
        filename_suffix = "test_suffix"
        max_duration    = "100s"
        max_bytes       = 1000
        avro_config = {
          write_metadata = true
        }
      }
    }
  }
}
# tftest modules=2 resources=4 fixtures=fixtures/gcs.tf inventory=cloud-storage-subscription.yaml e2e

Subscriptions with IAM

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-1 = {
      iam = {
        "roles/pubsub.subscriber" = ["serviceAccount:${var.service_account.email}"]
      }
    }
  }
}
# tftest modules=1 resources=3 inventory=subscription-iam.yaml e2e

Variables

name	description	type	required	default
name	PubSub topic name.	string	✓
project_id	Project used for resources.	string	✓
context	Context-specific interpolations.	object({…})		{}
iam	IAM bindings for topic in {ROLE => [MEMBERS]} format.	map(list(string))		{}
iam_bindings	Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary.	map(object({…}))		{}
iam_bindings_additive	Keyring individual additive IAM bindings. Keys are arbitrary.	map(object({…}))		{}
iam_by_principals	Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the iam variable.	map(list(string))		{}
kms_key	KMS customer managed encryption key.	string		null
labels	Labels.	map(string)		{}
message_retention_duration	Minimum duration to retain a message after it is published to the topic.	string		null
regions	List of regions used to set persistence policy.	list(string)		[]
schema	Topic schema. If set, all messages in this topic should follow this schema.	object({…})		null
subscriptions	Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null.	map(object({…}))		{}

Outputs

name	description	sensitive
id	Fully qualified topic id.
schema	Schema resource.
schema_id	Schema resource id.
subscription_id	Subscription ids.
subscriptions	Subscription resources.
topic	Topic resource.

Fixtures

• bigquery-dataset.tf
• gcs.tf