kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into fast-dev

Browse Source
This commit is contained in:
Ludo
2024-11-14 13:57:17 +01:00
parent 721e7689b4 27ee179792
commit 3acbf59c58
8 changed files with 240 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
modules/pubsub/README.md
View File

@@ -2,9 +2,21 @@
This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.
## Examples
<!-- BEGIN TOC -->
- [Simple topic with IAM](#simple-topic-with-iam)
- [Topic with schema](#topic-with-schema)
- [Subscriptions](#subscriptions)
- [Push subscriptions](#push-subscriptions)
- [BigQuery subscriptions](#bigquery-subscriptions)
- [BigQuery Subscription with service account email](#bigquery-subscription-with-service-account-email)
- [Cloud Storage subscriptions](#cloud-storage-subscriptions)
- [Subscriptions with IAM](#subscriptions-with-iam)
- [Variables](#variables)
- [Outputs](#outputs)
- [Fixtures](#fixtures)
<!-- END TOC -->
### Simple topic with IAM
## Simple topic with IAM
```hcl
module "pubsub" {
@@ -19,7 +31,7 @@ module "pubsub" {
# tftest modules=1 resources=3 inventory=simple.yaml e2e
```
### Topic with schema
## Topic with schema
```hcl
module "topic_with_schema" {
@@ -51,7 +63,7 @@ module "topic_with_schema" {
# tftest modules=1 resources=2 inventory=schema.yaml e2e
```
### Subscriptions
## Subscriptions
Subscriptions are defined with the `subscriptions` variable, allowing optional configuration of per-subscription defaults. Push subscriptions need extra configuration, shown in the following example.
@@ -72,7 +84,7 @@ module "pubsub" {
# tftest modules=1 resources=3 inventory=subscriptions.yaml e2e
```
### Push subscriptions
## Push subscriptions
Push subscriptions need extra configuration in the `push_configs` variable.
@@ -92,7 +104,7 @@ module "pubsub" {
# tftest modules=1 resources=2 inventory=push-subscription.yaml e2e
```
### BigQuery subscriptions
## BigQuery subscriptions
BigQuery subscriptions need extra configuration in the `bigquery_subscription_configs` variable.
@@ -115,7 +127,36 @@ module "pubsub" {
# tftest modules=2 resources=5 fixtures=fixtures/bigquery-dataset.tf inventory=bigquery-subscription.yaml e2e
```
### Cloud Storage subscriptions
## BigQuery Subscription with service account email
BigQuery subscription example configuration with service account email.
```hcl
module "iam-service-account" {
source = "./fabric/modules/iam-service-account"
project_id = var.project_id
name = "fixture-service-account"
}
module "pubsub" {
source = "./fabric/modules/pubsub"
project_id = var.project_id
name = "my-topic"
subscriptions = {
test-bigquery-with-service-account = {
bigquery = {
table = "${module.bigquery-dataset.tables["my_table"].project}:${module.bigquery-dataset.tables["my_table"].dataset_id}.${module.bigquery-dataset.tables["my_table"].table_id}"
use_table_schema = true
write_metadata = false
service_account_email = module.iam-service-account.email
}
}
}
}
# tftest modules=3 resources=6 fixtures=fixtures/bigquery-dataset.tf inventory=bigquery-subscription-with-service-account.yaml e2e
```
## Cloud Storage subscriptions
Cloud Storage subscriptions need extra configuration in the `cloud_storage_subscription_configs` variable.
@@ -142,7 +183,7 @@ module "pubsub" {
# tftest modules=2 resources=4 fixtures=fixtures/gcs.tf inventory=cloud-storage-subscription.yaml e2e
```
### Subscriptions with IAM
## Subscriptions with IAM
```hcl
module "pubsub" {
@@ -174,7 +215,7 @@ module "pubsub" {
| [message_retention_duration](variables.tf#L67) | Minimum duration to retain a message after it is published to the topic. | <code>string</code> | | <code>null</code> |
| [regions](variables.tf#L83) | List of regions used to set persistence policy. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [schema](variables.tf#L90) | Topic schema. If set, all messages in this topic should follow this schema. | <code title="object&#40;&#123;&#10; definition &#61; string&#10; msg_encoding &#61; optional&#40;string, &#34;ENCODING_UNSPECIFIED&#34;&#41;&#10; schema_type &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [subscriptions](variables.tf#L100) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; labels &#61; optional&#40;map&#40;string&#41;&#41;&#10; ack_deadline_seconds &#61; optional&#40;number&#41;&#10; message_retention_duration &#61; optional&#40;string&#41;&#10; retain_acked_messages &#61; optional&#40;bool, false&#41;&#10; expiration_policy_ttl &#61; optional&#40;string&#41;&#10; filter &#61; optional&#40;string&#41;&#10; enable_message_ordering &#61; optional&#40;bool, false&#41;&#10; enable_exactly_once_delivery &#61; optional&#40;bool, false&#41;&#10; dead_letter_policy &#61; optional&#40;object&#40;&#123;&#10; topic &#61; string&#10; max_delivery_attempts &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; retry_policy &#61; optional&#40;object&#40;&#123;&#10; minimum_backoff &#61; optional&#40;number&#41;&#10; maximum_backoff &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; bigquery &#61; optional&#40;object&#40;&#123;&#10; table &#61; string&#10; use_table_schema &#61; optional&#40;bool, false&#41;&#10; use_topic_schema &#61; optional&#40;bool, false&#41;&#10; write_metadata &#61; optional&#40;bool, false&#41;&#10; drop_unknown_fields &#61; optional&#40;bool, false&#41;&#10; &#125;&#41;&#41;&#10; cloud_storage &#61; optional&#40;object&#40;&#123;&#10; bucket &#61; string&#10; filename_prefix &#61; optional&#40;string&#41;&#10; filename_suffix &#61; optional&#40;string&#41;&#10; max_duration &#61; optional&#40;string&#41;&#10; max_bytes &#61; optional&#40;number&#41;&#10; avro_config &#61; optional&#40;object&#40;&#123;&#10; write_metadata &#61; optional&#40;bool, false&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10; push &#61; optional&#40;object&#40;&#123;&#10; endpoint &#61; string&#10; attributes &#61; optional&#40;map&#40;string&#41;&#41;&#10; no_wrapper &#61; optional&#40;bool, false&#41;&#10; oidc_token &#61; optional&#40;object&#40;&#123;&#10; audience &#61; optional&#40;string&#41;&#10; service_account_email &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#10;&#10; iam &#61; optional&#40;map&#40;list&#40;string&#41;&#41;, &#123;&#125;&#41;&#10; iam_bindings &#61; optional&#40;map&#40;object&#40;&#123;&#10; members &#61; list&#40;string&#41;&#10; role &#61; string&#10; condition &#61; optional&#40;object&#40;&#123;&#10; expression &#61; string&#10; title &#61; string&#10; description &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;, &#123;&#125;&#41;&#10; iam_bindings_additive &#61; optional&#40;map&#40;object&#40;&#123;&#10; member &#61; string&#10; role &#61; string&#10; condition &#61; optional&#40;object&#40;&#123;&#10; expression &#61; string&#10; title &#61; string&#10; description &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;, &#123;&#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [subscriptions](variables.tf#L100) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; ack_deadline_seconds &#61; optional&#40;number&#41;&#10; enable_exactly_once_delivery &#61; optional&#40;bool, false&#41;&#10; enable_message_ordering &#61; optional&#40;bool, false&#41;&#10; expiration_policy_ttl &#61; optional&#40;string&#41;&#10; filter &#61; optional&#40;string&#41;&#10; iam &#61; optional&#40;map&#40;list&#40;string&#41;&#41;, &#123;&#125;&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;&#41;&#10; message_retention_duration &#61; optional&#40;string&#41;&#10; retain_acked_messages &#61; optional&#40;bool, false&#41;&#10; bigquery &#61; optional&#40;object&#40;&#123;&#10; table &#61; string&#10; drop_unknown_fields &#61; optional&#40;bool, false&#41;&#10; service_account_email &#61; optional&#40;string&#41;&#10; use_table_schema &#61; optional&#40;bool, false&#41;&#10; use_topic_schema &#61; optional&#40;bool, false&#41;&#10; write_metadata &#61; optional&#40;bool, false&#41;&#10; &#125;&#41;&#41;&#10; cloud_storage &#61; optional&#40;object&#40;&#123;&#10; bucket &#61; string&#10; filename_prefix &#61; optional&#40;string&#41;&#10; filename_suffix &#61; optional&#40;string&#41;&#10; max_duration &#61; optional&#40;string&#41;&#10; max_bytes &#61; optional&#40;number&#41;&#10; avro_config &#61; optional&#40;object&#40;&#123;&#10; write_metadata &#61; optional&#40;bool, false&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10; dead_letter_policy &#61; optional&#40;object&#40;&#123;&#10; topic &#61; string&#10; max_delivery_attempts &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; iam_bindings &#61; optional&#40;map&#40;object&#40;&#123;&#10; members &#61; list&#40;string&#41;&#10; role &#61; string&#10; condition &#61; optional&#40;object&#40;&#123;&#10; expression &#61; string&#10; title &#61; string&#10; description &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;, &#123;&#125;&#41;&#10; iam_bindings_additive &#61; optional&#40;map&#40;object&#40;&#123;&#10; member &#61; string&#10; role &#61; string&#10; condition &#61; optional&#40;object&#40;&#123;&#10; expression &#61; string&#10; title &#61; string&#10; description &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;, &#123;&#125;&#41;&#10; push &#61; optional&#40;object&#40;&#123;&#10; endpoint &#61; string&#10; attributes &#61; optional&#40;map&#40;string&#41;&#41;&#10; no_wrapper &#61; optional&#40;bool, false&#41;&#10; oidc_token &#61; optional&#40;object&#40;&#123;&#10; audience &#61; optional&#40;string&#41;&#10; service_account_email &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10; retry_policy &#61; optional&#40;object&#40;&#123;&#10; minimum_backoff &#61; optional&#40;number&#41;&#10; maximum_backoff &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
## Outputs

23
modules/pubsub/main.tf
View File

@@ -80,8 +80,16 @@ resource "google_pubsub_subscription" "default" {
dynamic "retry_policy" {
for_each = each.value.retry_policy == null ? [] : [""]
content {
maximum_backoff = each.value.retry_policy.maximum_backoff != null ? "${each.value.retry_policy.maximum_backoff}s" : null
minimum_backoff = each.value.retry_policy.minimum_backoff != null ? "${each.value.retry_policy.minimum_backoff}s" : null
maximum_backoff = (
each.value.retry_policy.maximum_backoff != null
? "${each.value.retry_policy.maximum_backoff}s"
: null
)
minimum_backoff = (
each.value.retry_policy.minimum_backoff != null
? "${each.value.retry_policy.minimum_backoff}s"
: null
)
}
}
@@ -103,11 +111,12 @@ resource "google_pubsub_subscription" "default" {
dynamic "bigquery_config" {
for_each = each.value.bigquery == null ? [] : [""]
content {
table = each.value.bigquery.table
use_table_schema = each.value.bigquery.use_table_schema
use_topic_schema = each.value.bigquery.use_topic_schema
write_metadata = each.value.bigquery.write_metadata
drop_unknown_fields = each.value.bigquery.drop_unknown_fields
table = each.value.bigquery.table
use_table_schema = each.value.bigquery.use_table_schema
use_topic_schema = each.value.bigquery.use_topic_schema
write_metadata = each.value.bigquery.write_metadata
drop_unknown_fields = each.value.bigquery.drop_unknown_fields
service_account_email = each.value.bigquery.service_account_email
}
}

56
modules/pubsub/variables.tf
View File

@@ -100,28 +100,22 @@ variable "schema" {
variable "subscriptions" {
description = "Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null."
type = map(object({
labels = optional(map(string))
ack_deadline_seconds = optional(number)
message_retention_duration = optional(string)
retain_acked_messages = optional(bool, false)
enable_exactly_once_delivery = optional(bool, false)
enable_message_ordering = optional(bool, false)
expiration_policy_ttl = optional(string)
filter = optional(string)
enable_message_ordering = optional(bool, false)
enable_exactly_once_delivery = optional(bool, false)
dead_letter_policy = optional(object({
topic = string
max_delivery_attempts = optional(number)
}))
retry_policy = optional(object({
minimum_backoff = optional(number)
maximum_backoff = optional(number)
}))
iam = optional(map(list(string)), {})
labels = optional(map(string))
message_retention_duration = optional(string)
retain_acked_messages = optional(bool, false)
bigquery = optional(object({
table = string
use_table_schema = optional(bool, false)
use_topic_schema = optional(bool, false)
write_metadata = optional(bool, false)
drop_unknown_fields = optional(bool, false)
table = string
drop_unknown_fields = optional(bool, false)
service_account_email = optional(string)
use_table_schema = optional(bool, false)
use_topic_schema = optional(bool, false)
write_metadata = optional(bool, false)
}))
cloud_storage = optional(object({
bucket = string
@@ -133,17 +127,10 @@ variable "subscriptions" {
write_metadata = optional(bool, false)
}))
}))
push = optional(object({
endpoint = string
attributes = optional(map(string))
no_wrapper = optional(bool, false)
oidc_token = optional(object({
audience = optional(string)
service_account_email = string
}))
dead_letter_policy = optional(object({
topic = string
max_delivery_attempts = optional(number)
}))
iam = optional(map(list(string)), {})
iam_bindings = optional(map(object({
members = list(string)
role = string
@@ -162,6 +149,19 @@ variable "subscriptions" {
description = optional(string)
}))
})), {})
push = optional(object({
endpoint = string
attributes = optional(map(string))
no_wrapper = optional(bool, false)
oidc_token = optional(object({
audience = optional(string)
service_account_email = string
}))
}))
retry_policy = optional(object({
minimum_backoff = optional(number)
maximum_backoff = optional(number)
}))
}))
default = {}
nullable = false