kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

bigquery-dataset: fix issues (#3425)

Browse Source 
* bigquery-dataset: add missing return_type to routines, fix view schema
encode, add dependency between routines and views

* Fix test.

---------

Co-authored-by: Julio Castillo <jccb@google.com>
This commit is contained in:
Taneli Leppä
2025-10-16 15:12:23 +02:00
committed by GitHub
parent 5be136e2ea
commit 7c6211a494
4 changed files with 10 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
modules/bigquery-dataset/README.md
View File

@@ -370,10 +370,10 @@ module "bigquery-dataset" {
| [location](variables.tf#L122) | Dataset location. | <code>string</code> | | <code>&#34;EU&#34;</code> |
| [materialized_views](variables.tf#L128) | Materialized views definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; allow_non_incremental_definition &#61; optional&#40;bool&#41;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; enable_refresh &#61; optional&#40;bool&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; refresh_interval_ms &#61; optional&#40;bool&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; type &#61; string&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [options](variables.tf#L161) | Dataset options. | <code title="object&#40;&#123;&#10; default_collation &#61; optional&#40;string&#41;&#10; default_table_expiration_ms &#61; optional&#40;number&#41;&#10; default_partition_expiration_ms &#61; optional&#40;number&#41;&#10; delete_contents_on_destroy &#61; optional&#40;bool, false&#41;&#10; is_case_insensitive &#61; optional&#40;bool&#41;&#10; max_time_travel_hours &#61; optional&#40;number, 168&#41;&#10; storage_billing_model &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
| [routines](variables.tf#L180) | Routine definitions. | <code title="map&#40;object&#40;&#123;&#10; description &#61; optional&#40;string&#41;&#10; routine_type &#61; string&#10; language &#61; optional&#40;string&#41;&#10; definition_body &#61; string&#10; imported_libraries &#61; optional&#40;list&#40;string&#41;&#41;&#10; determinism_level &#61; optional&#40;string&#41;&#10; data_governance_type &#61; optional&#40;string&#41;&#10; return_table_type &#61; optional&#40;string&#41;&#10; arguments &#61; optional&#40;map&#40;object&#40;&#123;&#10; argument_kind &#61; optional&#40;string&#41;&#10; mode &#61; optional&#40;string&#41;&#10; data_type &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;, &#123;&#125;&#41;&#10; spark_options &#61; optional&#40;object&#40;&#123;&#10; archive_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; connection &#61; string&#10; container_image &#61; optional&#40;string&#41;&#10; file_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; jar_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; main_file_uri &#61; optional&#40;string&#41;&#10; main_class &#61; optional&#40;string&#41;&#10; properties &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; py_file_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; runtime_version &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; remote_function_options &#61; optional&#40;object&#40;&#123;&#10; connection &#61; string&#10; endpoint &#61; optional&#40;string&#41;&#10; max_batching_rows &#61; optional&#40;string&#41;&#10; user_defined_context &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [tables](variables.tf#L218) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map&#40;object&#40;&#123;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; schema &#61; optional&#40;string&#41;&#10; external_data_configuration &#61; optional&#40;object&#40;&#123;&#10; autodetect &#61; bool&#10; source_uris &#61; list&#40;string&#41;&#10; avro_logical_types &#61; optional&#40;bool&#41;&#10; compression &#61; optional&#40;string&#41;&#10; connection_id &#61; optional&#40;string&#41;&#10; file_set_spec_type &#61; optional&#40;string&#41;&#10; ignore_unknown_values &#61; optional&#40;bool&#41;&#10; metadata_cache_mode &#61; optional&#40;string&#41;&#10; object_metadata &#61; optional&#40;string&#41;&#10; json_options_encoding &#61; optional&#40;string&#41;&#10; reference_file_schema_uri &#61; optional&#40;string&#41;&#10; schema &#61; optional&#40;string&#41;&#10; source_format &#61; optional&#40;string&#41;&#10; max_bad_records &#61; optional&#40;number&#41;&#10; csv_options &#61; optional&#40;object&#40;&#123;&#10; quote &#61; string&#10; allow_jagged_rows &#61; optional&#40;bool&#41;&#10; allow_quoted_newlines &#61; optional&#40;bool&#41;&#10; encoding &#61; optional&#40;string&#41;&#10; field_delimiter &#61; optional&#40;string&#41;&#10; skip_leading_rows &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; google_sheets_options &#61; optional&#40;object&#40;&#123;&#10; range &#61; optional&#40;string&#41;&#10; skip_leading_rows &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; hive_partitioning_options &#61; optional&#40;object&#40;&#123;&#10; mode &#61; optional&#40;string&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; source_uri_prefix &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; parquet_options &#61; optional&#40;object&#40;&#123;&#10; enum_as_string &#61; optional&#40;bool&#41;&#10; enable_list_inference &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10;&#10;&#10; &#125;&#41;&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; encryption_key &#61; optional&#40;string&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; max_staleness &#61; optional&#40;string&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; type &#61; string&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10; table_constraints &#61; optional&#40;object&#40;&#123;&#10; primary_key_columns &#61; optional&#40;list&#40;string&#41;&#41;&#10; foreign_keys &#61; optional&#40;object&#40;&#123;&#10; referenced_table &#61; object&#40;&#123;&#10; project_id &#61; string&#10; dataset_id &#61; string&#10; table_id &#61; string&#10; &#125;&#41;&#10; column_references &#61; object&#40;&#123;&#10; referencing_column &#61; string&#10; referenced_column &#61; string&#10; &#125;&#41;&#10; name &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [tag_bindings](variables.tf#L303) | Tag bindings for this dataset, in key => tag value id format. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [views](variables.tf#L310) | View definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; use_legacy_sql &#61; optional&#40;bool&#41;&#10; schema &#61; optional&#40;list&#40;object&#40;&#123;&#10; name &#61; string&#10; type &#61; string&#10; description &#61; string&#10; mode &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [routines](variables.tf#L180) | Routine definitions. | <code title="map&#40;object&#40;&#123;&#10; description &#61; optional&#40;string&#41;&#10; routine_type &#61; string&#10; language &#61; optional&#40;string&#41;&#10; definition_body &#61; string&#10; imported_libraries &#61; optional&#40;list&#40;string&#41;&#41;&#10; determinism_level &#61; optional&#40;string&#41;&#10; data_governance_type &#61; optional&#40;string&#41;&#10; return_type &#61; optional&#40;string&#41;&#10; return_table_type &#61; optional&#40;string&#41;&#10; arguments &#61; optional&#40;map&#40;object&#40;&#123;&#10; argument_kind &#61; optional&#40;string&#41;&#10; mode &#61; optional&#40;string&#41;&#10; data_type &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;, &#123;&#125;&#41;&#10; spark_options &#61; optional&#40;object&#40;&#123;&#10; archive_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; connection &#61; string&#10; container_image &#61; optional&#40;string&#41;&#10; file_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; jar_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; main_file_uri &#61; optional&#40;string&#41;&#10; main_class &#61; optional&#40;string&#41;&#10; properties &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; py_file_uris &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10; runtime_version &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; remote_function_options &#61; optional&#40;object&#40;&#123;&#10; connection &#61; string&#10; endpoint &#61; optional&#40;string&#41;&#10; max_batching_rows &#61; optional&#40;string&#41;&#10; user_defined_context &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [tables](variables.tf#L219) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map&#40;object&#40;&#123;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; schema &#61; optional&#40;string&#41;&#10; external_data_configuration &#61; optional&#40;object&#40;&#123;&#10; autodetect &#61; bool&#10; source_uris &#61; list&#40;string&#41;&#10; avro_logical_types &#61; optional&#40;bool&#41;&#10; compression &#61; optional&#40;string&#41;&#10; connection_id &#61; optional&#40;string&#41;&#10; file_set_spec_type &#61; optional&#40;string&#41;&#10; ignore_unknown_values &#61; optional&#40;bool&#41;&#10; metadata_cache_mode &#61; optional&#40;string&#41;&#10; object_metadata &#61; optional&#40;string&#41;&#10; json_options_encoding &#61; optional&#40;string&#41;&#10; reference_file_schema_uri &#61; optional&#40;string&#41;&#10; schema &#61; optional&#40;string&#41;&#10; source_format &#61; optional&#40;string&#41;&#10; max_bad_records &#61; optional&#40;number&#41;&#10; csv_options &#61; optional&#40;object&#40;&#123;&#10; quote &#61; string&#10; allow_jagged_rows &#61; optional&#40;bool&#41;&#10; allow_quoted_newlines &#61; optional&#40;bool&#41;&#10; encoding &#61; optional&#40;string&#41;&#10; field_delimiter &#61; optional&#40;string&#41;&#10; skip_leading_rows &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; google_sheets_options &#61; optional&#40;object&#40;&#123;&#10; range &#61; optional&#40;string&#41;&#10; skip_leading_rows &#61; optional&#40;number&#41;&#10; &#125;&#41;&#41;&#10; hive_partitioning_options &#61; optional&#40;object&#40;&#123;&#10; mode &#61; optional&#40;string&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; source_uri_prefix &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; parquet_options &#61; optional&#40;object&#40;&#123;&#10; enum_as_string &#61; optional&#40;bool&#41;&#10; enable_list_inference &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10;&#10;&#10; &#125;&#41;&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; encryption_key &#61; optional&#40;string&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; max_staleness &#61; optional&#40;string&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; type &#61; string&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10; table_constraints &#61; optional&#40;object&#40;&#123;&#10; primary_key_columns &#61; optional&#40;list&#40;string&#41;&#41;&#10; foreign_keys &#61; optional&#40;object&#40;&#123;&#10; referenced_table &#61; object&#40;&#123;&#10; project_id &#61; string&#10; dataset_id &#61; string&#10; table_id &#61; string&#10; &#125;&#41;&#10; column_references &#61; object&#40;&#123;&#10; referencing_column &#61; string&#10; referenced_column &#61; string&#10; &#125;&#41;&#10; name &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [tag_bindings](variables.tf#L304) | Tag bindings for this dataset, in key => tag value id format. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [views](variables.tf#L311) | View definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; use_legacy_sql &#61; optional&#40;bool&#41;&#10; schema &#61; optional&#40;list&#40;object&#40;&#123;&#10; name &#61; string&#10; type &#61; string&#10; description &#61; string&#10; mode &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
## Outputs

5
modules/bigquery-dataset/main.tf
View File

@@ -373,7 +373,7 @@ resource "google_bigquery_table" "default" {
}
resource "google_bigquery_table" "views" {
depends_on = [google_bigquery_table.default]
depends_on = [google_bigquery_table.default, google_bigquery_routine.default]
for_each = var.views
project = local.project_id
dataset_id = google_bigquery_dataset.default.dataset_id
@@ -382,7 +382,7 @@ resource "google_bigquery_table" "views" {
description = each.value.description
labels = each.value.labels
deletion_protection = each.value.deletion_protection
schema = try(jsonencode(each.value.schema), null)
schema = each.value.schema != null ? jsonencode(each.value.schema) : null
view {
query = each.value.query
@@ -445,6 +445,7 @@ resource "google_bigquery_routine" "default" {
imported_libraries = each.value.imported_libraries
determinism_level = each.value.determinism_level
data_governance_type = each.value.data_governance_type
return_type = each.value.return_type
return_table_type = each.value.return_table_type
dynamic "arguments" {
for_each = each.value.arguments

1
modules/bigquery-dataset/variables.tf
View File

@@ -187,6 +187,7 @@ variable "routines" {
imported_libraries = optional(list(string))
determinism_level = optional(string)
data_governance_type = optional(string)
return_type = optional(string)
return_table_type = optional(string)
arguments = optional(map(object({
argument_kind = optional(string)

4
tests/modules/bigquery_dataset/examples/routines.yaml
View File

@@ -48,7 +48,7 @@ values:
project: my-project
remote_function_options: []
return_table_type: null
return_type: null
return_type: '{"typeKind":"STRING"}'
routine_id: custom_masking_routine
routine_type: SCALAR_FUNCTION
spark_options: []
@@ -58,4 +58,4 @@ counts:
google_bigquery_dataset: 1
google_bigquery_routine: 1
modules: 1
resources: 2
resources: 2