kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Marcwo/dataform module (#2001)

Browse Source 
* pre PR commit

* added IAM functionality

* clean up of readme

* versions.tf fix

* added separate iam.tf, facilitated existing secret manager module

* corrected optional variable defaults

* adjusted readme to new changes

* adjusted tftest line in readme for changed example

* reverted the module back to a state where it only manages one instance

* minor fix for main readme.md

---------

Co-authored-by: Julio Castillo <jccb@google.com>
This commit is contained in:
marcjwo
2024-01-24 17:13:21 +01:00
committed by GitHub
parent 15439c3f5b
commit 6b4dad01d6
7 changed files with 275 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
modules/dataform-repository/README.md Normal file
View File

@@ -0,0 +1,72 @@
# Google Cloud Dataform Repository module
This module allows managing a dataform repository, allows adding IAM permissions. Also enables attaching a remote repository.
## TODO
[] Add validation rules to variable.
## Examples
### Simple dataform repository with access configration
Simple dataform repository and specifying repository access via the IAM variable.
```hcl
module "dataform" {
source = "./fabric/modules/dataform-repository"
project_id = "my-project"
name = "my-repository"
region = "europe-west1"
iam = {
"roles/dataform.editor" = ["user:user1@example.org"]
}
}
# tftest modules=1 resources=2
```
### Repository with an attached remote repository
This creates a dataform repository with a remote repository attached to it. In order to enable dataform to communicate with a 3P GIT provider, an access token must be generated and stored as a secret on GCP. For that, we utilize the existing [secret-manager module](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/modules/secret-manager).
```hcl
module "secret" {
source = "./fabric/modules/secret-manager"
project_id = "fast-bi-fabric"
secrets = {
my-secret = {
}
}
versions = {
my-secret = {
v1 = { enabled = true, data = "MYTOKEN" }
}
}
}
module "dataform" {
source = "./fabric/modules/dataform-repository"
project_id = "fast-bi-fabric"
name = "my-repository"
region = "europe-west1"
remote_repository_settings = {
url = "my-url"
secret_name = "my-secret"
token = module.secret.version_ids["my-secret:v1"]
}
}
# tftest modules=2 resources=3
```
<!-- BEGIN TFDOC -->
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [name](variables.tf#L54) | Name of the dataform repository. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L59) | Id of the project where resources will be created. | <code>string</code> | ✓ | |
| [region](variables.tf#L64) | The repository's region. | <code>string</code> | ✓ | |
| [iam](variables.tf#L17) | IAM bindings in {ROLE => [MEMBERS]} format. Mutually exclusive with the access_* variables used for basic roles. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [iam_bindings](variables.tf#L24) | Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary. | <code title="map&#40;object&#40;&#123;&#10; members &#61; list&#40;string&#41;&#10; role &#61; string&#10; condition &#61; optional&#40;object&#40;&#123;&#10; expression &#61; string&#10; title &#61; string&#10; description &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [iam_bindings_additive](variables.tf#L39) | Keyring individual additive IAM bindings. Keys are arbitrary. | <code title="map&#40;object&#40;&#123;&#10; member &#61; string&#10; role &#61; string&#10; condition &#61; optional&#40;object&#40;&#123;&#10; expression &#61; string&#10; title &#61; string&#10; description &#61; optional&#40;string&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [remote_repository_settings](variables.tf#L69) | Remote settings required to attach the repository to a remote repository. | <code title="object&#40;&#123;&#10; url &#61; optional&#40;string&#41;&#10; branch &#61; optional&#40;string, &#34;main&#34;&#41;&#10; secret_name &#61; optional&#40;string&#41;&#10; secret_version &#61; optional&#40;string, &#34;v1&#34;&#41;&#10; token &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [service_account](variables.tf#L81) | Service account used to execute the dataform workflow. | <code>string</code> | | <code>&#34;&#34;</code> |
<!-- END TFDOC -->

57
modules/dataform-repository/iam.tf Normal file
View File

@@ -0,0 +1,57 @@
/**
* Copyright 2024 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
resource "google_dataform_repository_iam_binding" "authoritative" {
provider = google-beta
for_each = var.iam
role = each.key
members = each.value
repository = google_dataform_repository.default.name
}
resource "google_dataform_repository_iam_binding" "bindings" {
provider = google-beta
for_each = var.iam_bindings
role = each.value.role
members = each.value.members
repository = google_dataform_repository.default.name
dynamic "condition" {
for_each = each.value.condition == null ? [] : [""]
content {
expression = each.value.condition.expression
title = each.value.condition.title
description = each.value.condition.description
}
}
}
resource "google_dataform_repository_iam_member" "bindings" {
provider = google-beta
for_each = var.iam_bindings_additive
role = each.value.role
member = each.value.member
repository = google_dataform_repository.default.name
dynamic "condition" {
for_each = each.value.condition == null ? [] : [""]
content {
expression = each.value.condition.expression
title = each.value.condition.title
description = each.value.condition.description
}
}
}

32
modules/dataform-repository/main.tf Normal file
View File

@@ -0,0 +1,32 @@
/**
* Copyright 2024 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
resource "google_dataform_repository" "default" {
provider = google-beta
project = var.project_id
name = var.name
region = var.region
service_account = var.service_account
dynamic "git_remote_settings" {
for_each = var.remote_repository_settings != null ? [1] : []
content {
url = var.remote_repository_settings.url
default_branch = var.remote_repository_settings.branch
authentication_token_secret_version = var.remote_repository_settings.secret_version
}
}
}

85
modules/dataform-repository/variables.tf Normal file
View File

@@ -0,0 +1,85 @@
/**
* Copyright 2024 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
variable "iam" {
description = "IAM bindings in {ROLE => [MEMBERS]} format. Mutually exclusive with the access_* variables used for basic roles."
type = map(list(string))
default = {}
nullable = false
}
variable "iam_bindings" {
description = "Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary."
type = map(object({
members = list(string)
role = string
condition = optional(object({
expression = string
title = string
description = optional(string)
}))
}))
default = {}
nullable = false
}
variable "iam_bindings_additive" {
description = "Keyring individual additive IAM bindings. Keys are arbitrary."
type = map(object({
member = string
role = string
condition = optional(object({
expression = string
title = string
description = optional(string)
}))
}))
default = {}
nullable = false
}
variable "name" {
description = "Name of the dataform repository."
type = string
}
variable "project_id" {
description = "Id of the project where resources will be created."
type = string
}
variable "region" {
description = "The repository's region."
type = string
}
variable "remote_repository_settings" {
description = "Remote settings required to attach the repository to a remote repository."
type = object({
url = optional(string)
branch = optional(string, "main")
secret_name = optional(string)
secret_version = optional(string, "v1")
token = optional(string)
})
default = null
}
variable "service_account" {
description = "Service account used to execute the dataform workflow."
type = string
default = ""
}

27
modules/dataform-repository/versions.tf Normal file
View File

@@ -0,0 +1,27 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
terraform {
required_version = ">= 1.5.1"
required_providers {
google = {
source = "hashicorp/google"
version = ">= 5.11.0, < 6.0.0" # tftest
}
google-beta = {
source = "hashicorp/google-beta"
version = ">= 5.11.0, < 6.0.0" # tftest
}
}
}