FAST light implementation (#3255)
* data wip * wip data * update org schema, add note on expansion * all schemas, workload notes * Update WORKLOG.md * Update WORKLOG.md * Update WORKLOG.md * Update WORKLOG.md * wip * data wip * wip * wip * wip * wip * org module IAM context (using lookup) * new-style context expansion in project IAM * remove spurious file * project module contexts * finalize context replacement format for project module * revert org module changes * fix tag id interpolation in project * fix tag id interpolation in project * organization module context * organization context test * context expansion for folder tag bindings * test context expansion for tag bindings * service account module context * simplify context local * context for iam service account * nuke blueprints * remove links to blueprints * vpc sc context in project module * Add context to GCS module * Add inline deps to plan_summary script * Make context a top-level variable for folder, organization, sa * Add add context top-level to VPC-SC * move context out of factories_config variable * tfdoc * fix merge * fix merge * fix examples * net-vpc module context * add parent ids to folder context * rename folder parent context * fix folder parent check * new project factory stub * wip * wip * refactor defaults * project iam * bueckts and service accounts * start adding context replacements * better test data * automation resources for folders and projects * automation * add support for project id interpolation * first tested apply * improve IAM description in gcs module * add context to billing account module * add notification channels to billing account module context * add billing budgets to new pf * schemas and defaults * bootstrap wip * bootstrap wip * bootstrap wip * pf outputs * pf fixes * fix pf sample data * bootstrap lite fixes * add locations to organization module contexts * bootstrap lite fixes * org fixes, billing accounts * fix default project parent * bootstrap lite wip * add locations to gcs module context * add context support to logging bucket module * add context to pubsub module * split out iam variables in gcs module * fix logging bucket context test * bootstrap log sink destinations * streamline logging-bucket module variables * fix logging bucket context test * align logging bucket module interface in fast bootstrap * add support for project-level log buckets to project factory * support full context expansion in organization module log sinks * log buckets in fast-lite bootstrap * make og sink type optional in organization module * log sinks in fast-lite bootstrap * set tag values in factory context * bootstrap lite data * output files schema * billing account schema * output files * output providers * gcs output files * boilerplate * tflint * check documentation * check docs * fix project module parent variable validation * fix log bucket examples * allow null parent in project module * silence folder test errors * fix billing account sink example * fix project example * fix billing account module * fix folder tests * fix FAST * fix fast * tfvars outputs * wif * cicd service accounts * cicd * allow defaults in context, minimal org policies * support gcs managed folders in project factory and bootstrap lite * support prefix in provider output files * rename bootstrap stage * gitignore * gitignore * security folder, billing IAM * wip tfvars * fix typo * security IAM * control tag iam/context via variables in organization module * split tag creation from tag IAM to avoid circular refs * port organization module tag changes to project module * implement new-style context expansion in vpc-sc module * fix fast vpc-sc tests * boilerplate * vpc sc stage * schemas * fast-lite compatibility for vpc sc stage * make log project number optional in vpc-sc stage * networking * networking * networking * networking * rename and move new stage under fast * clone pf tests * use context replacement for internal notification channels in billing account module * support service agents in project module iam context replacements * support service agents in project module iam context replacements * add support for kms keys to project module context * experimental pf example test and fixes * fix schemas * fix tests * tfdoc * tfdoc * pf config * experimental pf * remove redundant dot from gcs managed folder IAM keys * bootstrap experimental test * project factory exp stage test * skip tflint for bootstrap experimental test * tflint * fix gcs test * documentation work * documentation work * Update README.md * tfdoc * tfdoc * readme * tfdoc * readme * readme * readme * readme * support universe in pf exp projects * missing universe service agents * org policies import, non-admin billing IAM * todo * fix test * custom constraints * fast classic dataset * fix test data * context replacements in billing module log sinks * fix typo * add support for billing log sinks * update docs * readme * cicd fix and test --------- Co-authored-by: Julio Castillo <jccb@google.com>
This commit is contained in:
Ludovico Magnocavallo
committed by
GitHub
GitHub
parent
dcd762370c
commit
36648b6b63
@@ -129,7 +129,7 @@ IAM is managed via several variables that implement different features and level
|
||||
|
||||
The authoritative and additive approaches can be used together, provided different roles are managed by each. Some care must also be taken with the `iam_by_principals` variable to ensure that variable keys are static values, so that Terraform is able to compute the dependency graph.
|
||||
|
||||
Refer to the [project module](../project/README.md#iam) for examples of the IAM interface.
|
||||
IAM also supports variable interpolation for both roles and principals, via the respective attributes in the `var.context` variable. Refer to the [project module](../project/README.md#iam) for examples of the IAM interface.
|
||||
|
||||
## Organization Policies
|
||||
|
||||
@@ -257,7 +257,7 @@ module "org" {
|
||||
|
||||
## Log Sinks
|
||||
|
||||
The following example shows how to define organization-level log sinks:
|
||||
The following example shows how to define organization-level log sinks, which support interpolation in the destination argument.
|
||||
|
||||
```hcl
|
||||
module "gcs" {
|
||||
@@ -282,10 +282,9 @@ module "pubsub" {
|
||||
}
|
||||
|
||||
module "bucket" {
|
||||
source = "./fabric/modules/logging-bucket"
|
||||
parent_type = "project"
|
||||
parent = var.project_id
|
||||
id = "${var.prefix}-bucket"
|
||||
source = "./fabric/modules/logging-bucket"
|
||||
parent = var.project_id
|
||||
name = "${var.prefix}-bucket"
|
||||
}
|
||||
|
||||
module "destination-project" {
|
||||
@@ -302,8 +301,17 @@ module "destination-project" {
|
||||
module "org" {
|
||||
source = "./fabric/modules/organization"
|
||||
organization_id = var.organization_id
|
||||
|
||||
context = {
|
||||
storage_buckets = {
|
||||
my_bucket = "test-prod-log-audit-0"
|
||||
}
|
||||
}
|
||||
logging_sinks = {
|
||||
audit = {
|
||||
destination = "$storage_buckets:my_bucket"
|
||||
filter = "log_id('cloudaudit.googleapis.com/activity')"
|
||||
type = "storage"
|
||||
}
|
||||
warnings = {
|
||||
destination = module.gcs.id
|
||||
filter = "severity=WARNING"
|
||||
@@ -338,7 +346,7 @@ module "org" {
|
||||
no-gce-instances = "resource.type=gce_instance"
|
||||
}
|
||||
}
|
||||
# tftest modules=6 resources=17 inventory=logging.yaml e2e serial
|
||||
# tftest modules=6 resources=19 inventory=logging.yaml
|
||||
```
|
||||
|
||||
## Data Access Logs
|
||||
@@ -521,25 +529,22 @@ Tags can also be specified via a factory in a similar way to organization polici
|
||||
- The structure of the YAML file allows defining the `description`, `iam` bindings, and a map of `values` for the tag key, including their own descriptions and IAM.
|
||||
- Tags defined via the `tags` and `network_tags` variables are merged with those from the factory, and will override factory definitions in case of duplicate names.
|
||||
|
||||
The example below deploys a `cost-center` tag key and its values from a YAML file.
|
||||
The example below deploys a `cost-center` tag key and its values from a YAML file. Context expansion supports interpolation via the `context.tag_keys` and `context.tag_values` variables, as shown below.
|
||||
|
||||
```hcl
|
||||
module "org" {
|
||||
source = "./fabric/modules/organization"
|
||||
organization_id = var.organization_id
|
||||
context = {
|
||||
tag_keys = {
|
||||
environment = "tagKeys/1234567890"
|
||||
}
|
||||
tag_values = {
|
||||
"environment/production" = "tagValues/1234567890"
|
||||
}
|
||||
}
|
||||
factories_config = {
|
||||
tags = "data/tags"
|
||||
context = {
|
||||
iam_principals = {
|
||||
gcp-admins = "group:gcp-organization-admins@example.com"
|
||||
}
|
||||
tag_keys = {
|
||||
foo = "tagKeys/1234567890"
|
||||
}
|
||||
tag_values = {
|
||||
"foo/bar" = "tagValues/789012345"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
# tftest modules=1 resources=7 files=0,1 inventory=tags-factory.yaml
|
||||
@@ -560,23 +565,21 @@ values:
|
||||
```
|
||||
|
||||
```yaml
|
||||
# tftest-file id=1 path=data/tags/foo.yaml
|
||||
# tftest-file id=1 path=data/tags/environment.yaml
|
||||
|
||||
description: "Tag that shows context expansion."
|
||||
id: foo
|
||||
id: $tag_keys:environment
|
||||
iam:
|
||||
"roles/resourcemanager.tagViewer":
|
||||
- "group:finance-team@example.com"
|
||||
- gcp-admins
|
||||
- "group:gcp-devops@example.com"
|
||||
values:
|
||||
bar:
|
||||
id: foo/bar
|
||||
description: "This tag value will be reused."
|
||||
development:
|
||||
description: "Development."
|
||||
production:
|
||||
id: $tag_values:environment/production
|
||||
iam:
|
||||
"roles/resourcemanager.tagViewer":
|
||||
- gcp-admins
|
||||
baz:
|
||||
description: "This tag value will be created."
|
||||
"roles/resourcemanager.tagUser":
|
||||
- "group:gcp-devops@example.com"
|
||||
|
||||
```
|
||||
|
||||
<!-- TFDOC OPTS files:1 -->
|
||||
@@ -602,12 +605,13 @@ values:
|
||||
|
||||
| name | description | type | required | default |
|
||||
|---|---|:---:|:---:|:---:|
|
||||
| [organization_id](variables.tf#L100) | Organization id in organizations/nnnnnn format. | <code>string</code> | ✓ | |
|
||||
| [organization_id](variables.tf#L113) | Organization id in organizations/nnnnnn format. | <code>string</code> | ✓ | |
|
||||
| [contacts](variables.tf#L17) | List of essential contacts for this resource. Must be in the form EMAIL -> [NOTIFICATION_TYPES]. Valid notification types are ALL, SUSPENSION, SECURITY, TECHNICAL, BILLING, LEGAL, PRODUCT_UPDATES. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [custom_roles](variables.tf#L24) | Map of role name => list of permissions to create in this project. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [factories_config](variables.tf#L31) | Paths to data files and folders that enable factory functionality. | <code title="object({ custom_roles = optional(string) org_policies = optional(string) org_policy_custom_constraints = optional(string) tags = optional(string) context = optional(object({ iam_principals = optional(map(string), {}) org_policies = optional(map(map(string)), {}) tag_keys = optional(map(string), {}) tag_values = optional(map(string), {}) }), {}) })">object({…})</code> | | <code>{}</code> |
|
||||
| [firewall_policy](variables.tf#L49) | Hierarchical firewall policies to associate to the organization. | <code title="object({ name = string policy = string })">object({…})</code> | | <code>null</code> |
|
||||
| [iam](variables-iam.tf#L17) | IAM bindings, in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [context](variables.tf#L24) | Context-specific interpolations. | <code title="object({ bigquery_datasets = optional(map(string), {}) custom_roles = optional(map(string), {}) iam_principals = optional(map(string), {}) locations = optional(map(string), {}) log_buckets = optional(map(string), {}) org_policies = optional(map(map(string)), {}) project_ids = optional(map(string), {}) pubsub_topics = optional(map(string), {}) storage_buckets = optional(map(string), {}) tag_keys = optional(map(string), {}) tag_values = optional(map(string), {}) })">object({…})</code> | | <code>{}</code> |
|
||||
| [custom_roles](variables.tf#L43) | Map of role name => list of permissions to create in this project. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [factories_config](variables.tf#L50) | Paths to data files and folders that enable factory functionality. | <code title="object({ custom_roles = optional(string) org_policies = optional(string) org_policy_custom_constraints = optional(string) tags = optional(string) })">object({…})</code> | | <code>{}</code> |
|
||||
| [firewall_policy](variables.tf#L62) | Hierarchical firewall policies to associate to the organization. | <code title="object({ name = string policy = string })">object({…})</code> | | <code>null</code> |
|
||||
| [iam](variables-iam.tf#L17) | Authoritative IAM bindings in {ROLE => [MEMBERS]} format. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
| [iam_bindings](variables-iam.tf#L24) | Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary. | <code title="map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [iam_bindings_additive](variables-iam.tf#L39) | Individual additive IAM bindings. Keys are arbitrary. | <code title="map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [iam_by_principals](variables-iam.tf#L61) | Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid errors. Merged internally with the `iam` variable. | <code>map(list(string))</code> | | <code>{}</code> |
|
||||
@@ -615,12 +619,13 @@ values:
|
||||
| [logging_data_access](variables-logging.tf#L17) | Control activation of data access logs. The special 'allServices' key denotes configuration for all services. | <code title="map(object({ ADMIN_READ = optional(object({ exempted_members = optional(list(string)) })), DATA_READ = optional(object({ exempted_members = optional(list(string)) })), DATA_WRITE = optional(object({ exempted_members = optional(list(string)) })) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [logging_exclusions](variables-logging.tf#L28) | Logging exclusions for this organization in the form {NAME -> FILTER}. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [logging_settings](variables-logging.tf#L35) | Default settings for logging resources. | <code title="object({ disable_default_sink = optional(bool) storage_location = optional(string) })">object({…})</code> | | <code>null</code> |
|
||||
| [logging_sinks](variables-logging.tf#L45) | Logging sinks to create for the organization. | <code title="map(object({ bq_partitioned_table = optional(bool, false) description = optional(string) destination = string disabled = optional(bool, false) exclusions = optional(map(string), {}) filter = optional(string) iam = optional(bool, true) include_children = optional(bool, true) intercept_children = optional(bool, false) type = string }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [network_tags](variables-tags.tf#L17) | Network tags by key name. If `id` is provided, key creation is skipped. The `iam` attribute behaves like the similarly named one at module level. | <code title="map(object({ description = optional(string, "Managed by the Terraform organization module.") iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) id = optional(string) network = string # project_id/vpc_name values = optional(map(object({ description = optional(string, "Managed by the Terraform organization module.") iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) })), {}) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [org_policies](variables.tf#L58) | Organization policies applied to this organization keyed by policy name. | <code title="map(object({ inherit_from_parent = optional(bool) # for list policies only. reset = optional(bool) rules = optional(list(object({ allow = optional(object({ all = optional(bool) values = optional(list(string)) })) deny = optional(object({ all = optional(bool) values = optional(list(string)) })) enforce = optional(bool) # for boolean policies only. condition = optional(object({ description = optional(string) expression = optional(string) location = optional(string) title = optional(string) }), {}) parameters = optional(string) })), []) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [org_policy_custom_constraints](variables.tf#L86) | Organization policy custom constraints keyed by constraint name. | <code title="map(object({ display_name = optional(string) description = optional(string) action_type = string condition = string method_types = list(string) resource_types = list(string) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [tag_bindings](variables-tags.tf#L81) | Tag bindings for this organization, in key => tag value id format. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [tags](variables-tags.tf#L88) | Tags by key name. If `id` is provided, key or value creation is skipped. The `iam` attribute behaves like the similarly named one at module level. | <code title="map(object({ description = optional(string, "Managed by the Terraform organization module.") iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) id = optional(string) values = optional(map(object({ description = optional(string, "Managed by the Terraform organization module.") iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) id = optional(string) })), {}) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [logging_sinks](variables-logging.tf#L45) | Logging sinks to create for the organization. | <code title="map(object({ destination = string bq_partitioned_table = optional(bool, false) description = optional(string) disabled = optional(bool, false) exclusions = optional(map(string), {}) filter = optional(string) iam = optional(bool, true) include_children = optional(bool, true) intercept_children = optional(bool, false) type = optional(string, "logging") }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [network_tags](variables-tags.tf#L17) | Network tags by key name. If `id` is provided, key creation is skipped. The `iam` attribute behaves like the similarly named one at module level. | <code title="map(object({ description = optional(string, "Managed by the Terraform organization module.") id = optional(string) network = string # project_id/vpc_name iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) values = optional(map(object({ description = optional(string, "Managed by the Terraform organization module.") id = optional(string) iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) })), {}) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [org_policies](variables.tf#L71) | Organization policies applied to this organization keyed by policy name. | <code title="map(object({ inherit_from_parent = optional(bool) # for list policies only. reset = optional(bool) rules = optional(list(object({ allow = optional(object({ all = optional(bool) values = optional(list(string)) })) deny = optional(object({ all = optional(bool) values = optional(list(string)) })) enforce = optional(bool) # for boolean policies only. condition = optional(object({ description = optional(string) expression = optional(string) location = optional(string) title = optional(string) }), {}) parameters = optional(string) })), []) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [org_policy_custom_constraints](variables.tf#L99) | Organization policy custom constraints keyed by constraint name. | <code title="map(object({ display_name = optional(string) description = optional(string) action_type = string condition = string method_types = list(string) resource_types = list(string) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [tag_bindings](variables-tags.tf#L82) | Tag bindings for this organization, in key => tag value id format. | <code>map(string)</code> | | <code>{}</code> |
|
||||
| [tags](variables-tags.tf#L89) | Tags by key name. If `id` is provided, key or value creation is skipped. The `iam` attribute behaves like the similarly named one at module level. | <code title="map(object({ description = optional(string, "Managed by the Terraform organization module.") iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) id = optional(string) values = optional(map(object({ description = optional(string, "Managed by the Terraform organization module.") iam = optional(map(list(string)), {}) iam_bindings = optional(map(object({ members = list(string) role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) iam_bindings_additive = optional(map(object({ member = string role = string condition = optional(object({ expression = string title = string description = optional(string) })) })), {}) id = optional(string) })), {}) }))">map(object({…}))</code> | | <code>{}</code> |
|
||||
| [tags_config](variables-tags.tf#L154) | Fine-grained control on tag resource and IAM creation. | <code title="object({ force_context_ids = optional(bool, false) ignore_iam = optional(bool, false) })">object({…})</code> | | <code>{}</code> |
|
||||
|
||||
## Outputs
|
||||
|
||||
@@ -628,12 +633,12 @@ values:
|
||||
|---|---|:---:|
|
||||
| [custom_constraint_ids](outputs.tf#L17) | Map of CUSTOM_CONSTRAINTS => ID in the organization. | |
|
||||
| [custom_role_id](outputs.tf#L22) | Map of custom role IDs created in the organization. | |
|
||||
| [custom_roles](outputs.tf#L32) | Map of custom roles resources created in the organization. | |
|
||||
| [id](outputs.tf#L37) | Fully qualified organization id. | |
|
||||
| [network_tag_keys](outputs.tf#L55) | Tag key resources. | |
|
||||
| [network_tag_values](outputs.tf#L64) | Tag value resources. | |
|
||||
| [organization_id](outputs.tf#L74) | Organization id dependent on module resources. | |
|
||||
| [sink_writer_identities](outputs.tf#L91) | Writer identities created for each sink. | |
|
||||
| [tag_keys](outputs.tf#L99) | Tag key resources. | |
|
||||
| [tag_values](outputs.tf#L108) | Tag value resources. | |
|
||||
| [custom_roles](outputs.tf#L27) | Map of custom roles resources created in the organization. | |
|
||||
| [id](outputs.tf#L32) | Fully qualified organization id. | |
|
||||
| [network_tag_keys](outputs.tf#L50) | Tag key resources. | |
|
||||
| [network_tag_values](outputs.tf#L59) | Tag value resources. | |
|
||||
| [organization_id](outputs.tf#L69) | Organization id dependent on module resources. | |
|
||||
| [sink_writer_identities](outputs.tf#L86) | Writer identities created for each sink. | |
|
||||
| [tag_keys](outputs.tf#L94) | Tag key resources. | |
|
||||
| [tag_values](outputs.tf#L103) | Tag value resources. | |
|
||||
<!-- END TFDOC -->
|
||||
|
||||
Reference in New Issue
Block a user