diff --git a/README.md b/README.md index 6aa292d76..ee10367a2 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ The current list of modules supports most of the core foundational and networkin Currently available modules: -- **foundational** - [billing budget](./modules/billing-budget), [Cloud Identity group](./modules/cloud-identity-group/), [folder](./modules/folder), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [organization](./modules/organization), [organization-policy](./modules/organization-policy), [project](./modules/project), [projects-data-source](./modules/projects-data-source) +- **foundational** - [billing budget](./modules/billing-budget), [Cloud Identity group](./modules/cloud-identity-group/), [folder](./modules/folder), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [organization](./modules/organization), [project](./modules/project), [projects-data-source](./modules/projects-data-source) - **networking** - [DNS](./modules/dns), [Cloud Endpoints](./modules/endpoints), [address reservation](./modules/net-address), [NAT](./modules/net-cloudnat), [Global Load Balancer (classic)](./modules/net-glb/), [L4 ILB](./modules/net-ilb), [L7 ILB](./modules/net-ilb-l7), [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC peering](./modules/net-vpc-peering), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [VPN static](./modules/net-vpn-static), [Service Directory](./modules/service-directory) - **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid), [GKE cluster](./modules/gke-cluster), [GKE hub](./modules/gke-hub), [GKE nodepool](./modules/gke-nodepool) - **data** - [BigQuery dataset](./modules/bigquery-dataset), [Bigtable instance](./modules/bigtable-instance), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag), [Datafusion](./modules/datafusion), [GCS](./modules/gcs), [Pub/Sub](./modules/pubsub) diff --git a/fast/stages/01-resman/README.md b/fast/stages/01-resman/README.md index dbeb8afb7..52e73c55e 100644 --- a/fast/stages/01-resman/README.md +++ b/fast/stages/01-resman/README.md @@ -182,17 +182,18 @@ Due to its simplicity, this stage lends itself easily to customizations: adding |---|---|:---:|:---:|:---:|:---:| | [automation](variables.tf#L20) | Automation resources created by the bootstrap stage. | object({…}) | ✓ | | 00-bootstrap | | [billing_account](variables.tf#L38) | Billing account id and organization id ('nnnnnnnn' or null). | object({…}) | ✓ | | 00-bootstrap | -| [organization](variables.tf#L191) | Organization details. | object({…}) | ✓ | | 00-bootstrap | -| [prefix](variables.tf#L215) | Prefix used for resources that need unique names. Use 9 characters or less. | string | ✓ | | 00-bootstrap | +| [organization](variables.tf#L197) | Organization details. | object({…}) | ✓ | | 00-bootstrap | +| [prefix](variables.tf#L221) | Prefix used for resources that need unique names. Use 9 characters or less. | string | ✓ | | 00-bootstrap | | [cicd_repositories](variables.tf#L47) | CI/CD repository configuration. Identity providers reference keys in the `automation.federated_identity_providers` variable. Set to null to disable, or set individual repositories to null if not needed. | object({…}) | | null | | | [custom_roles](variables.tf#L129) | Custom roles defined at the org level, in key => id format. | object({…}) | | null | 00-bootstrap | -| [fast_features](variables.tf#L138) | Selective control for top-level FAST features. | object({…}) | | {…} | 00-bootstrap | -| [groups](variables.tf#L158) | Group names to grant organization-level permissions. | map(string) | | {…} | 00-bootstrap | -| [locations](variables.tf#L173) | Optional locations for GCS, BigQuery, and logging buckets created here. | object({…}) | | {…} | 00-bootstrap | -| [organization_policy_configs](variables.tf#L201) | Organization policies customization. | object({…}) | | null | | -| [outputs_location](variables.tf#L209) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | string | | null | | -| [tag_names](variables.tf#L226) | Customized names for resource management tags. | object({…}) | | {…} | | -| [team_folders](variables.tf#L243) | Team folders to be created. Format is described in a code comment. | map(object({…})) | | null | | +| [data_dir](variables.tf#L138) | Relative path for the folder storing configuration data. | string | | "data" | | +| [fast_features](variables.tf#L144) | Selective control for top-level FAST features. | object({…}) | | {…} | 00-bootstrap | +| [groups](variables.tf#L164) | Group names to grant organization-level permissions. | map(string) | | {…} | 00-bootstrap | +| [locations](variables.tf#L179) | Optional locations for GCS, BigQuery, and logging buckets created here. | object({…}) | | {…} | 00-bootstrap | +| [organization_policy_configs](variables.tf#L207) | Organization policies customization. | object({…}) | | null | | +| [outputs_location](variables.tf#L215) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable | string | | null | | +| [tag_names](variables.tf#L232) | Customized names for resource management tags. | object({…}) | | {…} | | +| [team_folders](variables.tf#L249) | Team folders to be created. Format is described in a code comment. | map(object({…})) | | null | | ## Outputs diff --git a/fast/stages/01-resman/data/org-policies/compute.yaml b/fast/stages/01-resman/data/org-policies/compute.yaml new file mode 100644 index 000000000..0d27ac426 --- /dev/null +++ b/fast/stages/01-resman/data/org-policies/compute.yaml @@ -0,0 +1,73 @@ +# skip boilerplate check +# +# sample subset of useful organization policies, edit to suit requirements + +compute.disableGuestAttributesAccess: + enforce: true + +compute.requireOsLogin: + enforce: true + +compute.restrictLoadBalancerCreationForTypes: + allow: + values: + - in:INTERNAL + +compute.skipDefaultNetworkCreation: + enforce: true + +compute.vmExternalIpAccess: + deny: + all: true + + +# compute.disableInternetNetworkEndpointGroup: +# enforce: true + +# compute.disableNestedVirtualization: +# enforce: true + +# compute.disableSerialPortAccess: +# enforce: true + +# compute.restrictCloudNATUsage: +# deny: +# all: true + +# compute.restrictDedicatedInterconnectUsage: +# deny: +# all: true + +# compute.restrictPartnerInterconnectUsage: +# deny: +# all: true + +# compute.restrictProtocolForwardingCreationForTypes: +# deny: +# all: true + +# compute.restrictSharedVpcHostProjects: +# deny: +# all: true + +# compute.restrictSharedVpcSubnetworks: +# deny: +# all: true + +# compute.restrictVpcPeering: +# deny: +# all: true + +# compute.restrictVpnPeerIPs: +# deny: +# all: true + +# compute.restrictXpnProjectLienRemoval: +# enforce: true + +# compute.setNewProjectDefaultToZonalDNSOnly: +# enforce: true + +# compute.vmCanIpForward: +# deny: +# all: true diff --git a/fast/stages/01-resman/data/org-policies/iam.yaml b/fast/stages/01-resman/data/org-policies/iam.yaml new file mode 100644 index 000000000..4d83f827f --- /dev/null +++ b/fast/stages/01-resman/data/org-policies/iam.yaml @@ -0,0 +1,12 @@ +# skip boilerplate check +# +# sample subset of useful organization policies, edit to suit requirements + +iam.automaticIamGrantsForDefaultServiceAccounts: + enforce: true + +iam.disableServiceAccountKeyCreation: + enforce: true + +iam.disableServiceAccountKeyUpload: + enforce: true diff --git a/fast/stages/01-resman/data/org-policies/serverless.yaml b/fast/stages/01-resman/data/org-policies/serverless.yaml new file mode 100644 index 000000000..de62e6c70 --- /dev/null +++ b/fast/stages/01-resman/data/org-policies/serverless.yaml @@ -0,0 +1,26 @@ +# skip boilerplate check +# +# sample subset of useful organization policies, edit to suit requirements + +run.allowedIngress: + allow: + values: + - is:internal + +# run.allowedVPCEgress: +# allow: +# values: +# - is:private-ranges-only + +# cloudfunctions.allowedIngressSettings: +# allow: +# values: +# - is:ALLOW_INTERNAL_ONLY + +# cloudfunctions.allowedVpcConnectorEgressSettings: +# allow: +# values: +# - is:PRIVATE_RANGES_ONLY + +# cloudfunctions.requireVPCConnector: +# enforce: true diff --git a/fast/stages/01-resman/data/org-policies/sql.yaml b/fast/stages/01-resman/data/org-policies/sql.yaml new file mode 100644 index 000000000..88b84d9d5 --- /dev/null +++ b/fast/stages/01-resman/data/org-policies/sql.yaml @@ -0,0 +1,9 @@ +# skip boilerplate check +# +# sample subset of useful organization policies, edit to suit requirements + +sql.restrictAuthorizedNetworks: + enforce: true + +sql.restrictPublicIp: + enforce: true diff --git a/fast/stages/01-resman/data/org-policies/storage.yaml b/fast/stages/01-resman/data/org-policies/storage.yaml new file mode 100644 index 000000000..6c0a673f3 --- /dev/null +++ b/fast/stages/01-resman/data/org-policies/storage.yaml @@ -0,0 +1,6 @@ +# skip boilerplate check +# +# sample subset of useful organization policies, edit to suit requirements + +storage.uniformBucketLevelAccess: + enforce: true diff --git a/fast/stages/01-resman/organization.tf b/fast/stages/01-resman/organization.tf index 40a789eec..7ecf79523 100644 --- a/fast/stages/01-resman/organization.tf +++ b/fast/stages/01-resman/organization.tf @@ -66,44 +66,12 @@ module "organization" { ) } : {} ) - # sample subset of useful organization policies, edit to suit requirements + # sample subset of useful organization policies, edit to suit requirements org_policies = { - "compute.disableGuestAttributesAccess" = { enforce = true } - "compute.requireOsLogin" = { enforce = true } - "compute.restrictLoadBalancerCreationForTypes" = { allow = { values = ["in:INTERNAL"] } } - "compute.skipDefaultNetworkCreation" = { enforce = true } - "compute.vmExternalIpAccess" = { deny = { all = true } } - "iam.allowedPolicyMemberDomains" = { allow = { values = local.all_drs_domains } } - "iam.automaticIamGrantsForDefaultServiceAccounts" = { enforce = true } - "iam.disableServiceAccountKeyCreation" = { enforce = true } - "iam.disableServiceAccountKeyUpload" = { enforce = true } - "run.allowedIngress" = { allow = { values = ["is:internal"] } } - "sql.restrictAuthorizedNetworks" = { enforce = true } - "sql.restrictPublicIp" = { enforce = true } - "storage.uniformBucketLevelAccess" = { enforce = true } - # "cloudfunctions.allowedIngressSettings" = { - # allow = { values = ["is:ALLOW_INTERNAL_ONLY"] } - # } - # "cloudfunctions.allowedVpcConnectorEgressSettings" = { - # allow = { values = ["is:PRIVATE_RANGES_ONLY"] } - # } - # "cloudfunctions.requireVPCConnector" = { enforce = true } - # "compute.disableInternetNetworkEndpointGroup" = { enforce = true } - # "compute.disableNestedVirtualization" = { enforce = true } - # "compute.disableSerialPortAccess" = { enforce = true } - # "compute.restrictCloudNATUsage" = { deny = { all = true }} - # "compute.restrictDedicatedInterconnectUsage" = { deny = { all = true }} - # "compute.restrictPartnerInterconnectUsage" = { deny = { all = true }} - # "compute.restrictProtocolForwardingCreationForTypes" = { deny = { all = true }} - # "compute.restrictSharedVpcHostProjects" = { deny = { all = true }} - # "compute.restrictSharedVpcSubnetworks" = { deny = { all = true }} - # "compute.restrictVpcPeering" = { deny = { all = true }} - # "compute.restrictVpnPeerIPs" = { deny = { all = true }} - # "compute.restrictXpnProjectLienRemoval" = { enforce = true } - # "compute.setNewProjectDefaultToZonalDNSOnly" = { enforce = true } - # "compute.vmCanIpForward" = { deny = { all = true }} - # "gcp.resourceLocations" = { + "iam.allowedPolicyMemberDomains" = { allow = { values = local.all_drs_domains } } + + #"gcp.resourceLocations" = { # allow = { values = local.allowed_regions } # } # "iam.workloadIdentityPoolProviders" = { @@ -114,8 +82,9 @@ module "organization" { # ] # } # } - # "run.allowedVPCEgress" = { allow = { values = ["is:private-ranges-only"] } } } + org_policies_data_path = "${var.data_dir}/org-policies" + tags = { (var.tag_names.context) = { description = "Resource management context." diff --git a/fast/stages/01-resman/variables.tf b/fast/stages/01-resman/variables.tf index 8da869670..6de9a7fad 100644 --- a/fast/stages/01-resman/variables.tf +++ b/fast/stages/01-resman/variables.tf @@ -135,6 +135,12 @@ variable "custom_roles" { default = null } +variable "data_dir" { + description = "Relative path for the folder storing configuration data." + type = string + default = "data" +} + variable "fast_features" { # tfdoc:variable:source 00-bootstrap description = "Selective control for top-level FAST features." diff --git a/modules/README.md b/modules/README.md index 129a8b8f9..9995c4ebe 100644 --- a/modules/README.md +++ b/modules/README.md @@ -36,7 +36,6 @@ These modules are used in the examples included in this repository. If you are u - [service accounts](./iam-service-account) - [logging bucket](./logging-bucket) - [organization](./organization) -- [organization-policy](./organization-policy) - [project](./project) - [projects-data-source](./projects-data-source) diff --git a/modules/folder/README.md b/modules/folder/README.md index d543004a2..2190eaacc 100644 --- a/modules/folder/README.md +++ b/modules/folder/README.md @@ -75,6 +75,10 @@ module "folder" { # tftest modules=1 resources=8 ``` +### Organization policy factory + +See the [organization policy factory in the project module](../project#organization-policy-factory). + ### Firewall policy factory In the same way as for the [organization](../organization) module, the in-built factory allows you to define a single policy, using one file for rules, and an optional file for CIDR range substitution variables. Remember that non-absolute paths are relative to the root module (the folder where you run `terraform`). @@ -311,8 +315,9 @@ module "folder" { | [logging_sinks](variables.tf#L105) | Logging sinks to create for this folder. | map(object({…})) | | {} | | [name](variables.tf#L126) | Folder name. | string | | null | | [org_policies](variables.tf#L132) | Organization policies applied to this folder keyed by policy name. | map(object({…})) | | {} | -| [parent](variables.tf#L172) | Parent in folders/folder_id or organizations/org_id format. | string | | null | -| [tag_bindings](variables.tf#L182) | Tag bindings for this folder, in key => tag value id format. | map(string) | | null | +| [org_policies_data_path](variables.tf#L172) | Path containing org policies in YAML format. | string | | null | +| [parent](variables.tf#L178) | Parent in folders/folder_id or organizations/org_id format. | string | | null | +| [tag_bindings](variables.tf#L188) | Tag bindings for this folder, in key => tag value id format. | map(string) | | null | ## Outputs diff --git a/modules/folder/organization-policies.tf b/modules/folder/organization-policies.tf index 8fc021400..3766005ae 100644 --- a/modules/folder/organization-policies.tf +++ b/modules/folder/organization-policies.tf @@ -17,8 +17,57 @@ # tfdoc:file:description Folder-level organization policies. locals { + _factory_data_raw = ( + var.org_policies_data_path == null + ? tomap({}) + : merge([ + for f in fileset(var.org_policies_data_path, "*.yaml") : + yamldecode(file("${var.org_policies_data_path}/${f}")) + ]...) + ) + + # simulate applying defaults to data coming from yaml files + _factory_data = { + for k, v in local._factory_data_raw : + k => { + inherit_from_parent = try(v.inherit_from_parent, null) + reset = try(v.reset, null) + allow = can(v.allow) ? { + all = try(v.allow.all, null) + values = try(v.allow.values, null) + } : null + deny = can(v.deny) ? { + all = try(v.deny.all, null) + values = try(v.deny.values, null) + } : null + enforce = try(v.enforce, true) + + rules = [ + for r in try(v.rules, []) : { + allow = can(r.allow) ? { + all = try(r.allow.all, null) + values = try(r.allow.values, null) + } : null + deny = can(r.deny) ? { + all = try(r.deny.all, null) + values = try(r.deny.values, null) + } : null + enforce = try(r.enforce, true) + condition = { + description = try(r.condition.description, null) + expression = try(r.condition.expression, null) + location = try(r.condition.location, null) + title = try(r.condition.title, null) + } + } + ] + } + } + + _org_policies = merge(local._factory_data, var.org_policies) + org_policies = { - for k, v in var.org_policies : + for k, v in local._org_policies : k => merge(v, { name = "${local.folder.name}/policies/${k}" parent = local.folder.name diff --git a/modules/folder/variables.tf b/modules/folder/variables.tf index a00e147f6..359531b7d 100644 --- a/modules/folder/variables.tf +++ b/modules/folder/variables.tf @@ -169,6 +169,12 @@ variable "org_policies" { nullable = false } +variable "org_policies_data_path" { + description = "Path containing org policies in YAML format." + type = string + default = null +} + variable "parent" { description = "Parent in folders/folder_id or organizations/org_id format." type = string diff --git a/modules/organization-policy/README.md b/modules/organization-policy/README.md deleted file mode 100644 index 3b9141704..000000000 --- a/modules/organization-policy/README.md +++ /dev/null @@ -1,166 +0,0 @@ -# Google Cloud Organization Policy - -This module allows creation and management of [GCP Organization Policies](https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints) by defining them in a well formatted `yaml` files or with HCL. - -Yaml based factory can simplify centralized management of Org Policies for a DevSecOps team by providing a simple way to define/structure policies and exclusions. - -> **_NOTE:_** This module uses experimental feature `module_variable_optional_attrs` which will be included into [terraform release 1.3](https://github.com/hashicorp/terraform/releases/tag/v1.3.0-alpha20220706). - -## Example - -### Terraform code - -```hcl -# using configuration provided in a set of yaml files -module "org-policy-factory" { - source = "./fabric/modules/organization-policy" - - config_directory = "./policies" -} - -# using configuration provided in the module variable -module "org-policy" { - source = "./fabric/modules/organization-policy" - - policies = { - "folders/1234567890" = { - # enforce boolean policy with no conditions - "iam.disableServiceAccountKeyUpload" = { - rules = [ - { - enforce = true - } - ] - }, - # Deny All for compute.vmCanIpForward policy - "compute.vmCanIpForward" = { - inherit_from_parent = false - rules = [ - deny = [] # stands for deny_all - ] - } - }, - "organizations/1234567890" = { - # allow only internal ingress when match condition env=prod - "run.allowedIngress" = { - rules = [ - { - allow = ["internal"] - condition = { - description= "allow ingress" - expression = "resource.matchTag('123456789/environment', 'prod')" - title = "allow-for-prod-org" - } - } - ] - } - } - } -} -# tftest skip -``` - -## Org Policy definition format and structure - -### Structure of `policies` variable - -```hcl -policies = { - "parent_id" = { # parent id in format projects/project-id, folders/1234567890 or organizations/1234567890. - "policy_name" = { # policy constraint id, for example compute.vmExternalIpAccess. - inherit_from_parent = true|false # (Optional) Only for list constraints. Determines the inheritance behavior for this policy. - reset = true|false # (Optional) Ignores policies set above this resource and restores the constraint_default enforcement behavior. - rules = [ # Up to 10 PolicyRules are allowed. - { - allow = ["value1", "value2"] # (Optional) Only for list constraints. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values - denyl = ["value3", "value4"] # (Optional) Only for list constraints. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values - enforce = true|false # (Optional) Only for boolean constraints. If true, then the Policy is enforced. - condition = { # (Optional) A condition which determines whether this rule is used in the evaluation of the policy. - description = "Condition description" # (Optional) - expression = "Condition expression" # (Optional) For example "resource.matchTag('123456789/environment', 'prod')". - location = "policy-error.log" # (Optional) String indicating the location of the expression for error reporting. - title = "condition-title" # (Optional) - } - } - ] - } - } -} -# tftest skip -``` - -### Structure of configuration provided in a yaml file/s - -Configuration should be placed in a set of yaml files in the config directory. Policy entry structure as follows: - -```yaml -parent_id: # parent id in format projects/project-id, folders/1234567890 or organizations/1234567890. - policy_name1: # policy constraint id, for example compute.vmExternalIpAccess. - inherit_from_parent: true|false # (Optional) Only for list constraints. Determines the inheritance behavior for this policy. - reset: true|false # (Optional) Ignores policies set above this resource and restores the constraint_default enforcement behavior. - rules: - - allow: ["value1", "value2"] # (Optional) Only for list constraints. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values - deny: ["value3", "value4"] # (Optional) Only for list constraints. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values - enforce: true|false # (Optional) Only for boolean constraints. If true, then the Policy is enforced. - condition: # (Optional) A condition which determines whether this rule is used in the evaluation of the policy. - description: Condition description # (Optional) - expression: Condition expression # (Optional) For example resource.matchTag("123456789/environment", "prod") - location: policy-error.log # (Optional) String indicating the location of the expression for error reporting. - title: condition-title # (Optional) -``` - -Module allows policies to be distributed into multiple yaml files for a better management and navigation. - -```bash -├── org-policies -│ ├── baseline.yaml -│   ├── image-import-projects.yaml -│   └── exclusions.yaml -``` - -Organization policies example yaml configuration - -```bash -cat ./policies/baseline.yaml -organizations/1234567890: - constraints/compute.vmExternalIpAccess: - rules: - - deny: [] # Stands for deny_all = true -folders/1234567890: - compute.vmCanIpForward: - inherit_from_parent: false - reset: false - rules: - - allow: [] # Stands for allow_all = true -projects/my-project-id: - run.allowedIngress: - inherit_from_parent: true - rules: - - allow: ['internal'] # Stands for values.allowed_values - condition: - description: allow internal ingress - expression: resource.matchTag("123456789/environment", "prod") - location: test.log - title: allow-for-prod - iam.allowServiceAccountCredentialLifetimeExtension: - rules: - - deny: [] # Stands for deny_all = true - compute.disableGlobalLoadBalancing: - reset: true -``` - - -## Variables - -| name | description | type | required | default | -|---|---|:---:|:---:|:---:| -| [config_directory](variables.tf#L17) | Paths to a folder where organization policy configs are stored in yaml format. Files suffix must be `.yaml`. | string | | null | -| [policies](variables.tf#L23) | Organization policies keyed by parent in format `projects/project-id`, `folders/1234567890` or `organizations/1234567890`. | map(map(object({…}))) | | {} | - -## Outputs - -| name | description | sensitive | -|---|---|:---:| -| [policies](outputs.tf#L17) | Organization policies. | | - - diff --git a/modules/organization-policy/main.tf b/modules/organization-policy/main.tf deleted file mode 100644 index 960a8462e..000000000 --- a/modules/organization-policy/main.tf +++ /dev/null @@ -1,102 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - -locals { - policy_files = var.config_directory == null ? [] : concat( - [ - for config_file in fileset("${path.root}/${var.config_directory}", "**/*.yaml") : - "${path.root}/${var.config_directory}/${config_file}" - ] - ) - - policies_raw = merge( - merge( - [ - for config_file in local.policy_files : - try(yamldecode(file(config_file)), {}) - ]... - ), var.policies) - - policies_list = flatten([ - for parent, policies in local.policies_raw : [ - for policy_name, policy in policies : { - parent = parent, - policy_name = policy_name, - inherit_from_parent = try(policy["inherit_from_parent"], null), - reset = try(policy["reset"], null), - rules = [ - for rule in try(policy["rules"], []) : { - allow_all = try(length(rule["allow"]), -1) == 0 ? "TRUE" : null - deny_all = try(length(rule["deny"]), -1) == 0 ? "TRUE" : null - enforce = try(rule["enforce"], null) == true ? "TRUE" : try( - rule["enforce"], null) == false ? "FALSE" : null, - condition = try(rule["condition"], null) != null ? { - description = try(rule["condition"]["description"], null), - expression = try(rule["condition"]["expression"], null), - location = try(rule["condition"]["location"], null), - title = try(rule["condition"]["title"], null) - } : null, - values = try(length(rule["allow"]), 0) > 0 || try(length(rule["deny"]), 0) > 0 ? { - allowed_values = try(length(rule["allow"]), 0) > 0 ? rule["allow"] : null - denied_values = try(length(rule["deny"]), 0) > 0 ? rule["deny"] : null - } : null - } - ] - } - ] - ]) - - policies_map = { - for item in local.policies_list : - format("%s-%s", item["parent"], item["policy_name"]) => item - } -} - -resource "google_org_policy_policy" "primary" { - for_each = local.policies_map - name = format("%s/policies/%s", each.value.parent, each.value.policy_name) - parent = each.value.parent - - spec { - inherit_from_parent = each.value.inherit_from_parent - reset = each.value.reset - dynamic "rules" { - for_each = each.value.rules - content { - allow_all = rules.value.allow_all - deny_all = rules.value.deny_all - enforce = rules.value.enforce - dynamic "condition" { - for_each = rules.value.condition != null ? [""] : [] - content { - description = rules.value.condition.description - expression = rules.value.condition.expression - location = rules.value.condition.location - title = rules.value.condition.title - } - } - dynamic "values" { - for_each = rules.value.values != null ? [""] : [] - content { - allowed_values = rules.value.values.allowed_values - denied_values = rules.value.values.denied_values - } - } - } - } - } -} diff --git a/modules/organization-policy/outputs.tf b/modules/organization-policy/outputs.tf deleted file mode 100644 index 6134d8760..000000000 --- a/modules/organization-policy/outputs.tf +++ /dev/null @@ -1,20 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -output "policies" { - description = "Organization policies." - value = google_org_policy_policy.primary -} diff --git a/modules/organization-policy/variables.tf b/modules/organization-policy/variables.tf deleted file mode 100644 index ff842dd98..000000000 --- a/modules/organization-policy/variables.tf +++ /dev/null @@ -1,45 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -variable "config_directory" { - description = "Paths to a folder where organization policy configs are stored in yaml format. Files suffix must be `.yaml`." - type = string - default = null -} - -variable "policies" { - description = "Organization policies keyed by parent in format `projects/project-id`, `folders/1234567890` or `organizations/1234567890`." - type = map(map(object({ - inherit_from_parent = optional(bool) # List policy only. - reset = optional(bool) - rules = optional( - list(object({ - allow = optional(list(string)) # List policy only. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values - deny = optional(list(string)) # List policy only. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values - enforce = optional(bool) # Boolean policy only. - condition = optional( - object({ - description = optional(string) - expression = optional(string) - location = optional(string) - title = optional(string) - }) - ) - })) - ) - }))) - default = {} -} diff --git a/modules/organization-policy/versions.tf b/modules/organization-policy/versions.tf deleted file mode 100644 index 286536a65..000000000 --- a/modules/organization-policy/versions.tf +++ /dev/null @@ -1,29 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# https://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -terraform { - required_version = ">= 1.3.1" - required_providers { - google = { - source = "hashicorp/google" - version = ">= 4.40.0" # tftest - } - google-beta = { - source = "hashicorp/google-beta" - version = ">= 4.40.0" # tftest - } - } -} - - diff --git a/modules/organization/README.md b/modules/organization/README.md index 3c57b7434..84a7da84d 100644 --- a/modules/organization/README.md +++ b/modules/organization/README.md @@ -76,6 +76,10 @@ If you set audit policies via the `iam_audit_config_authoritative` variable, be Some care must also be takend with the `groups_iam` variable (and in some situations with the additive variables) to ensure that variable keys are static values, so that Terraform is able to compute the dependency graph. +### Organization policy factory + +See the [organization policy factory in the project module](../project#organization-policy-factory). + ## Hierarchical firewall policies Hirerarchical firewall policies can be managed in two ways: @@ -336,8 +340,9 @@ module "org" { | [logging_exclusions](variables.tf#L122) | Logging exclusions for this organization in the form {NAME -> FILTER}. | map(string) | | {} | | [logging_sinks](variables.tf#L129) | Logging sinks to create for this organization. | map(object({…})) | | {} | | [org_policies](variables.tf#L151) | Organization policies applied to this organization keyed by policy name. | map(object({…})) | | {} | -| [tag_bindings](variables.tf#L200) | Tag bindings for this organization, in key => tag value id format. | map(string) | | null | -| [tags](variables.tf#L206) | Tags by key name. The `iam` attribute behaves like the similarly named one at module level. | map(object({…})) | | null | +| [org_policies_data_path](variables.tf#L200) | Path containing org policies in YAML format. | string | | null | +| [tag_bindings](variables.tf#L206) | Tag bindings for this organization, in key => tag value id format. | map(string) | | null | +| [tags](variables.tf#L212) | Tags by key name. The `iam` attribute behaves like the similarly named one at module level. | map(object({…})) | | null | ## Outputs diff --git a/modules/organization/organization-policies.tf b/modules/organization/organization-policies.tf index 4dba1c85e..fcde16588 100644 --- a/modules/organization/organization-policies.tf +++ b/modules/organization/organization-policies.tf @@ -17,8 +17,57 @@ # tfdoc:file:description Organization-level organization policies. locals { + _factory_data_raw = ( + var.org_policies_data_path == null + ? tomap({}) + : merge([ + for f in fileset(var.org_policies_data_path, "*.yaml") : + yamldecode(file("${var.org_policies_data_path}/${f}")) + ]...) + ) + + # simulate applying defaults to data coming from yaml files + _factory_data = { + for k, v in local._factory_data_raw : + k => { + inherit_from_parent = try(v.inherit_from_parent, null) + reset = try(v.reset, null) + allow = can(v.allow) ? { + all = try(v.allow.all, null) + values = try(v.allow.values, null) + } : null + deny = can(v.deny) ? { + all = try(v.deny.all, null) + values = try(v.deny.values, null) + } : null + enforce = try(v.enforce, true) + + rules = [ + for r in try(v.rules, []) : { + allow = can(r.allow) ? { + all = try(r.allow.all, null) + values = try(r.allow.values, null) + } : null + deny = can(r.deny) ? { + all = try(r.deny.all, null) + values = try(r.deny.values, null) + } : null + enforce = try(r.enforce, true) + condition = { + description = try(r.condition.description, null) + expression = try(r.condition.expression, null) + location = try(r.condition.location, null) + title = try(r.condition.title, null) + } + } + ] + } + } + + _org_policies = merge(local._factory_data, var.org_policies) + org_policies = { - for k, v in var.org_policies : + for k, v in local._org_policies : k => merge(v, { name = "${var.organization_id}/policies/${k}" parent = var.organization_id diff --git a/modules/organization/variables.tf b/modules/organization/variables.tf index 7499d6d67..7093a1188 100644 --- a/modules/organization/variables.tf +++ b/modules/organization/variables.tf @@ -197,6 +197,12 @@ variable "organization_id" { } } +variable "org_policies_data_path" { + description = "Path containing org policies in YAML format." + type = string + default = null +} + variable "tag_bindings" { description = "Tag bindings for this organization, in key => tag value id format." type = map(string) diff --git a/modules/project/README.md b/modules/project/README.md index dbb66fcd1..37af720c9 100644 --- a/modules/project/README.md +++ b/modules/project/README.md @@ -211,6 +211,71 @@ module "project" { # tftest modules=1 resources=10 ``` +### Organization policy factory + +Organization policies can be loaded from a directory containing YAML files where each file defines one or more constraints. The structure of the YAML files is exactly the same as the `org_policies` variable. + +Note that contraints defined via `org_policies` take precedence over those in `org_policies_data_path`. In other words, if you specify the same contraint in a YAML file *and* in the `org_policies` variable, the latter will take priority. + +The example below deploys a few organization policies split between two YAML files. + +```hcl +module "folder" { + source = "./fabric/modules/folder" + parent = "organizations/1234567890" + name = "Folder name" + org_policies_data_path = "/my/path" +} +# tftest skip +``` + +```yaml +# /my/path/boolean.yaml +iam.disableServiceAccountKeyCreation: + enforce: true + +iam.disableServiceAccountKeyUpload: + enforce: false + rules: + - condition: + expression: resource.matchTagId("tagKeys/1234", "tagValues/1234") + title: condition + description: test condition + location: xxx + enforce: true +``` + +```yaml +# /my/path/list.yaml +compute.vmExternalIpAccess: + deny: + all: true + +iam.allowedPolicyMemberDomains: + allow: + values: + - C0xxxxxxx + - C0yyyyyyy + +compute.restrictLoadBalancerCreationForTypes: + deny: + values: ["in:EXTERNAL"] + rules: + - condition: + expression: resource.matchTagId("tagKeys/1234", "tagValues/1234") + title: condition + description: test condition + allow: + values: ["in:EXTERNAL"] + - condition: + expression: resource.matchTagId("tagKeys/12345", "tagValues/12345") + title: condition2 + description: test condition2 + allow: + all: true +``` + + ## Logging Sinks ```hcl @@ -407,21 +472,22 @@ output "compute_robot" { | [logging_sinks](variables.tf#L102) | Logging sinks to create for this project. | map(object({…})) | | {} | | [metric_scopes](variables.tf#L124) | List of projects that will act as metric scopes for this project. | list(string) | | [] | | [org_policies](variables.tf#L136) | Organization policies applied to this project keyed by policy name. | map(object({…})) | | {} | -| [oslogin](variables.tf#L176) | Enable OS Login. | bool | | false | -| [oslogin_admins](variables.tf#L182) | List of IAM-style identities that will be granted roles necessary for OS Login administrators. | list(string) | | [] | -| [oslogin_users](variables.tf#L190) | List of IAM-style identities that will be granted roles necessary for OS Login users. | list(string) | | [] | -| [parent](variables.tf#L197) | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | string | | null | -| [prefix](variables.tf#L207) | Prefix used to generate project id and name. | string | | null | -| [project_create](variables.tf#L213) | Create project. When set to false, uses a data source to reference existing project. | bool | | true | -| [service_config](variables.tf#L219) | Configure service API activation. | object({…}) | | {…} | -| [service_encryption_key_ids](variables.tf#L231) | Cloud KMS encryption key in {SERVICE => [KEY_URL]} format. | map(list(string)) | | {} | -| [service_perimeter_bridges](variables.tf#L238) | Name of VPC-SC Bridge perimeters to add project into. See comment in the variables file for format. | list(string) | | null | -| [service_perimeter_standard](variables.tf#L245) | Name of VPC-SC Standard perimeter to add project into. See comment in the variables file for format. | string | | null | -| [services](variables.tf#L251) | Service APIs to enable. | list(string) | | [] | -| [shared_vpc_host_config](variables.tf#L257) | Configures this project as a Shared VPC host project (mutually exclusive with shared_vpc_service_project). | object({…}) | | null | -| [shared_vpc_service_config](variables.tf#L266) | Configures this project as a Shared VPC service project (mutually exclusive with shared_vpc_host_config). | object({…}) | | null | -| [skip_delete](variables.tf#L276) | Allows the underlying resources to be destroyed without destroying the project itself. | bool | | false | -| [tag_bindings](variables.tf#L282) | Tag bindings for this project, in key => tag value id format. | map(string) | | null | +| [org_policies_data_path](variables.tf#L176) | Path containing org policies in YAML format. | string | | null | +| [oslogin](variables.tf#L182) | Enable OS Login. | bool | | false | +| [oslogin_admins](variables.tf#L188) | List of IAM-style identities that will be granted roles necessary for OS Login administrators. | list(string) | | [] | +| [oslogin_users](variables.tf#L196) | List of IAM-style identities that will be granted roles necessary for OS Login users. | list(string) | | [] | +| [parent](variables.tf#L203) | Parent folder or organization in 'folders/folder_id' or 'organizations/org_id' format. | string | | null | +| [prefix](variables.tf#L213) | Prefix used to generate project id and name. | string | | null | +| [project_create](variables.tf#L219) | Create project. When set to false, uses a data source to reference existing project. | bool | | true | +| [service_config](variables.tf#L225) | Configure service API activation. | object({…}) | | {…} | +| [service_encryption_key_ids](variables.tf#L237) | Cloud KMS encryption key in {SERVICE => [KEY_URL]} format. | map(list(string)) | | {} | +| [service_perimeter_bridges](variables.tf#L244) | Name of VPC-SC Bridge perimeters to add project into. See comment in the variables file for format. | list(string) | | null | +| [service_perimeter_standard](variables.tf#L251) | Name of VPC-SC Standard perimeter to add project into. See comment in the variables file for format. | string | | null | +| [services](variables.tf#L257) | Service APIs to enable. | list(string) | | [] | +| [shared_vpc_host_config](variables.tf#L263) | Configures this project as a Shared VPC host project (mutually exclusive with shared_vpc_service_project). | object({…}) | | null | +| [shared_vpc_service_config](variables.tf#L272) | Configures this project as a Shared VPC service project (mutually exclusive with shared_vpc_host_config). | object({…}) | | null | +| [skip_delete](variables.tf#L282) | Allows the underlying resources to be destroyed without destroying the project itself. | bool | | false | +| [tag_bindings](variables.tf#L288) | Tag bindings for this project, in key => tag value id format. | map(string) | | null | ## Outputs diff --git a/modules/project/organization-policies.tf b/modules/project/organization-policies.tf index 696bba75b..d64dd9560 100644 --- a/modules/project/organization-policies.tf +++ b/modules/project/organization-policies.tf @@ -17,8 +17,57 @@ # tfdoc:file:description Project-level organization policies. locals { + _factory_data_raw = ( + var.org_policies_data_path == null + ? tomap({}) + : merge([ + for f in fileset(var.org_policies_data_path, "*.yaml") : + yamldecode(file("${var.org_policies_data_path}/${f}")) + ]...) + ) + + # simulate applying defaults to data coming from yaml files + _factory_data = { + for k, v in local._factory_data_raw : + k => { + inherit_from_parent = try(v.inherit_from_parent, null) + reset = try(v.reset, null) + allow = can(v.allow) ? { + all = try(v.allow.all, null) + values = try(v.allow.values, null) + } : null + deny = can(v.deny) ? { + all = try(v.deny.all, null) + values = try(v.deny.values, null) + } : null + enforce = try(v.enforce, true) + + rules = [ + for r in try(v.rules, []) : { + allow = can(r.allow) ? { + all = try(r.allow.all, null) + values = try(r.allow.values, null) + } : null + deny = can(r.deny) ? { + all = try(r.deny.all, null) + values = try(r.deny.values, null) + } : null + enforce = try(r.enforce, true) + condition = { + description = try(r.condition.description, null) + expression = try(r.condition.expression, null) + location = try(r.condition.location, null) + title = try(r.condition.title, null) + } + } + ] + } + } + + _org_policies = merge(local._factory_data, var.org_policies) + org_policies = { - for k, v in var.org_policies : + for k, v in local._org_policies : k => merge(v, { name = "projects/${local.project.project_id}/policies/${k}" parent = "projects/${local.project.project_id}" diff --git a/modules/project/variables.tf b/modules/project/variables.tf index a58affc92..7cd36c82b 100644 --- a/modules/project/variables.tf +++ b/modules/project/variables.tf @@ -173,6 +173,12 @@ variable "org_policies" { nullable = false } +variable "org_policies_data_path" { + description = "Path containing org policies in YAML format." + type = string + default = null +} + variable "oslogin" { description = "Enable OS Login." type = bool diff --git a/tests/modules/folder/fixture/main.tf b/tests/modules/folder/fixture/main.tf index 8290f82ec..a347f61bb 100644 --- a/tests/modules/folder/fixture/main.tf +++ b/tests/modules/folder/fixture/main.tf @@ -27,4 +27,5 @@ module "test" { logging_sinks = var.logging_sinks logging_exclusions = var.logging_exclusions org_policies = var.org_policies + org_policies_data_path = var.org_policies_data_path } diff --git a/tests/modules/folder/fixture/variables.tf b/tests/modules/folder/fixture/variables.tf index 7c03e0568..e2d7a293b 100644 --- a/tests/modules/folder/fixture/variables.tf +++ b/tests/modules/folder/fixture/variables.tf @@ -58,3 +58,8 @@ variable "org_policies" { type = any default = {} } + +variable "org_policies_data_path" { + type = any + default = null +} diff --git a/tests/modules/folder/test_plan_org_policies.py b/tests/modules/folder/test_plan_org_policies.py index f84d50fb0..0a6b97290 100644 --- a/tests/modules/folder/test_plan_org_policies.py +++ b/tests/modules/folder/test_plan_org_policies.py @@ -12,31 +12,106 @@ # See the License for the specific language governing permissions and # limitations under the License. +import hcl2 +import yaml + +BOOLEAN_POLICIES = '''{ + "iam.disableServiceAccountKeyCreation" = { + enforce = true + } + "iam.disableServiceAccountKeyUpload" = { + enforce = false + rules = [ + { + condition = { + expression = "resource.matchTagId(aa, bb)" + title = "condition" + description = "test condition" + location = "xxx" + } + enforce = true + } + ] + } +}''' + +LIST_POLICIES = '''{ + "compute.vmExternalIpAccess" = { + deny = { all = true } + } + "iam.allowedPolicyMemberDomains" = { + allow = { + values = ["C0xxxxxxx", "C0yyyyyyy"] + } + } + "compute.restrictLoadBalancerCreationForTypes" = { + deny = { values = ["in:EXTERNAL"] } + rules = [ + { + condition = { + expression = "resource.matchTagId(aa, bb)" + title = "condition" + description = "test condition" + location = "xxx" + } + allow = { + values = ["EXTERNAL_1"] + } + }, + { + condition = { + expression = "resource.matchTagId(cc, dd)" + title = "condition2" + description = "test condition2" + location = "xxx" + } + allow = { + all = true + } + } + ] + } +}''' + def test_policy_boolean(plan_runner): "Test boolean org policy." - policies = '''{ - "iam.disableServiceAccountKeyCreation" = { - enforce = true - } - "iam.disableServiceAccountKeyUpload" = { - enforce = false - rules = [ - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/1234\\", \\"tagValues/1234\\")" - title = "condition" - description = "test condition" - location = "xxx" - } - enforce = true - } - ] - } - }''' - _, resources = plan_runner(org_policies=policies) - assert len(resources) == 3 + _, resources = plan_runner(org_policies=BOOLEAN_POLICIES) + validate_policy_boolean_resources(resources) + +def test_policy_list(plan_runner): + "Test list org policy." + _, resources = plan_runner(org_policies=LIST_POLICIES) + validate_policy_list_resources(resources) + + +def test_policy_boolean_factory(plan_runner, tmp_path): + # convert hcl policies to yaml + hcl_policies = f'p = {BOOLEAN_POLICIES}' + yaml_policies = yaml.dump(hcl2.loads(hcl_policies)['p']) + + yaml_file = tmp_path / 'policies.yaml' + yaml_file.write_text(yaml_policies) + + _, resources = plan_runner(org_policies_data_path=f'"{tmp_path}"') + validate_policy_boolean_resources(resources) + + +def test_policy_list_factory(plan_runner, tmp_path): + # convert hcl policies to yaml + hcl_policies = f'p = {LIST_POLICIES}' + yaml_policies = yaml.dump(hcl2.loads(hcl_policies)['p']) + + yaml_file = tmp_path / 'policies.yaml' + yaml_file.write_text(yaml_policies) + + _, resources = plan_runner(org_policies_data_path=f'"{tmp_path}"') + validate_policy_list_resources(resources) + + +def validate_policy_boolean_resources(resources): + assert len(resources) == 3 policies = [r for r in resources if r['type'] == 'google_org_policy_policy'] assert len(policies) == 2 @@ -76,7 +151,7 @@ def test_policy_boolean(plan_runner): 'allow_all': None, 'condition': [{ 'description': 'test condition', - 'expression': 'resource.matchTagId("tagKeys/1234", "tagValues/1234")', + 'expression': 'resource.matchTagId(aa, bb)', 'location': 'xxx', 'title': 'condition' }], @@ -86,46 +161,7 @@ def test_policy_boolean(plan_runner): } -def test_policy_list(plan_runner): - "Test list org policy." - policies = '''{ - "compute.vmExternalIpAccess" = { - deny = { all = true } - } - "iam.allowedPolicyMemberDomains" = { - allow = { - values = ["C0xxxxxxx", "C0yyyyyyy"] - } - } - "compute.restrictLoadBalancerCreationForTypes" = { - deny = { values = ["in:EXTERNAL"] } - rules = [ - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/1234\\", \\"tagValues/1234\\")" - title = "condition" - description = "test condition" - location = "xxx" - } - allow = { - values = ["EXTERNAL_1"] - } - }, - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/12345\\", \\"tagValues/12345\\")" - title = "condition2" - description = "test condition2" - location = "xxx" - } - allow = { - all = true - } - } - ] - } - }''' - _, resources = plan_runner(org_policies=policies) +def validate_policy_list_resources(resources): assert len(resources) == 4 policies = [r for r in resources if r['type'] == 'google_org_policy_policy'] @@ -193,7 +229,7 @@ def test_policy_list(plan_runner): 'allow_all': None, 'condition': [{ 'description': 'test condition', - 'expression': 'resource.matchTagId("tagKeys/1234", "tagValues/1234")', + 'expression': 'resource.matchTagId(aa, bb)', 'location': 'xxx', 'title': 'condition' }], @@ -208,14 +244,10 @@ def test_policy_list(plan_runner): assert p3['rules'][2] == { 'allow_all': 'TRUE', 'condition': [{ - 'description': - 'test condition2', - 'expression': - 'resource.matchTagId("tagKeys/12345", "tagValues/12345")', - 'location': - 'xxx', - 'title': - 'condition2' + 'description': 'test condition2', + 'expression': 'resource.matchTagId(cc, dd)', + 'location': 'xxx', + 'title': 'condition2' }], 'deny_all': None, 'enforce': None, diff --git a/tests/modules/organization/fixture/main.tf b/tests/modules/organization/fixture/main.tf index 13f8e335f..4f5df9e26 100644 --- a/tests/modules/organization/fixture/main.tf +++ b/tests/modules/organization/fixture/main.tf @@ -29,6 +29,7 @@ module "test" { logging_sinks = var.logging_sinks logging_exclusions = var.logging_exclusions org_policies = var.org_policies + org_policies_data_path = var.org_policies_data_path tag_bindings = var.tag_bindings tags = var.tags } diff --git a/tests/modules/organization/fixture/variables.tf b/tests/modules/organization/fixture/variables.tf index f56e51dcc..2508fb062 100644 --- a/tests/modules/organization/fixture/variables.tf +++ b/tests/modules/organization/fixture/variables.tf @@ -74,6 +74,11 @@ variable "org_policies" { default = {} } +variable "org_policies_data_path" { + type = any + default = null +} + variable "tag_bindings" { type = any default = null diff --git a/tests/modules/organization/test_plan_org_policies.py b/tests/modules/organization/test_plan_org_policies.py index 33ee7e3f3..053457ce4 100644 --- a/tests/modules/organization/test_plan_org_policies.py +++ b/tests/modules/organization/test_plan_org_policies.py @@ -15,31 +15,106 @@ import difflib from pathlib import Path +import hcl2 +import yaml + +BOOLEAN_POLICIES = '''{ + "iam.disableServiceAccountKeyCreation" = { + enforce = true + } + "iam.disableServiceAccountKeyUpload" = { + enforce = false + rules = [ + { + condition = { + expression = "resource.matchTagId(aa, bb)" + title = "condition" + description = "test condition" + location = "xxx" + } + enforce = true + } + ] + } +}''' + +LIST_POLICIES = '''{ + "compute.vmExternalIpAccess" = { + deny = { all = true } + } + "iam.allowedPolicyMemberDomains" = { + allow = { + values = ["C0xxxxxxx", "C0yyyyyyy"] + } + } + "compute.restrictLoadBalancerCreationForTypes" = { + deny = { values = ["in:EXTERNAL"] } + rules = [ + { + condition = { + expression = "resource.matchTagId(aa, bb)" + title = "condition" + description = "test condition" + location = "xxx" + } + allow = { + values = ["EXTERNAL_1"] + } + }, + { + condition = { + expression = "resource.matchTagId(cc, dd)" + title = "condition2" + description = "test condition2" + location = "xxx" + } + allow = { + all = true + } + } + ] + } +}''' + def test_policy_boolean(plan_runner): "Test boolean org policy." - policies = '''{ - "iam.disableServiceAccountKeyCreation" = { - enforce = true - } - "iam.disableServiceAccountKeyUpload" = { - enforce = false - rules = [ - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/1234\\", \\"tagValues/1234\\")" - title = "condition" - description = "test condition" - location = "xxx" - } - enforce = true - } - ] - } - }''' - _, resources = plan_runner(org_policies=policies) - assert len(resources) == 2 + _, resources = plan_runner(org_policies=BOOLEAN_POLICIES) + validate_policy_boolean_resources(resources) + +def test_policy_list(plan_runner): + "Test list org policy." + _, resources = plan_runner(org_policies=LIST_POLICIES) + validate_policy_list_resources(resources) + + +def test_policy_boolean_factory(plan_runner, tmp_path): + # convert hcl policies to yaml + hcl_policies = f'p = {BOOLEAN_POLICIES}' + yaml_policies = yaml.dump(hcl2.loads(hcl_policies)['p']) + + yaml_file = tmp_path / 'policies.yaml' + yaml_file.write_text(yaml_policies) + + _, resources = plan_runner(org_policies_data_path=f'"{tmp_path}"') + validate_policy_boolean_resources(resources) + + +def test_policy_list_factory(plan_runner, tmp_path): + # convert hcl policies to yaml + hcl_policies = f'p = {LIST_POLICIES}' + yaml_policies = yaml.dump(hcl2.loads(hcl_policies)['p']) + + yaml_file = tmp_path / 'policies.yaml' + yaml_file.write_text(yaml_policies) + + _, resources = plan_runner(org_policies_data_path=f'"{tmp_path}"') + validate_policy_list_resources(resources) + + +def validate_policy_boolean_resources(resources): + assert len(resources) == 2 policies = [r for r in resources if r['type'] == 'google_org_policy_policy'] assert len(policies) == 2 assert all( @@ -81,7 +156,7 @@ def test_policy_boolean(plan_runner): 'allow_all': None, 'condition': [{ 'description': 'test condition', - 'expression': 'resource.matchTagId("tagKeys/1234", "tagValues/1234")', + 'expression': 'resource.matchTagId(aa, bb)', 'location': 'xxx', 'title': 'condition' }], @@ -91,46 +166,7 @@ def test_policy_boolean(plan_runner): } -def test_policy_list(plan_runner): - "Test list org policy." - policies = '''{ - "compute.vmExternalIpAccess" = { - deny = { all = true } - } - "iam.allowedPolicyMemberDomains" = { - allow = { - values = ["C0xxxxxxx", "C0yyyyyyy"] - } - } - "compute.restrictLoadBalancerCreationForTypes" = { - deny = { values = ["in:EXTERNAL"] } - rules = [ - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/1234\\", \\"tagValues/1234\\")" - title = "condition" - description = "test condition" - location = "xxx" - } - allow = { - values = ["EXTERNAL_1"] - } - }, - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/12345\\", \\"tagValues/12345\\")" - title = "condition2" - description = "test condition2" - location = "xxx" - } - allow = { - all = true - } - } - ] - } - }''' - _, resources = plan_runner(org_policies=policies) +def validate_policy_list_resources(resources): assert len(resources) == 3 policies = [r for r in resources if r['type'] == 'google_org_policy_policy'] @@ -200,7 +236,7 @@ def test_policy_list(plan_runner): 'allow_all': None, 'condition': [{ 'description': 'test condition', - 'expression': 'resource.matchTagId("tagKeys/1234", "tagValues/1234")', + 'expression': 'resource.matchTagId(aa, bb)', 'location': 'xxx', 'title': 'condition' }], @@ -215,14 +251,10 @@ def test_policy_list(plan_runner): assert p3['rules'][2] == { 'allow_all': 'TRUE', 'condition': [{ - 'description': - 'test condition2', - 'expression': - 'resource.matchTagId("tagKeys/12345", "tagValues/12345")', - 'location': - 'xxx', - 'title': - 'condition2' + 'description': 'test condition2', + 'expression': 'resource.matchTagId(cc, dd)', + 'location': 'xxx', + 'title': 'condition2' }], 'deny_all': None, 'enforce': None, @@ -244,7 +276,7 @@ def test_policy_implementation(plan_runner): assert list(diff1) == [ '--- \n', '+++ \n', - '@@ -14,14 +14,14 @@\n', + '@@ -14,7 +14,7 @@\n', ' * limitations under the License.\n', ' */\n', ' \n', @@ -252,8 +284,10 @@ def test_policy_implementation(plan_runner): '+# tfdoc:file:description Folder-level organization policies.\n', ' \n', ' locals {\n', + ' _factory_data_raw = (\n', + '@@ -69,8 +69,8 @@\n', ' org_policies = {\n', - ' for k, v in var.org_policies :\n', + ' for k, v in local._org_policies :\n', ' k => merge(v, {\n', '- name = "projects/${local.project.project_id}/policies/${k}"\n', '- parent = "projects/${local.project.project_id}"\n', @@ -268,7 +302,7 @@ def test_policy_implementation(plan_runner): assert list(diff2) == [ '--- \n', '+++ \n', - '@@ -14,14 +14,14 @@\n', + '@@ -14,7 +14,7 @@\n', ' * limitations under the License.\n', ' */\n', ' \n', @@ -276,8 +310,10 @@ def test_policy_implementation(plan_runner): '+# tfdoc:file:description Organization-level organization policies.\n', ' \n', ' locals {\n', + ' _factory_data_raw = (\n', + '@@ -69,8 +69,8 @@\n', ' org_policies = {\n', - ' for k, v in var.org_policies :\n', + ' for k, v in local._org_policies :\n', ' k => merge(v, {\n', '- name = "${local.folder.name}/policies/${k}"\n', '- parent = local.folder.name\n', @@ -286,7 +322,7 @@ def test_policy_implementation(plan_runner): ' \n', ' is_boolean_policy = v.allow == null && v.deny == null\n', ' has_values = (\n', - '@@ -94,4 +94,12 @@\n', + '@@ -143,4 +143,12 @@\n', ' }\n', ' }\n', ' }\n', diff --git a/tests/modules/organization_policy/__init__.py b/tests/modules/organization_policy/__init__.py deleted file mode 100644 index 6d6d1266c..000000000 --- a/tests/modules/organization_policy/__init__.py +++ /dev/null @@ -1,13 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. diff --git a/tests/modules/organization_policy/fixture/main.tf b/tests/modules/organization_policy/fixture/main.tf deleted file mode 100644 index 09a09267c..000000000 --- a/tests/modules/organization_policy/fixture/main.tf +++ /dev/null @@ -1,22 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -module "org-policy" { - source = "../../../../modules/organization-policy" - - config_directory = var.config_directory - policies = var.policies -} diff --git a/tests/modules/organization_policy/fixture/policies/test.yaml b/tests/modules/organization_policy/fixture/policies/test.yaml deleted file mode 100644 index 4b81e524c..000000000 --- a/tests/modules/organization_policy/fixture/policies/test.yaml +++ /dev/null @@ -1,40 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -organizations/1234567890: - constraints/compute.vmExternalIpAccess: - rules: - - deny_all: true -folders/1234567890: - compute.vmCanIpForward: - inherit_from_parent: false - reset: false - rules: - - allow: [] -projects/my-project-id: - run.allowedIngress: - inherit_from_parent: true - rules: - - allow: ['internal'] - condition: - description: allow internal ingress - expression: resource.matchTag("123456789/environment", "prod") - location: test.log - title: allow-for-prod - iam.allowServiceAccountCredentialLifetimeExtension: - rules: - - deny: [] - compute.disableGlobalLoadBalancing: - reset: true diff --git a/tests/modules/organization_policy/fixture/variables.tf b/tests/modules/organization_policy/fixture/variables.tf deleted file mode 100644 index 8196bcff3..000000000 --- a/tests/modules/organization_policy/fixture/variables.tf +++ /dev/null @@ -1,46 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - -variable "config_directory" { - description = "Paths to a folder where organization policy configs are stored in yaml format. Files suffix must be `.yaml`." - type = string - default = null -} - -variable "policies" { - description = "Organization policies keyed by parent in format `projects/project-id`, `folders/1234567890` or `organizations/1234567890`." - type = map(map(object({ - inherit_from_parent = optional(bool) # List policy only. - reset = optional(bool) - rules = optional( - list(object({ - allow = optional(list(string)) # List policy only. Stands for `allow_all` if set to empty list `[]` or to `values.allowed_values` if set to a list of values - deny = optional(list(string)) # List policy only. Stands for `deny_all` if set to empty list `[]` or to `values.denied_values` if set to a list of values - enforce = optional(bool) # Boolean policy only. - condition = optional( - object({ - description = optional(string) - expression = optional(string) - location = optional(string) - title = optional(string) - }) - ) - })) - ) - }))) - default = {} -} diff --git a/tests/modules/organization_policy/test_plan.py b/tests/modules/organization_policy/test_plan.py deleted file mode 100644 index fa7e5cd7a..000000000 --- a/tests/modules/organization_policy/test_plan.py +++ /dev/null @@ -1,89 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -def test_org_policy_simple(plan_runner): - "Test vpc with no extra options." - org_policies = ( - '{' - '"folders/1234567890" = {' - ' "constraints/iam.disableServiceAccountKeyUpload" = {' - ' rules = [' - ' {' - ' enforce = true,' - ' }' - ' ]' - ' }' - ' },' - ' "organizations/1234567890" = {' - ' "run.allowedIngress" = {' - ' rules = [' - ' {' - ' allow = ["internal"],' - ' condition = {' - ' description= "allow ingress",' - ' expression = "resource.matchTag(\'123456789/environment\', \'prod\')",' - ' title = "allow-for-prod-org"' - ' }' - ' }' - ' ]' - ' }' - ' }' - '}' - ) - _, resources = plan_runner( - policies = org_policies - ) - assert len(resources) == 2 - - org_policy = [r for r in resources if r["values"] - ["name"].endswith('iam.disableServiceAccountKeyUpload')][0]["values"] - assert org_policy["parent"] == "folders/1234567890" - assert org_policy["spec"][0]["rules"][0]["enforce"] == "TRUE" - - -def test_org_policy_factory(plan_runner): - "Test yaml based configuration" - _, resources = plan_runner( - config_directory="./policies", - ) - assert len(resources) == 5 - - org_policy = [r for r in resources if r["values"] - ["name"].endswith('run.allowedIngress')][0]["values"]["spec"][0] - assert org_policy["inherit_from_parent"] == True - assert org_policy["rules"][0]["condition"][0]["title"] == "allow-for-prod" - assert set(org_policy["rules"][0]["values"][0]["allowed_values"]) == set(["internal"]) - - -def test_combined_org_policy_config(plan_runner): - "Test combined (yaml, hcl) policy configuration" - org_policies = ( - '{' - '"folders/3456789012" = {' - ' "constraints/iam.disableServiceAccountKeyUpload" = {' - ' rules = [' - ' {' - ' enforce = true' - ' }' - ' ]' - ' }' - ' }' - '}' - ) - _, resources = plan_runner( - config_directory="./policies", - policies = org_policies - ) - - assert len(resources) == 6 diff --git a/tests/modules/project/fixture/main.tf b/tests/modules/project/fixture/main.tf index 4c7441ac5..08cf49dc6 100644 --- a/tests/modules/project/fixture/main.tf +++ b/tests/modules/project/fixture/main.tf @@ -26,6 +26,7 @@ module "test" { labels = var.labels lien_reason = var.lien_reason org_policies = var.org_policies + org_policies_data_path = var.org_policies_data_path oslogin = var.oslogin oslogin_admins = var.oslogin_admins oslogin_users = var.oslogin_users diff --git a/tests/modules/project/fixture/variables.tf b/tests/modules/project/fixture/variables.tf index 236cb69f3..938433967 100644 --- a/tests/modules/project/fixture/variables.tf +++ b/tests/modules/project/fixture/variables.tf @@ -69,6 +69,11 @@ variable "org_policies" { default = {} } +variable "org_policies_data_path" { + type = any + default = null +} + variable "oslogin" { type = bool default = false diff --git a/tests/modules/project/test_plan_org_policies.py b/tests/modules/project/test_plan_org_policies.py index a9c4df68b..284944676 100644 --- a/tests/modules/project/test_plan_org_policies.py +++ b/tests/modules/project/test_plan_org_policies.py @@ -12,31 +12,106 @@ # See the License for the specific language governing permissions and # limitations under the License. +import hcl2 +import yaml + +BOOLEAN_POLICIES = '''{ + "iam.disableServiceAccountKeyCreation" = { + enforce = true + } + "iam.disableServiceAccountKeyUpload" = { + enforce = false + rules = [ + { + condition = { + expression = "resource.matchTagId(aa, bb)" + title = "condition" + description = "test condition" + location = "xxx" + } + enforce = true + } + ] + } +}''' + +LIST_POLICIES = '''{ + "compute.vmExternalIpAccess" = { + deny = { all = true } + } + "iam.allowedPolicyMemberDomains" = { + allow = { + values = ["C0xxxxxxx", "C0yyyyyyy"] + } + } + "compute.restrictLoadBalancerCreationForTypes" = { + deny = { values = ["in:EXTERNAL"] } + rules = [ + { + condition = { + expression = "resource.matchTagId(aa, bb)" + title = "condition" + description = "test condition" + location = "xxx" + } + allow = { + values = ["EXTERNAL_1"] + } + }, + { + condition = { + expression = "resource.matchTagId(cc, dd)" + title = "condition2" + description = "test condition2" + location = "xxx" + } + allow = { + all = true + } + } + ] + } +}''' + def test_policy_boolean(plan_runner): "Test boolean org policy." - policies = '''{ - "iam.disableServiceAccountKeyCreation" = { - enforce = true - } - "iam.disableServiceAccountKeyUpload" = { - enforce = false - rules = [ - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/1234\\", \\"tagValues/1234\\")" - title = "condition" - description = "test condition" - location = "xxx" - } - enforce = true - } - ] - } - }''' - _, resources = plan_runner(org_policies=policies) - assert len(resources) == 6 + _, resources = plan_runner(org_policies=BOOLEAN_POLICIES) + validate_policy_boolean_resources(resources) + +def test_policy_list(plan_runner): + "Test list org policy." + _, resources = plan_runner(org_policies=LIST_POLICIES) + validate_policy_list_resources(resources) + + +def test_policy_boolean_factory(plan_runner, tmp_path): + # convert hcl policies to yaml + hcl_policies = f'p = {BOOLEAN_POLICIES}' + yaml_policies = yaml.dump(hcl2.loads(hcl_policies)['p']) + + yaml_file = tmp_path / 'policies.yaml' + yaml_file.write_text(yaml_policies) + + _, resources = plan_runner(org_policies_data_path=f'"{tmp_path}"') + validate_policy_boolean_resources(resources) + + +def test_policy_list_factory(plan_runner, tmp_path): + # convert hcl policies to yaml + hcl_policies = f'p = {LIST_POLICIES}' + yaml_policies = yaml.dump(hcl2.loads(hcl_policies)['p']) + + yaml_file = tmp_path / 'policies.yaml' + yaml_file.write_text(yaml_policies) + + _, resources = plan_runner(org_policies_data_path=f'"{tmp_path}"') + validate_policy_list_resources(resources) + + +def validate_policy_boolean_resources(resources): + assert len(resources) == 6 policies = [r for r in resources if r['type'] == 'google_org_policy_policy'] assert len(policies) == 2 assert all(x['values']['parent'] == 'projects/my-project' for x in policies) @@ -77,7 +152,7 @@ def test_policy_boolean(plan_runner): 'allow_all': None, 'condition': [{ 'description': 'test condition', - 'expression': 'resource.matchTagId("tagKeys/1234", "tagValues/1234")', + 'expression': 'resource.matchTagId(aa, bb)', 'location': 'xxx', 'title': 'condition' }], @@ -87,46 +162,7 @@ def test_policy_boolean(plan_runner): } -def test_policy_list(plan_runner): - "Test list org policy." - policies = '''{ - "compute.vmExternalIpAccess" = { - deny = { all = true } - } - "iam.allowedPolicyMemberDomains" = { - allow = { - values = ["C0xxxxxxx", "C0yyyyyyy"] - } - } - "compute.restrictLoadBalancerCreationForTypes" = { - deny = { values = ["in:EXTERNAL"] } - rules = [ - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/1234\\", \\"tagValues/1234\\")" - title = "condition" - description = "test condition" - location = "xxx" - } - allow = { - values = ["EXTERNAL_1"] - } - }, - { - condition = { - expression = "resource.matchTagId(\\"tagKeys/12345\\", \\"tagValues/12345\\")" - title = "condition2" - description = "test condition2" - location = "xxx" - } - allow = { - all = true - } - } - ] - } - }''' - _, resources = plan_runner(org_policies=policies) +def validate_policy_list_resources(resources): assert len(resources) == 7 policies = [r for r in resources if r['type'] == 'google_org_policy_policy'] @@ -195,7 +231,7 @@ def test_policy_list(plan_runner): 'allow_all': None, 'condition': [{ 'description': 'test condition', - 'expression': 'resource.matchTagId("tagKeys/1234", "tagValues/1234")', + 'expression': 'resource.matchTagId(aa, bb)', 'location': 'xxx', 'title': 'condition' }], @@ -210,14 +246,10 @@ def test_policy_list(plan_runner): assert p3['rules'][2] == { 'allow_all': 'TRUE', 'condition': [{ - 'description': - 'test condition2', - 'expression': - 'resource.matchTagId("tagKeys/12345", "tagValues/12345")', - 'location': - 'xxx', - 'title': - 'condition2' + 'description': 'test condition2', + 'expression': 'resource.matchTagId(cc, dd)', + 'location': 'xxx', + 'title': 'condition2' }], 'deny_all': None, 'enforce': None, diff --git a/tests/requirements.txt b/tests/requirements.txt index d14f099f2..3eb583abf 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -3,3 +3,4 @@ PyYAML>=6.0 tftest>=1.7.6 marko>=1.2.0 deepdiff>=5.7.0 +python-hcl2>=3.0.5