kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f2d4e937d1a4f2d802644773862e97a628866260
hunfabric/fast/stages/1-vpcsc
History

README.md

VPC Service Controls

This stage sets up VPC Service Controls (VPC-SC) for the whole organization and is a thing FAST-compliant wrapper on the VPC-SC module, with some minimal defaults.

Design overview and choices

The approach to VPC-SC design implemented in this stage aims at providing the simplest possible configuration that can be set to enforced mode, so as to provide immediate protection while minimizing operational complexity (which can be very high for VPC-SC).

Single perimeter with built-in extensibility

This stage uses a single VPC-SC perimeter by default, which is enough to provide protection against data exfiltration and use of credentials from outside of established boundaries, while minimizing operational toil.

The perimeter is set to dry-run mode by default, but the suggestion is to switch to enforced mode immediately after defining the initial set of access level and ingress/egress policies. This prevents the common situation where a complex design is deployed in dry-run mode, and then never enforced as the burden of addressing all violations is too high. A simpler design like the one presented here that employs very coarse access levels can be enforced quickly, and then refined iteratively as operations are streamlined and familiarity with VPC-SC quirks increases.

The stage is designed to allow defining additional perimeters via the perimeters variable, with a few caveats:

  • there's no support for perimeter bridges, if those are needed they need to be integrated via code (which is easy enough to do anyway)
  • only one set of discovered resources is supported and made available via the $resource_sets:discovered_projects context expansion
  • the factory files for access levels and ingress/egress policies use the same folder regardless of perimeter, but their inclusion in a perimeter is controlled by the perimeter-level access_levels, ingress_policies, and egress_policies attributes.

Factories for VPC-SC configuration

Restricted services, access levels, ingress and egress policies and perimeters can all be configured via YAML-based files, which allow intuitive editing and minimize the complexity of running operations.

The default setup only contains a single access level and an initial list of restricted services in the datasets/classic/access-levels folder and the datasets/classic/restricted-services.yaml file.

To configure ingress and egress policies simply add ingress_policies and/or egress_policies folders under data, or point the factories to your own folders by changing the factories_config variable values. For more details on how the factories_config interface works, refer to the documentation for stage 0.

Examples on how to write access level and policy YAML files are provided further down in this document.

Default geo-based access level

The way we set up the perimeter for broad access is via a single geo-based access level, which is configured to allow access from one or more countries and deny all other traffic coming from outside the perimeter.

The datasets/classic/access-levels/geo.yaml file serves as an example and should be edited to contain the countries you need (or replaced/removed for more granular configuration).

conditions:
  - regions:
      # replace the following country codes with those you need
      - ES
      - IT

More access levels can be of course added to better tailor the configuration to specific needs. Some use cases are addressed further down in this document.

Ingress policy for organization-level log sinks

An ingress policy that allows ingress to the perimeter for identities used in organization-level sinks is automatically created, but needs to be explicitly referenced in the perimeter via the $identity_sets:logging_identities context expansion.

This only supports sinks defined in the bootstrap stage, but it can easily be used as a reference for different, specific needs (or replaced with a policy leveraging Asset Inventory for automatic inclusion of folder-level sinks).

Perimeter membership

The set of resources protected by each perimeter can be defined in two main ways:

  • central and authoritative, where protected resources are only defined in this stage
  • delegated and additive, where perimeter is defined in this stage and resources are added separately (e.g. by a project factory)

The first approach is more secure as it does not require granting editing permission to other actors, but it's also operationally heavier as it requires adding projects to the perimeter right after creation, before many operations can be run. For example, Shared VPC attachment for a service project cannot happen until the project is in the same perimeter as its host project. The main advantage of this approach is being able to leverage the resource discovery features provided by this stage.

The second approach is more flexible, but requires delegating a measure of control over perimeters to other actors, and losing control over perimeter membership which stops being enforced by Terraform.

When using the second approach, after applying this stage, provide perimeter information in your defaults.yaml file, for example:

projects:
  overrides:
    vpc_sc:
      perimeter_name: accessPolicies/12345/servicePerimeters/default

And then apply 0-org-setup stage again. For later stages (such as networking, project factory), add the perimeter in a similar way, but there you can use context to provide perimeter:

projects:
  overrides:
    vpc_sc:
      perimeter_name: default

Caution

Do not add any resources to the perimeter in this stage when using the second approach. Any resources added in this stage will not be properly removed from perimeter, if the terraform apply is also changing the perimeter definition.

Resource discovery

If the first approach is desired in combination with resource discovery, you can simply tweak exclusions via the resource_discovery variable as the feature is enabled by default.

Discovered resources are made available via the $resource_sets:discovered_projects context expansion, which is already part of the definition of the default perimeter.

This approach is suitable for simple requirements, and provided out of the box in this stage's default configuration.

Manual resource membership

When resource discovery is not used, resources can be added to perimeters via a combination of:

  • explicit project numbers inclusion
  • individual project expansion via the $project_numbers:xxx context namespace
  • set-based project expansion via the $resource_sets:xxx context namespace

A selected number of auto-generated context expansions are generated by this stage, and described in the Context section below.This is what each of three methods above looks like in a perimeter definition.

spec:
  resources:
    # explicit reference
    - projects/123456
    # individual expansion
    - $project_numbers:iac-0
    # set expansion
    - $resource_sets:org_setup_projects

When partial control is delegated to external actors, the perimeter needs to be configured to ignore changes to member resources so as not to trigger permadiffs between different stages. This is achieved via the ignore_resource_changes perimeter attribute.

ignore_resource_changes: true
use_explicit_dry_run_spec: true
spec:
  # spec definition

Enforced vs dry-run perimeters

As mentioned above, the default configuration uses a single perimeter configured in dry-run mode. A dry-run mode perimeter has the following format.

# datasets/xxx/perimeters/myperimeter.yaml
use_explicit_dry_run_spec: true
spec:
  # perimeter definition here

A perimeter in enforced mode uses status instead of spec.

# datasets/xxx/perimeters/myperimeter.yaml
use_explicit_dry_run_spec: true
status:
  # perimeter definition here

If the dry-run and enforced configurations are different, define both explicitly in separate spec and status blocks, and set the use_explicit_dry_run_spec to true.

Context expansion

In much the same way as other FAST stages and underlying modules, context expansion is supported here in two ways:

  • static values can be defined in the stage's defaults.yaml file or context variable
  • specific values derived by other FAST stages are automatically added to the relevant context namespaces

These are the available context namespaces in this stage.

context type
iam_principals map(string)
identity_sets map(map(string))
project_numbers map(string)
resource_sets map(map(string))
service_sets map(map(string))
storage_buckets map(string)

The following sub-sections illustrate how both work.

Static contexts

Static contexts can be defined via the defaults.yaml file or the context variable, and are internally merged together and with the stage's built-in contexts.

This is an example of defining a context via defaults.yaml.

# datasets/xxx/defaults.yaml
context:
  iam_principals:
    me: user:foo@example.com
  resource_sets:
    my_projects:
      - projects/12345
      - projects/67890

These can then be reused in YAML definitions like shown below.

# datasets/xxx/perimeters/myperimeter.yaml
use_explicit_dry_run_spec: true
status:
  resources:
    - $resource_sets:my_projects

# datasets/xxx/access-levels/identity_me.yaml
conditions:
  - members:
    - $iam_principals:me

Stage-generated contexts

When this stage is used as part of a FAST installation, it consumes data made available from other stages and makes it available as context expansions. The resource discovery feature when enabled also add its own context expansion.

  • $iam_principals:service_accounts/xxx
    service accounts created in the 0-org-setup stage
  • $identity_sets:logging_identities
    set of identities used by the log sinks created in the 0-org-setup stage
  • $project_numbers:xxx
    project numbers for projects created in the 0-org-setup stage
  • $resource_sets:discovered_projects
    set of projects numbers for discovered projects (when discovery is enabled)
  • $resource_sets:org_setup_projects
    set of projects numbers for projects created in the 0-org-setup stage
  • $service_sets:restricted_services
    set of services defined in the services factory file
  • $storage_buckets:xxx
    storage buckets created in the 0-org-setup stage

How to run this stage

This stage is meant to be executed after the bootstrap stage has run, as it leverages the automation service account and bucket created there. It does not depend from any other stage and no other stage requires it, so it can be run in any order or even skipped entirely.

It's of course possible to run this stage in isolation, but that's outside the scope of this document, and you would need to refer to the code for the previous stage for the environment requirements.

Provider and Terraform variables

As all other FAST stages, the mechanism used to pass variable values and pre-built provider files from one stage to the next is also leveraged here.

The commands to link or copy the provider and terraform variable files can be get from the fast-links.sh script in the FAST stages folder, passing it a single argument with the local output files folder or GCS output bucket. The following examples demonstrate both cases, and the resulting commands that then need to be copy/pasted and run.

../fast-links.sh ~/fast-config

# File linking commands for vpc service controls stage

# provider file
ln -s ~/fast-config/fast-test-00/providers/1-vpcsc-providers.tf ./

# input files from other stages
ln -s ~/fast-config/fast-test-00/tfvars/0-globals.auto.tfvars.json ./
ln -s ~/fast-config/fast-test-00/tfvars/0-org-setup.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
ln -s ~/fast-config/fast-test-00/1-vpcsc.auto.tfvars ./

# the outputs bucket name is in the stage 0 outputs and tfvars file
../fast-links.sh gs://xxx-prod-iac-core-outputs-0

# File linking commands for vpc service controls stage

# provider file
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/providers/1-vpcsc-providers.tf ./

# input files from other stages
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-globals.auto.tfvars.json ./
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-org-setup.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/1-vpcsc.auto.tfvars ./

Impersonating the automation service account

The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The gcp-devops and organization-admins groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.

Variable configuration

Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:

  • variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the 0-globals.auto.tfvars.json file linked or copied above
  • variables which refer to resources managed by previous stages, which are prepopulated here via the 0-org-setup.auto.tfvars.json file linked or copied above
  • and finally variables that optionally control this stage's behaviour and customizations, that you are expected to configure in a custom terraform.tfvars file

The latter set is explained in the Customization sections below, and the full list can be found in the Variables table at the bottom of this document.

Note that the outputs_location variable is disabled by default, you need to explicitly set it in your terraform.tfvars file if you want output files to be generated by this stage. This is an example of doing it:

outputs_location = "~/fast-config"

Running the stage

Once provider and variable values are in place and the correct user is configured, the stage can be run:

terraform init
terraform apply

Customizations

The stage is a thin wrapper that implements a single-perimeter design via the vpc-sc module, and most of the customizations available in the module are also available here. A few examples are provided below for ease of reference.

Access policy

The stage creates the org-level access policy by default. A pre-existing policy can instead be used by populating the access_policy variable with the policy id. In tenant-level mode this is done automatically by the FAST input/output files mechanism.

Perimeters

The default perimeter is defined in the perimeters/default.yaml file within each dataset. The file can be easily customized and used as a basis to create additional perimeters.

# yaml-language-server: $schema=../../../schemas/perimeter.schema.json

use_explicit_dry_run_spec: true
# default to dr-run
spec:
  access_levels:
    - geo
  # include discovered projects
  resources:
    - $resource_sets:discovered_projects
  ingress_policies:
    - fast-org-log-sinks
  # protect the full list of services
  restricted_services:
    - $service_sets:restricted_services

Access levels

Access levels can be defined via tfvars or factory files using the same mechanism in the underlying vpc-sc module.

These are example access levels defined via tfvars:

access_levels = {
  a1 = {
    conditions = [
      { members = ["user:user1@example.com"] }
    ]
  }
  a2 = {
    combining_function = "OR"
    conditions = [
      { regions = ["IT", "FR"] },
      { ip_subnetworks = ["101.101.101.0/24"] }
    ]
  }
}

And the same defined instead via factory files.

# datasets/classic/access-levels/a1.yaml
conditions:
  - members:
    - "user:user1@example.com"

# datasets/classic/access-levels/a2.yaml
combining_function: OR
conditions:
  - regions:
    - ES
    - IT
  - ip_subnetworks:
    - 8.8.0.0/16

Remember that in order for them to deployed, access levels need to be referenced by name in the perimeter definition.

Ingress/egress policies

Like access levels, ingress and egress policies can be defined via tfvars or factory files using the same mechanism in the underlying vpc-sc module.

This is an example ingress policy defined in yaml:

# datasets/classic/ingress-policies/sa-tf-test.yaml
from:
  access_levels:
    - "*"
  identities:
    - serviceAccount:test-tf@myproject.iam.gserviceaccount.com
to:
  operations:
    - service_name: compute.googleapis.com
      method_selectors:
        - ProjectsService.Get
        - RegionsService.Get
  resources:
    - "*"

And this is an example egress policy:

# datasets/classic/egress-policies/gcs-sa-foo.yaml
from:
  identities:
    - serviceAccount:foo@myproject.iam.gserviceaccount.com
to:
  operations:
    - method_selectors:
        - "*"
      service_name: storage.googleapis.com
  resources:
    - projects/123456789

Remember that in order for them to deployed, ingress/egress policies need to be referenced by name in the perimeter definition.

Notes

Some references that might be useful in setting up this stage:

Files

name description modules resources
main.tf Module-level locals and resources. projects-data-source · vpc-sc
outputs.tf Module outputs. google_storage_bucket_object · local_file
variables-fast.tf None
variables.tf Module variables.

Variables

name description type required default producer
organization Organization details. object({…}) ✓ 0-org-setup
access_levels Access level definitions. map(object({…})) {}
access_policy Access policy id (used for tenant-level VPC-SC configurations). number null
context External context used in replacements. object({…}) {}
egress_policies Egress policy definitions that can be referenced in perimeters. map(object({…})) {}
factories_config Paths to folders that enable factory functionality. object({…}) {}
iam_principals Org-level IAM principals. map(string) {} 0-org-setup
ingress_policies Ingress policy definitions that can be referenced in perimeters. map(object({…})) {}
logging_sinks Log sinks for the organization. map(object({…})) {} 0-org-setup
perimeters Perimeter definitions. map(object({…})) {}
project_ids Project IDs. map(string) {} 0-org-setup
project_numbers Project numbers. map(number) {} 0-org-setup
resource_discovery Automatic discovery of perimeter projects. object({…}) {}
root_node Root node for the hierarchy, if running in tenant mode. string null 0-org-setup
service_accounts Org-level service accounts. map(string) {} 0-org-setup
storage_buckets Storage buckets created in the bootstrap stage. map(string) {} 0-org-setup

Outputs

name description sensitive consumers
tfvars Terraform variable files for the following stages. ✓
vpc_sc_perimeter_default Raw default perimeter resource. ✓