Rename FAST stages preparing for eventual deprecation (#3298)
* renames * links * readme * docs * update pf modules tests for renames * condition_vars context in modules * data platform dataset * fix links in stage 3 docs * schema changes * schema docs * tfdoc * update duplicates check * fast legacy tests * legacy schema * fix tests
This commit is contained in:
Ludovico Magnocavallo
committed by
GitHub
GitHub
parent
1f59fd6bc7
commit
bc6950e205
@@ -11,11 +11,7 @@ The following diagram illustrates the high-level design of SecOps instance confi
|
||||
<!-- BEGIN TOC -->
|
||||
- [Design overview and choices](#design-overview-and-choices)
|
||||
- [How to run this stage](#how-to-run-this-stage)
|
||||
- [Resource management configuration](#resource-management-configuration)
|
||||
- [Provider and Terraform variables](#provider-and-terraform-variables)
|
||||
- [Impersonating the automation service account](#impersonating-the-automation-service-account)
|
||||
- [Variable configuration](#variable-configuration)
|
||||
- [Running the stage](#running-the-stage)
|
||||
- [FAST prerequisites](#fast-prerequisites)
|
||||
- [Customizations](#customizations)
|
||||
- [Data RBAC](#data-rbac)
|
||||
- [SecOps rules and reference list management](#secops-rules-and-reference-list-management)
|
||||
@@ -35,99 +31,22 @@ Some high level features of the current version of the stage are:
|
||||
- Data RBAC configuration with labels and scopes
|
||||
- IAM setup for the SecOps instance based on groups from Cloud Identity or WIF (with supports for Data RBAC)
|
||||
- Detection Rules and reference lists management via terraform (leveraging [secops-rules](../../../modules/secops-rules) module)
|
||||
- API Key setup for Webhook feeds
|
||||
- API Key setup for Webhook feeds
|
||||
- Integration with Workspace for alerts and logs ingestion via SecOps Feeds
|
||||
|
||||
## How to run this stage
|
||||
|
||||
This stage is meant to be executed after the FAST "foundational" stages: bootstrap, resource management, secops stages.
|
||||
If this stage is deployed within a FAST-based GCP organization, we recommend executing it after foundational FAST `stage-2` components like `networking` and `security`. This is the recommended flow as specific data platform features in this stage might depend on configurations from these earlier stages. Although this stage can be run independently, instructions for such a standalone setup are beyond the scope of this document.
|
||||
|
||||
It's of course possible to run this stage in isolation, refer to the *[Running in isolation](#running-in-isolation)* section below for details.
|
||||
### FAST prerequisites
|
||||
|
||||
Before running this stage, you need to make sure you have the correct credentials and permissions, and localize variables by assigning values that match your configuration.
|
||||
This stage needs specific automation resources, and permissions granted on those that allow control of selective IAM roles on specific networking and security resources.
|
||||
|
||||
### Resource management configuration
|
||||
Network permissions are needed to associate data domain or product projects to Shared VPC hosts and grant network permissions to data platform managed service accounts. They are mandatory when deploying Composer.
|
||||
|
||||
Some configuration changes are needed in resource management before this stage can be run.
|
||||
Security permissions are only needed when using CMEK encryption, to grant the relevant IAM roles to data platform service agents on the encryption keys used.
|
||||
|
||||
Make sure the stage 3 is enabled in the `data/stage-3` folder [in the resource management stage](../1-resman/data/stage-3/). As an example, this YAML definition saved as `secops-dev.yaml` enables this stage 3 for the development environment:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=../../schemas/fast-stage3.schema.json
|
||||
|
||||
short_name: secops
|
||||
environment: dev
|
||||
folder_config:
|
||||
name: Development
|
||||
parent_id: secops
|
||||
```
|
||||
|
||||
Make sure the stage 3 definitions are aligned with the environments you would like to setup for SecOps and coherent with the environments definitions in the stage 2 [2-secops](../2-secops) in order to have a dedicated stage 3 for SecOps for each environment (dev and prod as an example).
|
||||
|
||||
### Provider and Terraform variables
|
||||
|
||||
As all other FAST stages, the [mechanism used to pass variable values and pre-built provider files from one stage to the next](../0-bootstrap/README.md#output-files-and-cross-stage-variables) is also leveraged here.
|
||||
|
||||
The commands to link or copy the provider and terraform variable files can be easily derived from the `fast-links.sh` script in the FAST root folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following examples demonstrate both cases, and the resulting commands that then need to be copy/pasted and run.
|
||||
|
||||
```bash
|
||||
../fast-links.sh ~/fast-config
|
||||
|
||||
# File linking commands for GKE (dev) stage
|
||||
|
||||
# provider file
|
||||
ln -s ~/fast-config/providers/3-secops-dev-providers.tf ./
|
||||
|
||||
# input files from other stages
|
||||
ln -s ~/fast-config/tfvars/0-globals.auto.tfvars.json ./
|
||||
ln -s ~/fast-config/tfvars/0-bootstrap.auto.tfvars.json ./
|
||||
ln -s ~/fast-config/tfvars/1-resman.auto.tfvars.json ./
|
||||
ln -s ~/fast-config/tfvars/2-secops.auto.tfvars.json ./
|
||||
|
||||
# conventional place for stage tfvars (manually created)
|
||||
ln -s ~/fast-config/3-secops-dev.auto.tfvars ./
|
||||
```
|
||||
|
||||
```bash
|
||||
../fast-links.sh gs://xxx-prod-iac-core-outputs-0
|
||||
|
||||
# File linking commands for GKE (dev) stage
|
||||
|
||||
# provider file
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/providers/3-secops-dev-providers.tf ./
|
||||
|
||||
# input files from other stages
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-globals.auto.tfvars.json ./
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-bootstrap.auto.tfvars.json ./
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/1-resman.auto.tfvars.json ./
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/2-secops.auto.tfvars.json ./
|
||||
|
||||
# conventional place for stage tfvars (manually created)
|
||||
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/3-secops-dev.auto.tfvars ./
|
||||
```
|
||||
|
||||
### Impersonating the automation service account
|
||||
|
||||
The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The `gcp-devops` and `organization-admins` groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.
|
||||
|
||||
### Variable configuration
|
||||
|
||||
Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:
|
||||
|
||||
- variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the `0-globals.auto.tfvars.json` file linked or copied above
|
||||
- variables which refer to resources managed by previous stage, which are prepopulated here via the `*.auto.tfvars.json` files linked or copied above
|
||||
- and finally variables that optionally control this stage's behaviour and customizations, and can to be set in a custom `terraform.tfvars` file
|
||||
|
||||
The latter set is explained in the [Customization](#customizations) sections below, and the full list can be found in the [Variables](#variables) table at the bottom of this document.
|
||||
|
||||
### Running the stage
|
||||
|
||||
Once provider and variable values are in place and the correct user is configured, the stage can be run:
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform apply
|
||||
```
|
||||
The ["Classic FAST" dataset](../0-bootstrap/README.md#classic-fast-dataset) in the bootstrap stage contains the configuration for a development Data Platform that can be easily adapted to serve for this stage.
|
||||
|
||||
## Customizations
|
||||
|
||||
@@ -137,7 +56,7 @@ This stage is designed with few basic integrations provided out of the box which
|
||||
|
||||
This stage supports configuration of [SecOps Data RBAC](https://cloud.google.com/chronicle/docs/administration/datarbac-overview) using two separate variables:
|
||||
|
||||
- `secops_data_rbac_config`: specifies Data RBAC [label and scopes](https://cloud.google.com/chronicle/docs/administration/configure-datarbac-users) in Google SecOps
|
||||
- `secops_data_rbac_config`: specifies Data RBAC [label and scopes](https://cloud.google.com/chronicle/docs/administration/configure-datarbac-users) in Google SecOps
|
||||
- `secops_iam`: defines SecOps IAM configuration in {PRINCIPAL => {roles => [ROLES], scopes => [SCOPES]}} format referencing previously defined scopes. When scope is populated a [IAM condition](https://cloud.google.com/chronicle/docs/administration/configure-datarbac-users#assign-scope-to-users) restrict access to those scopes.
|
||||
|
||||
Example of a Data RBAC configuration is reported below.
|
||||
|
||||
Reference in New Issue
Block a user