Factories refactor (#1843)
* factories refactor doc * Adds file schema and filesystem organization * Update 20231106-factories.md * move factories out of blueprints and create new factories README * align factory in billing-account module * align factory in dataplex-datascan module * align factory in billing-account module * align factory in net-firewall-policy module * align factory in dns-response-policy module * align factory in net-vpc-firewall module * align factory in net-vpc module * align factory variable names in FAST * remove decentralized firewall blueprint * bump terraform version * bump module versions * update top-level READMEs * move project factory to modules * fix variable names and tests * tfdoc * remove changelog link * add project factory to top-level README * fix cludrun eventarc diff * fix README * fix cludrun eventarc diff --------- Co-authored-by: Simone Ruffilli <sruffilli@google.com>
This commit is contained in:
Ludovico Magnocavallo
committed by
GitHub
GitHub
parent
8e86f0e108
commit
6941313c7d
102
modules/__docs/20231106-factories.md
Normal file
102
modules/__docs/20231106-factories.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Factories Refactor and Plan Forward
|
||||
|
||||
**authors:** [Ludo](https://github.com/ludoo)
|
||||
**last modified:** February 16, 2024
|
||||
|
||||
## Status
|
||||
|
||||
Under discussion.
|
||||
|
||||
## Context
|
||||
|
||||
Factories evolved progressively in Fabric, from the original firewall factory module, to a semi-standardized approach to management of repeated resources. This progression happened piecemeal and it's now time to define a clear strategy for factories in both Fabric and FAST, so that we can remove guesswork from new developments and provide a predictive approach to users.
|
||||
|
||||
The remainder of this section provides a summary of the current status.
|
||||
|
||||
### Modules
|
||||
|
||||
Several modules implement factories for repeated resources which are typically dependent from the main resource managed in the module:
|
||||
|
||||
- `billing-account` provides a factory for billing alert rules tied to the billing account
|
||||
- `dns-response-policy` provides a factory for rules in within the policy
|
||||
- `net-firewall-policy` provides a factory for rules within the policy
|
||||
- `net-vpc` provides a factory for subnets in the VPC
|
||||
- `net-vpc-firewall` provides a factory for VPC firewall rules
|
||||
- `organization` and `folder` provide a factory for hierarchical firewall rules within their policy
|
||||
- `organization`, `folder` and `project` provide a factory for organization policies
|
||||
|
||||
The common pattern for modules is management of *multiple resources* typically dependent from the single *main resource* managed by the module.
|
||||
|
||||
### Blueprints
|
||||
|
||||
The `factories` folder in blueprints contains a collection of factories with a fuzzier approach
|
||||
|
||||
- `bigquery-factory` manages tables and views for 1-n datasets by wrapping the `bigquery-dataset` module via simple locals
|
||||
- `cloud-identity-group-factory` manages Cloud Identity group members for 1-n groups by wrapping the `cloud-identity-group` via simple locals
|
||||
- `net-vpc-firewall-yaml` is the original factory module managing VPC firewall rules, superseded by the factory in the `net-vpc-firewall` module
|
||||
- `project-factory` combines the project, service account, and (planned) billing account and VPC modules to implement end-to-end project creation and configuration
|
||||
|
||||
There's no clear common pattern for these factories, where some could be moved to the respective module and the project factory combines a collection of modules to implement a process.
|
||||
|
||||
### FAST
|
||||
|
||||
FAST currently leverages module-level factories (organization policies, subnets, firewalls, etc.), and also provides the project factory as a dedicated level 3 stage by wrapping the relevant blueprint and localizing a few variables for the environment (`prefix`, `labels`).
|
||||
|
||||
## Proposal
|
||||
|
||||
While the current approach is reasonably clear in regards to modules, it has never been formalized in a set of guidelines that can help authors define when and how new factories would made sense.
|
||||
|
||||
On top of this, the `factories` blueprints folder contains code that that should really be moved to module-level factories, and the project factory which could/should be published directly as a FAST stage, since those are consumable as standalone modules.
|
||||
|
||||
This proposal aims at addressing the above problems.
|
||||
|
||||
### Module-level factory approach
|
||||
|
||||
The current approach for module-level factories can be summarized in a single principle:
|
||||
|
||||
> factories implemented in modules manage multiple resources which depend from one single main resource (or a small set of main resources) which are the main driver of the module.
|
||||
|
||||
For example, the module managing a firewall policy exposes a factory for its rules, or the module managing a VPC exposes a factory for its subnets. But the project module would not expose a projects factory, as one project maps to a single module invocation.
|
||||
|
||||
The proposal on factory modules then is to:
|
||||
|
||||
- align all factory variables to the same standard, outlined below
|
||||
- move the groups and bigquery factories from blueprints to the respective modules
|
||||
- eventually add more factories when it makes sense to do so (e.g. for KMS keys, service accounts, etc.)
|
||||
|
||||
The variable interface for module-level factories should use a single top-level `factory_configs` variable, whose type is an object with one or more attributes which are named according to the specific factory. This will allow composing multiple factory configurations into a single variable in FAST stages, by avoiding name overlaps. An example:
|
||||
|
||||
```hcl
|
||||
variable "factory_configs" {
|
||||
description = "Path to folder containing budget alerts data files."
|
||||
type = object({
|
||||
budgets_data_path = optional(string, "data/billing-budgets")
|
||||
})
|
||||
nullable = false
|
||||
default = {}
|
||||
}
|
||||
```
|
||||
|
||||
### Blueprint factories
|
||||
|
||||
The `factories` folder in blueprints will be emptied, and a single README left in it pointing to all the module-level and FAST stage factories available.
|
||||
|
||||
As outlined above, the existing factories will be moved to modules (bigquery and groups), FAST (project factory), or deleted (firewall rules).
|
||||
|
||||
### FAST factories
|
||||
|
||||
The only change for FAST factories will be moving the project factory from blueprints to the stage folder, and updating the path used for the environment-level wrapping stage.
|
||||
|
||||
### File schema and filesystem organization
|
||||
|
||||
Factory files schema must mimick and implement the variable interface for the module, including optionals and validation - which are implemented in code and checks.
|
||||
|
||||
With notable exceptions (currently only the `cidrs.yaml` file consumed by firewall factories), the following convention for files/directory is proposed:
|
||||
|
||||
- Factories should consume directories (vs single files)
|
||||
- All files should contain a dictionary of resources or a single resource
|
||||
- If the factory accepts one resource per file (e.g. VPC subnets), the file name should be used for the resource name and the YAML should allow defining a `name:` override
|
||||
- Files in a directory should be parsed together and flattened into a single dictionary
|
||||
|
||||
This allows developers to implement multiple resources in a single file or to use one file per resource, as they see fit.
|
||||
|
||||
Reference in New Issue
Block a user