a4eb4d24fd
* add ad for compute-vm refactor * Exclue nic_type from validated fields, add split of main.tf and template.tf * boot disk * fix examples and fixtures * attached disks * fix further examples and module-level tests * remove extra file * fix mig examples * finish refactoring variables * align fast and other modules * refactor(compute-vm): align examples and ADR with the newly implemented interface This commit addresses the remaining references of the `instance_type` and `confidential_compute` parameters in the testing environment and updates the ADR. * feat(compute-vm): add network_performance_config to instance and templates This change implements the usage of the `network_performance_tier` variable we added earlier into the actual Terraform resources. --------- Co-authored-by: Wiktor Niesiobędzki <wiktorn@google.com>
327 lines
13 KiB
Markdown
327 lines
13 KiB
Markdown
# FAST Gitlab repository management
|
|
|
|
This small extra stage allows creating and populating Gitlab groups and projects
|
|
used to host FAST stages code, including rewriting of module sources and secrets
|
|
used for private modules repository access.
|
|
|
|
It is designed for use in a Gitlab self-managed, and is only meant as a one-shot
|
|
solution with perishable state especially when used for initial population, as
|
|
you don't want Terraform to keep overwriting your changes with initial versions
|
|
of files.
|
|
|
|
## Gitlab provider credentials
|
|
|
|
A [Gitlab token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)
|
|
is needed to authenticate against their API. The token needs admin permissions,
|
|
like shown in this screenshot:
|
|
|
|
<p align="center">
|
|
<img src="gitlab_token.png" alt="Gitlab token scopes.">
|
|
</p>
|
|
|
|
Once a token is available set it in the `gitlab_config.access_token` terraform
|
|
variable before running Terraform. You might also want to update the default
|
|
Gitlab hostname and SSH port on the `gitlab_config` section.
|
|
|
|
## Variable configuration
|
|
|
|
### Gitlab Config
|
|
|
|
The `gitlab_config` variable defines where projects will be hosted.
|
|
GitLab can either be the SaaS offering (with the default hostname `gitlab.com`)
|
|
or a self-hosted instance with a custom FQDN. If hostname is set to gitlab.com, you must
|
|
also set the `saas_group` variable to specify the group path where resources will be created.
|
|
|
|
This is an example that configures a SaaS gitlab instance using `my_group/gcp` as default group :
|
|
|
|
```hcl
|
|
gitlab_config = {
|
|
access_token = "glpat-XXX"
|
|
hostname = "gitlab.com"
|
|
ssh_port = 22
|
|
saas_group = "my_group/gcp"
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
This is an example that configures an on-premise gitlab instance :
|
|
|
|
```hcl
|
|
gitlab_config = {
|
|
access_token = "glpat-XXX"
|
|
hostname = "my-gitlab.example.com"
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
### Modules project and sources
|
|
|
|
The `modules_config` variable controls creation and management of both the
|
|
Gitlab project hosting fast modules and key and secret used to access it, and
|
|
indirectly control population of initial files: if the `modules_config` variable
|
|
is not specified no module repository is know to the code, so module source
|
|
paths cannot be replaced, and initial population of files cannot happen. If the
|
|
variable is specified, an optional `source_ref` attribute can be set to the
|
|
reference used to pin modules versions.
|
|
|
|
This is an example that configures the modules project in Gitlab in a "shared"
|
|
group with source ref:
|
|
|
|
```hcl
|
|
modules_config = {
|
|
project_name = "modules"
|
|
key_config = {
|
|
create_key = true
|
|
create_secrets = true
|
|
}
|
|
group = "shared"
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
If the modules are located in a non modules only repository, use the
|
|
module_prefix attribute to set the location of your modules within the
|
|
repository:
|
|
|
|
```hcl
|
|
modules_config = {
|
|
project_name = "modules"
|
|
key_config = {
|
|
create_key = true
|
|
create_secrets = true
|
|
}
|
|
group = "shared"
|
|
module_prefix = "modules/"
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
In the above example, no key options are set so it's assumed modules will be
|
|
fetched from a public repository. If modules repository authentication is needed
|
|
the `key_config` attribute also needs to be set.
|
|
|
|
If no keypair path is specified an internally generated key will be stored as a
|
|
deploy key in the modules project, and as secrets in the stage repositories:
|
|
|
|
```hcl
|
|
modules_config = {
|
|
project_name = "modules"
|
|
key_config = {
|
|
create_key = true
|
|
create_secrets = true
|
|
}
|
|
group = "shared"
|
|
key_config = {
|
|
create_key = true
|
|
create_secrets = true
|
|
}
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
To use an existing keypair pass the path to the private key, the public key name
|
|
is assumed to have the same name ending with the `.pub` suffix. This is useful
|
|
in cases where the access key has already been set in the modules repository,
|
|
and new repositories need to be created and their corresponding secret set:
|
|
|
|
```hcl
|
|
modules_config = {
|
|
project_name = "modules"
|
|
key_config = {
|
|
create_key = true
|
|
create_secrets = true
|
|
}
|
|
group = "shared"
|
|
key_config = {
|
|
create_secrets = true
|
|
keypair_path = "~/modules-repository-key"
|
|
}
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
### Projects
|
|
|
|
The `projects` variable is where you configure which projects to create in which
|
|
groups and whether initial population of files is desired.
|
|
|
|
This is an example that creates repositories for stages 00 and 01 and 02:
|
|
|
|
```tfvars
|
|
projects = {
|
|
fast_0_org_setup = {
|
|
create_options = {
|
|
description = "FAST org setup."
|
|
features = {
|
|
issues = true
|
|
}
|
|
}
|
|
group = "org-admins"
|
|
populate_from = "../../stages/0-org-setup"
|
|
workflow_file = "org-setup-workflow.yaml"
|
|
}
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
The `create_options` repository attribute controls creation: if the attribute is
|
|
not present, the repository is assumed to be already existing.
|
|
|
|
The group attribute
|
|
|
|
Initial population depends on a modules repository being configured in
|
|
the `modules_config` variable described in the preceding section and on
|
|
the`populate_from` attributes in each repository where population is required,
|
|
which point to the folder holding the files to be committed.
|
|
|
|
Each repository may contain some sample tfvars and data files that can be used
|
|
as a starting point for your own files. By default, the samples are not
|
|
populate. However, you can enable this by setting the `populate_samples`
|
|
attribute to `true`. Here's an updated example:
|
|
|
|
```tfvars
|
|
projects = {
|
|
fast_0_org_setup = {
|
|
create_options = {
|
|
description = "FAST org setup."
|
|
features = {
|
|
issues = true
|
|
}
|
|
}
|
|
group = "org-admins"
|
|
populate_from = "../../stages/0-org-setup"
|
|
populate_sample = true
|
|
workflow_file = "org-setup-workflow.yaml"
|
|
}
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
Please note that setting `populate_samples` to `true` will populate the sample
|
|
files to the repository, potentially overwriting any existing files with the
|
|
same name. To minimize the risk of overwriting existing files, we populate the
|
|
original `data` directory to a `data.sample` directory. In any case, be careful
|
|
when enabling this option and review commit history to check any changes made to
|
|
the sample files.
|
|
|
|
Each project is created inside a group, this mapping is done via `group`
|
|
attribute being set on the project. The `group` attribute is either a reference
|
|
to the key of a group to create in the `groups` variable or the id of an
|
|
existing group in Gilab. Please find below a full example of projects + groups
|
|
initialization.
|
|
|
|
```tfvars
|
|
projects = {
|
|
fast_0_org_setup = {
|
|
create_options = {
|
|
description = "FAST org setup."
|
|
features = {
|
|
issues = true
|
|
}
|
|
}
|
|
group = "org-admins"
|
|
populate_from = "../../stages/0-org-setup"
|
|
populate_sample = true
|
|
workflow_file = "org-setup-workflow.yaml"
|
|
}
|
|
}
|
|
|
|
groups = {
|
|
org-admins = {
|
|
name = "gcp-org-admins"
|
|
path = "gcp-org-admins"
|
|
description = "GCP Organization administrators"
|
|
}
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
### Groups configuration
|
|
|
|
The `groups` variable can be used to create groups in Gitlab, please find below
|
|
a sample usage of this variable.
|
|
|
|
```tfvars
|
|
groups = {
|
|
org-admins = {
|
|
name = "gcp-org-admins"
|
|
path = "gcp-org-admins"
|
|
description = "GCP Organization administrators"
|
|
}
|
|
}
|
|
# tftest skip
|
|
```
|
|
|
|
### Commit configuration
|
|
|
|
An optional variable `commit_config` can be used to configure the author, email,
|
|
and message used in commits for the initial population of files. Its defaults
|
|
are probably fine for most use cases.
|
|
|
|
## How to run this stage
|
|
|
|
Connect to Gitlab as root user and create a personal access token with permissions as per the [gitlab provider credentials](#gitlab-provider-credentials).
|
|
Set the newly created personal access as `gitlab_config.access_token` variable
|
|
and then issue the following commands:
|
|
|
|
```bash
|
|
gcloud storage cp gs://${prefix}-prod-iac-core-outputs-0/workflows/*-workflow.yaml ./workflows/
|
|
```
|
|
|
|
This will download Gitlab CICD workflow files generated during 0-org-setup stage
|
|
on the local .workflows directory for later being uploaded on the new Gitlab
|
|
projects.
|
|
|
|
Set `http_proxy` and `https_proxy` env vars to <http://localhost:3128> and then
|
|
run:
|
|
|
|
```bash
|
|
terraform init
|
|
terraform apply
|
|
```
|
|
|
|
Navigate Gitlab URL and check whether new groups and projects has been
|
|
bootstrapped successfully. Update the default pipeline template replacing the
|
|
ssh-keyscan default configuration with the following commands:
|
|
|
|
```bash
|
|
ssh-keyscan -p 2222 -H 'gitlab.gcp.example.com' >> ~/.ssh/known_hosts
|
|
ssh-keyscan -p 2222 gitlab.gcp.example.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_hosts
|
|
```
|
|
|
|
Please specify the right hostname and port for the gitlab executor to fetch the
|
|
right host key otherwise you might get an _Host key verification failed._ during
|
|
terraform init.
|
|
|
|
Try to create a merge request to trigger a CI pipeline on one of FAST stages and
|
|
check if the plan pipeline executes successfully.
|
|
|
|
<!-- TFDOC OPTS files:1 -->
|
|
<!-- BEGIN TFDOC -->
|
|
## Files
|
|
|
|
| name | description | resources |
|
|
|---|---|---|
|
|
| [cicd-versions.tf](./cicd-versions.tf) | Provider version. | |
|
|
| [main.tf](./main.tf) | Module-level locals and resources. | <code>gitlab_deploy_key</code> · <code>gitlab_group</code> · <code>gitlab_project</code> · <code>gitlab_project_variable</code> · <code>gitlab_repository_file</code> · <code>tls_private_key</code> |
|
|
| [outputs.tf](./outputs.tf) | Module outputs. | |
|
|
| [providers.tf](./providers.tf) | Provider configuration. | |
|
|
| [variables.tf](./variables.tf) | Module variables. | |
|
|
|
|
## Variables
|
|
|
|
| name | description | type | required | default |
|
|
|---|---|:---:|:---:|:---:|
|
|
| [gitlab_config](variables.tf#L28) | Gitlab config. | <code title="object({ access_token = string hostname = optional(string, "gitlab.com") ssh_port = optional(number, 22) saas_group = optional(string, "") })">object({…})</code> | ✓ | |
|
|
| [groups](variables.tf#L45) | Gitlab groups. | <code title="map(object({ name = string path = string description = string }))">map(object({…}))</code> | ✓ | |
|
|
| [commit_config](variables.tf#L17) | Configure commit metadata. | <code title="object({ author = optional(string, "FAST loader") email = optional(string, "fast-loader@fast.gcp.tf") message = optional(string, "FAST initial loading") })">object({…})</code> | | <code>{}</code> |
|
|
| [modules_config](variables.tf#L54) | Gitlab modules config. | <code title="object({ bootstrap = optional(bool, true) module_prefix = optional(string, "") group = optional(string) project_name = string source_ref = optional(string) key_config = optional(object({ create_key = optional(bool, false) create_secrets = optional(bool, false) keypair_path = optional(string) }), {}) })">object({…})</code> | | <code>null</code> |
|
|
| [projects](variables.tf#L79) | Gitlab projects to create. | <code title="map(object({ create_options = optional(object({ allow = optional(object({ auto_merge = optional(bool) merge_commit = optional(bool) rebase_merge = optional(bool) squash_merge = optional(bool) })) auto_init = optional(bool) description = optional(string) features = optional(object({ issues = optional(bool) projects = optional(bool) wiki = optional(bool) })) templates = optional(object({ gitignore = optional(string, "Terraform") license = optional(string) repository = optional(object({ name = string owner = string })) }), {}) visibility = optional(string, "private") })) group = string populate_from = optional(string) populate_samples = optional(bool, false) workflow_file = optional(string, null) }))">map(object({…}))</code> | | <code>{}</code> |
|
|
|
|
## Outputs
|
|
|
|
| name | description | sensitive |
|
|
|---|---|:---:|
|
|
| [clone](outputs.tf#L17) | Clone projects commands. | |
|
|
<!-- END TFDOC -->
|