# FAST Gitlab repository management
This small extra stage allows creating and populating Gitlab groups and projects
used to host FAST stages code, including rewriting of module sources and secrets
used for private modules repository access.
It is designed for use in a Gitlab self-managed, and is only meant as a one-shot
solution with perishable state especially when used for initial population, as
you don't want Terraform to keep overwriting your changes with initial versions
of files.
## Gitlab provider credentials
A [Gitlab token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)
is needed to authenticate against their API. The token needs admin permissions,
like shown in this screenshot:
Once a token is available set it in the `gitlab_config.access_token` terraform
variable before running Terraform. You might also want to update the default
Gitlab hostname and SSH port on the `gitlab_config` section.
## Variable configuration
### Gitlab Config
The `gitlab_config` variable defines where projects will be hosted.
GitLab can either be the SaaS offering (with the default hostname `gitlab.com`)
or a self-hosted instance with a custom FQDN. If hostname is set to gitlab.com, you must
also set the `saas_group` variable to specify the group path where resources will be created.
This is an example that configures a SaaS gitlab instance using `my_group/gcp` as default group :
```hcl
gitlab_config = {
access_token = "glpat-XXX"
hostname = "gitlab.com"
ssh_port = 22
saas_group = "my_group/gcp"
}
# tftest skip
```
This is an example that configures an on-premise gitlab instance :
```hcl
gitlab_config = {
access_token = "glpat-XXX"
hostname = "my-gitlab.example.com"
}
# tftest skip
```
### Modules project and sources
The `modules_config` variable controls creation and management of both the
Gitlab project hosting fast modules and key and secret used to access it, and
indirectly control population of initial files: if the `modules_config` variable
is not specified no module repository is know to the code, so module source
paths cannot be replaced, and initial population of files cannot happen. If the
variable is specified, an optional `source_ref` attribute can be set to the
reference used to pin modules versions.
This is an example that configures the modules project in Gitlab in a "shared"
group with source ref:
```hcl
modules_config = {
project_name = "modules"
key_config = {
create_key = true
create_secrets = true
}
group = "shared"
}
# tftest skip
```
If the modules are located in a non modules only repository, use the
module_prefix attribute to set the location of your modules within the
repository:
```hcl
modules_config = {
project_name = "modules"
key_config = {
create_key = true
create_secrets = true
}
group = "shared"
module_prefix = "modules/"
}
# tftest skip
```
In the above example, no key options are set so it's assumed modules will be
fetched from a public repository. If modules repository authentication is needed
the `key_config` attribute also needs to be set.
If no keypair path is specified an internally generated key will be stored as a
deploy key in the modules project, and as secrets in the stage repositories:
```hcl
modules_config = {
project_name = "modules"
key_config = {
create_key = true
create_secrets = true
}
group = "shared"
key_config = {
create_key = true
create_secrets = true
}
}
# tftest skip
```
To use an existing keypair pass the path to the private key, the public key name
is assumed to have the same name ending with the `.pub` suffix. This is useful
in cases where the access key has already been set in the modules repository,
and new repositories need to be created and their corresponding secret set:
```hcl
modules_config = {
project_name = "modules"
key_config = {
create_key = true
create_secrets = true
}
group = "shared"
key_config = {
create_secrets = true
keypair_path = "~/modules-repository-key"
}
}
# tftest skip
```
### Projects
The `projects` variable is where you configure which projects to create in which
groups and whether initial population of files is desired.
This is an example that creates repositories for stages 00 and 01 and 02:
```tfvars
projects = {
fast_0_org_setup = {
create_options = {
description = "FAST org setup."
features = {
issues = true
}
}
group = "org-admins"
populate_from = "../../stages/0-org-setup"
workflow_file = "org-setup-workflow.yaml"
}
}
# tftest skip
```
The `create_options` repository attribute controls creation: if the attribute is
not present, the repository is assumed to be already existing.
The group attribute
Initial population depends on a modules repository being configured in
the `modules_config` variable described in the preceding section and on
the`populate_from` attributes in each repository where population is required,
which point to the folder holding the files to be committed.
Each repository may contain some sample tfvars and data files that can be used
as a starting point for your own files. By default, the samples are not
populate. However, you can enable this by setting the `populate_samples`
attribute to `true`. Here's an updated example:
```tfvars
projects = {
fast_0_org_setup = {
create_options = {
description = "FAST org setup."
features = {
issues = true
}
}
group = "org-admins"
populate_from = "../../stages/0-org-setup"
populate_sample = true
workflow_file = "org-setup-workflow.yaml"
}
}
# tftest skip
```
Please note that setting `populate_samples` to `true` will populate the sample
files to the repository, potentially overwriting any existing files with the
same name. To minimize the risk of overwriting existing files, we populate the
original `data` directory to a `data.sample` directory. In any case, be careful
when enabling this option and review commit history to check any changes made to
the sample files.
Each project is created inside a group, this mapping is done via `group`
attribute being set on the project. The `group` attribute is either a reference
to the key of a group to create in the `groups` variable or the id of an
existing group in Gilab. Please find below a full example of projects + groups
initialization.
```tfvars
projects = {
fast_0_org_setup = {
create_options = {
description = "FAST org setup."
features = {
issues = true
}
}
group = "org-admins"
populate_from = "../../stages/0-org-setup"
populate_sample = true
workflow_file = "org-setup-workflow.yaml"
}
}
groups = {
org-admins = {
name = "gcp-org-admins"
path = "gcp-org-admins"
description = "GCP Organization administrators"
}
}
# tftest skip
```
### Groups configuration
The `groups` variable can be used to create groups in Gitlab, please find below
a sample usage of this variable.
```tfvars
groups = {
org-admins = {
name = "gcp-org-admins"
path = "gcp-org-admins"
description = "GCP Organization administrators"
}
}
# tftest skip
```
### Commit configuration
An optional variable `commit_config` can be used to configure the author, email,
and message used in commits for the initial population of files. Its defaults
are probably fine for most use cases.
## How to run this stage
Connect to Gitlab as root user and create a personal access token with permissions as per the [gitlab provider credentials](#gitlab-provider-credentials).
Set the newly created personal access as `gitlab_config.access_token` variable
and then issue the following commands:
```bash
gcloud storage cp gs://${prefix}-prod-iac-core-outputs-0/workflows/*-workflow.yaml ./workflows/
```
This will download Gitlab CICD workflow files generated during 0-org-setup stage
on the local .workflows directory for later being uploaded on the new Gitlab
projects.
Set `http_proxy` and `https_proxy` env vars to and then
run:
```bash
terraform init
terraform apply
```
Navigate Gitlab URL and check whether new groups and projects has been
bootstrapped successfully. Update the default pipeline template replacing the
ssh-keyscan default configuration with the following commands:
```bash
ssh-keyscan -p 2222 -H 'gitlab.gcp.example.com' >> ~/.ssh/known_hosts
ssh-keyscan -p 2222 gitlab.gcp.example.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_hosts
```
Please specify the right hostname and port for the gitlab executor to fetch the
right host key otherwise you might get an _Host key verification failed._ during
terraform init.
Try to create a merge request to trigger a CI pipeline on one of FAST stages and
check if the plan pipeline executes successfully.
## Files
| name | description | resources |
|---|---|---|
| [cicd-versions.tf](./cicd-versions.tf) | Provider version. | |
| [main.tf](./main.tf) | Module-level locals and resources. | gitlab_deploy_key · gitlab_group · gitlab_project · gitlab_project_variable · gitlab_repository_file · tls_private_key |
| [outputs.tf](./outputs.tf) | Module outputs. | |
| [providers.tf](./providers.tf) | Provider configuration. | |
| [variables.tf](./variables.tf) | Module variables. | |
## Variables
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [gitlab_config](variables.tf#L28) | Gitlab config. | object({…}) | ✓ | |
| [groups](variables.tf#L45) | Gitlab groups. | map(object({…})) | ✓ | |
| [commit_config](variables.tf#L17) | Configure commit metadata. | object({…}) | | {} |
| [modules_config](variables.tf#L54) | Gitlab modules config. | object({…}) | | null |
| [projects](variables.tf#L79) | Gitlab projects to create. | map(object({…})) | | {} |
## Outputs
| name | description | sensitive |
|---|---|:---:|
| [clone](outputs.tf#L17) | Clone projects commands. | |