Files
hunfabric/.github/workflows/tests.yml
Ludovico Magnocavallo 81f72e8068 Add FAST Prerequisites Skill and Gemini Skill Test Harness (#3979)
* initial version of a FAST pre-install skill

* first round of testing

* Update fast-0-org-setup-prereqs skill with improved UX and local path handling

- Add explicit lockout warning and stop condition if the user is not a member of the provided Admin Principal group.
- Streamline bootstrap project selection to only prompt for an override if the active gcloud project is rejected.
- Restrict dataset discovery strictly to the `fast/stages/0-org-setup/datasets/` directory.
- Improve location handling by referencing `defaults.schema.json` for Standard GCP and auto-configuring fixed regions for GCD.
- Add comprehensive `local_path` management: prompt for customization, create directories, move `defaults.yaml` to the local data folder, and symlink `0-org-setup.auto.tfvars` back to the stage directory.

* add testing scenarios, implement initial changes for scenario 2

* move skills

* move to a skills/fast subfolder

* Refactor fast-0-org-setup prereqs skill

* Add skill-turn-harness utility tool

* Use relative markdown links for skill references

* Use descriptive titles for markdown links in skill references

* Add descriptions to each phase in the prerequisites workflow map

* Use backslash for markdown line breaks in skill map

* Update README security warning to mention default .gitignore

* shebang

* Update fast prereqs skill rules to force sequential question flow and refine harness tool with proper ctrl+c handling and slugified log paths

* Move playbook-gcp-dev.yaml to fast/prerequisites/gcp-dev.yaml and update fast prerequisites

* docs(skill-turn-harness): detail autonomous pond testing approach

* docs(skill-turn-harness): add final_state_checks to pond architecture and update toc

* Refine fast prereqs SKILL and gcp-dev playbook to strictly align with one-question-at-a-time rule

* feat(skill-turn-harness): update playbook schema for autonomous persona mode

* feat(skill-turn-harness): implement autonomous persona testing mode and fallback logic

* docs(skill-turn-harness): document the three modes of testing and update ToC

* implement timeout, schema validation, configurable cli

* chore: remove accidentally committed log files

* chore: ignore logs directory

* feat(skill-harness): implement tool execution interception, configurable workspace, and modularized validation

* feat(skill-harness): add model configuration and update README

* fix(skill-harness): automatically inject -y flag to gemini commands

* docs(skill-harness): add TODO.md with analysis for skill environment dependencies

* feat(skill-harness): add working_dir support and clean up fixtures

- Implement working_dir in harness to run tests in specific directories.
- Rename test fixtures and playbooks to be more descriptive.
- Add E2E test for working_dir.
- Apply code quality improvements to harness.py (imports, linting).
- Update README with working directory considerations and usage notes.
- Update phase3-bootstrap-and-iam.md skill doc to add execution rule against creating temp scripts.

* fix: capture customer_id and respect relative paths

* Implement isolated temp workspace sandboxing with symlinks in test harness

* Configure GCD manual autonomous playbook and align Phase 3/4 steps order

* Fix linting and schema tests failures

- Add missing license headers to tools/skill-turn-harness files.

- Fix trailing spaces and newlines in playbooks.

- Ignore tools directory in schema tests workflow.

TAG=agy

CONV=1bb75453-c3e2-448b-bae9-8e332a068012

* Fix Python formatting with yapf

TAG=agy

CONV=1bb75453-c3e2-448b-bae9-8e332a068012

* Refactor skill-turn-harness to use Antigravity SDK

- Migrated harness from gemini-cli subprocesses to Antigravity SDK.
- Implemented real-time step streaming and console logging.
- Added color-coded terminal output (dark gray headers, blue inputs, pink outputs).
- Collapsed excessive newlines in streamed thoughts.
- Excluded harness codebase from workspace copy to prevent agent cheating.
- Enabled skills folder copy to resolve agent lookup loops.
- Added key validation and CLI --debug flag.

* Fix autonomous turn layout: print Turn ID before execution

- Moved the [Autonomous Turn X] header print to before running the agent turn.
- This groups the real-time thinking and tool calls under the correct Turn ID block, instead of displaying them before the label.

* Remove obsolete .log.md from prerequisites skill directory
2026-05-22 17:16:54 +00:00

285 lines
9.3 KiB
YAML

# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "Tests"
# yamllint disable-line rule:truthy
on:
push:
branches:
- master
- fast-dev
pull_request:
branches:
- master
- fast-dev
workflow_dispatch:
inputs:
terraform_version:
description: "Use '1.5.7' to test last MPLv2 Terraform version"
required: true
default: 1.12.2
type: string
env:
GOOGLE_APPLICATION_CREDENTIALS: "/home/runner/credentials.json"
PYTEST_ADDOPTS: "--color=yes"
PYTHON_VERSION: "3.10"
TF_PLUGIN_CACHE_DIR: "/home/runner/.terraform.d/plugin-cache"
TFTEST_COPY: 1
DEFAULT_TERRAFORM_FLAVOUR: terraform
DEFAULT_TERRAFORM_VERSION: ${{ inputs.terraform_version || '1.12.2' }}
DEFAULT_TOFU_VERSION: "1.11.0"
permissions:
contents: read
jobs:
setup-tf-providers:
runs-on: ubuntu-latest
continue-on-error: true # run terraform tests, even if setup for tofu fails
strategy:
fail-fast: false
matrix:
include:
- flavour: terraform
- flavour: tofu
steps:
- uses: actions/checkout@v4
- name: Set Terraform versions
run: |
set -e -o xtrace
if [[ ${{ matrix.flavour }} == 'terraform' ]] ; then
echo TERRAFORM_VERSION=${DEFAULT_TERRAFORM_VERSION} | tee -a ${GITHUB_ENV}
elif [[ ${{ matrix.flavour }} == 'tofu' ]] ; then
echo TERRAFORM_VERSION=${{ env.DEFAULT_TOFU_VERSION }} | tee -a ${GITHUB_ENV}
else
echo TERRAFORM_VERSION=unkown_flavor | tee -a ${GITHUB_ENV}
fi
- uses: hashicorp/setup-terraform@v3
if: ${{ matrix.flavour == 'terraform' }}
with:
terraform_version: ${{ env.TERRAFORM_VERSION }}
terraform_wrapper: false
- uses: opentofu/setup-opentofu@v1
if: ${{ matrix.flavour == 'tofu' }}
with:
tofu_version: ${{ env.TERRAFORM_VERSION }}
tofu_wrapper: false
- name: Build lockfile and fetch providers
shell: bash
run: |
mkdir -p ${{ env.TF_PLUGIN_CACHE_DIR }}
echo 'plugin_cache_dir = "${{ env.TF_PLUGIN_CACHE_DIR }}"' | tee -a /home/runner/.terraformrc
echo 'disable_checkpoint = true' | tee -a /home/runner/.terraformrc
sed -i -e 's/>=\(.*# tftest\)/=\1/g' tools/lockfile/versions.tf tools/lockfile/versions.tofu
# change terraform version to the one that is running
sed -i "s/required_version = .\*$/required_version = \">= ${TERRAFORM_VERSION}\"/g" \
tools/lockfile/versions.tf
cd tools/lockfile
${{ matrix.flavour }} init -upgrade=true
- name: Upload Terraform provider cache
uses: actions/cache@v4
with:
path: ${{ env.TF_PLUGIN_CACHE_DIR }}
# yamllint disable-line rule:line-length
key: ${{ runner.os }}-${{ matrix.flavour }}-${{ env.TERRAFORM_VERSION }}-${{ hashFiles('tools/lockfile/.terraform.lock.hcl') }}
- name: Upload lockfile
uses: actions/upload-artifact@v4
with:
name: lockfile-${{ runner.os }}-${{ matrix.flavour }}-${{ env.TERRAFORM_VERSION }}
path: tools/lockfile/.terraform.lock.hcl
overwrite: true
include-hidden-files: true
examples-project-templates:
runs-on: ubuntu-latest
needs: setup-tf-providers
steps:
- uses: actions/checkout@v4
- name: Call composite action fabric-tests
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.DEFAULT_TERRAFORM_VERSION }}
TERRAFORM_FLAVOUR: ${{ env.DEFAULT_TERRAFORM_FLAVOUR }}
- name: Run tests on documentation examples
run: pytest -vv -n4 --tb=line --junit-xml=test-results-raw.xml -k fast/project-templates/ tests/examples
- name: Create report
uses: ./.github/actions/post-fabric-tests
if: always()
with:
MODULE: Project Templates Examples
examples-modules:
runs-on: ubuntu-latest
needs:
- setup-tf-providers
strategy:
fail-fast: false
matrix:
include:
- flavour: terraform
- flavour: tofu
steps:
- uses: actions/checkout@v4
- name: Set Terraform versions
run: |
set -e -o xtrace
if [[ ${{ matrix.flavour }} == 'terraform' ]] ; then
echo TERRAFORM_VERSION=${DEFAULT_TERRAFORM_VERSION} | tee -a ${GITHUB_ENV}
elif [[ ${{ matrix.flavour }} == 'tofu' ]] ; then
echo TERRAFORM_VERSION=${{ env.DEFAULT_TOFU_VERSION }} | tee -a ${GITHUB_ENV}
else
echo TERRAFORM_VERSION=unkown_flavor | tee -a ${GITHUB_ENV}
fi
- name: Call composite action fabric-tests
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_FLAVOUR: ${{ matrix.flavour }}
- name: Run tests on documentation examples
env:
TERRAFORM: ${{ matrix.flavour }}
run: |
pytest -vv ${{ matrix.flavour == 'terraform' && '-n4' || '-n4' }} --tb=line \
--junit-xml=test-results-raw.xml -k "terraform and modules/" tests/examples
- name: Create report
uses: ./.github/actions/post-fabric-tests
if: always()
with:
MODULE: Module Examples
modules:
runs-on: ubuntu-latest
needs:
- setup-tf-providers
strategy:
fail-fast: false
matrix:
include:
- flavour: terraform
- flavour: tofu
steps:
- uses: actions/checkout@v4
- name: Set Terraform versions
run: |
set -e -o xtrace
if [[ ${{ matrix.flavour }} == 'terraform' ]] ; then
echo TERRAFORM_VERSION=${DEFAULT_TERRAFORM_VERSION} | tee -a ${GITHUB_ENV}
elif [[ ${{ matrix.flavour }} == 'tofu' ]] ; then
echo TERRAFORM_VERSION=${{ env.DEFAULT_TOFU_VERSION }} | tee -a ${GITHUB_ENV}
else
echo TERRAFORM_VERSION=unkown_flavor | tee -a ${GITHUB_ENV}
fi
- name: Call composite action fabric-tests
uses: ./.github/actions/fabric-tests
env:
TERRAFORM: ${{ matrix.flavour }}
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_FLAVOUR: ${{ matrix.flavour }}
- name: Run tests modules
env:
TERRAFORM: ${{ matrix.flavour }}
run: |
pytest -vv ${{ matrix.flavour == 'terraform' && '-n4' || '-n4' }} --tb=line \
--junit-xml=test-results-raw.xml tests/modules
- name: Create report
uses: ./.github/actions/post-fabric-tests
if: always()
with:
MODULE: Modules
fast:
runs-on: ubuntu-latest
needs: setup-tf-providers
strategy:
fail-fast: false
matrix:
include:
- flavour: terraform
# - flavour: tofu # tofu fails to find the terraform binary for FAST tests
steps:
- uses: actions/checkout@v4
- name: Set Terraform versions
run: |
set -e -o xtrace
if [[ ${{ matrix.flavour }} == 'terraform' ]] ; then
echo TERRAFORM_VERSION=${DEFAULT_TERRAFORM_VERSION} | tee -a ${GITHUB_ENV}
elif [[ ${{ matrix.flavour }} == 'tofu' ]] ; then
echo TERRAFORM_VERSION=${{ env.DEFAULT_TOFU_VERSION }} | tee -a ${GITHUB_ENV}
else
echo TERRAFORM_VERSION=unkown_flavor | tee -a ${GITHUB_ENV}
fi
- name: Call composite action fabric-tests
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.TERRAFORM_VERSION }}
TERRAFORM_FLAVOUR: ${{ matrix.flavour }}
- name: Run tests on FAST stages
run: pytest -vv -n4 --tb=line --junit-xml=test-results-raw.xml tests/fast
- name: Create report
uses: ./.github/actions/post-fabric-tests
if: always()
with:
MODULE: FAST
schemas:
runs-on: ubuntu-latest
needs: setup-tf-providers
steps:
- uses: actions/checkout@v4
- name: Call composite action fabric-tests
uses: ./.github/actions/fabric-tests
with:
PYTHON_VERSION: ${{ env.PYTHON_VERSION }}
TERRAFORM_VERSION: ${{ env.DEFAULT_TERRAFORM_VERSION }}
TERRAFORM_FLAVOUR: ${{ env.DEFAULT_TERRAFORM_FLAVOUR }}
- name: Run schema tests
run: |
pytest -vv --tb=line --junit-xml=test-results-raw.xml --ignore=tools \
-k "(tests and schemas) or (fast and schema) or (examples and yaml)"
- name: Create report
uses: ./.github/actions/post-fabric-tests
if: always()
with:
MODULE: Schemas