kovagoadi/hunfabric
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
81f72e8068a9b5ea52f31410a436058ddedb3ee8
hunfabric/skills/fast/prerequisites/TESTING.md
Ludovico Magnocavallo 81f72e8068 Add FAST Prerequisites Skill and Gemini Skill Test Harness (#3979) 
* initial version of a FAST pre-install skill

* first round of testing

* Update fast-0-org-setup-prereqs skill with improved UX and local path handling

- Add explicit lockout warning and stop condition if the user is not a member of the provided Admin Principal group.
- Streamline bootstrap project selection to only prompt for an override if the active gcloud project is rejected.
- Restrict dataset discovery strictly to the `fast/stages/0-org-setup/datasets/` directory.
- Improve location handling by referencing `defaults.schema.json` for Standard GCP and auto-configuring fixed regions for GCD.
- Add comprehensive `local_path` management: prompt for customization, create directories, move `defaults.yaml` to the local data folder, and symlink `0-org-setup.auto.tfvars` back to the stage directory.

* add testing scenarios, implement initial changes for scenario 2

* move skills

* move to a skills/fast subfolder

* Refactor fast-0-org-setup prereqs skill

* Add skill-turn-harness utility tool

* Use relative markdown links for skill references

* Use descriptive titles for markdown links in skill references

* Add descriptions to each phase in the prerequisites workflow map

* Use backslash for markdown line breaks in skill map

* Update README security warning to mention default .gitignore

* shebang

* Update fast prereqs skill rules to force sequential question flow and refine harness tool with proper ctrl+c handling and slugified log paths

* Move playbook-gcp-dev.yaml to fast/prerequisites/gcp-dev.yaml and update fast prerequisites

* docs(skill-turn-harness): detail autonomous pond testing approach

* docs(skill-turn-harness): add final_state_checks to pond architecture and update toc

* Refine fast prereqs SKILL and gcp-dev playbook to strictly align with one-question-at-a-time rule

* feat(skill-turn-harness): update playbook schema for autonomous persona mode

* feat(skill-turn-harness): implement autonomous persona testing mode and fallback logic

* docs(skill-turn-harness): document the three modes of testing and update ToC

* implement timeout, schema validation, configurable cli

* chore: remove accidentally committed log files

* chore: ignore logs directory

* feat(skill-harness): implement tool execution interception, configurable workspace, and modularized validation

* feat(skill-harness): add model configuration and update README

* fix(skill-harness): automatically inject -y flag to gemini commands

* docs(skill-harness): add TODO.md with analysis for skill environment dependencies

* feat(skill-harness): add working_dir support and clean up fixtures

- Implement working_dir in harness to run tests in specific directories.
- Rename test fixtures and playbooks to be more descriptive.
- Add E2E test for working_dir.
- Apply code quality improvements to harness.py (imports, linting).
- Update README with working directory considerations and usage notes.
- Update phase3-bootstrap-and-iam.md skill doc to add execution rule against creating temp scripts.

* fix: capture customer_id and respect relative paths

* Implement isolated temp workspace sandboxing with symlinks in test harness

* Configure GCD manual autonomous playbook and align Phase 3/4 steps order

* Fix linting and schema tests failures

- Add missing license headers to tools/skill-turn-harness files.

- Fix trailing spaces and newlines in playbooks.

- Ignore tools directory in schema tests workflow.

TAG=agy

CONV=1bb75453-c3e2-448b-bae9-8e332a068012

* Fix Python formatting with yapf

TAG=agy

CONV=1bb75453-c3e2-448b-bae9-8e332a068012

* Refactor skill-turn-harness to use Antigravity SDK

- Migrated harness from gemini-cli subprocesses to Antigravity SDK.
- Implemented real-time step streaming and console logging.
- Added color-coded terminal output (dark gray headers, blue inputs, pink outputs).
- Collapsed excessive newlines in streamed thoughts.
- Excluded harness codebase from workspace copy to prevent agent cheating.
- Enabled skills folder copy to resolve agent lookup loops.
- Added key validation and CLI --debug flag.

* Fix autonomous turn layout: print Turn ID before execution

- Moved the [Autonomous Turn X] header print to before running the agent turn.
- This groups the real-time thinking and tool calls under the correct Turn ID block, instead of displaying them before the label.

* Remove obsolete .log.md from prerequisites skill directory
2026-05-22 17:16:54 +00:00

3.1 KiB
Raw Blame History

FAST 0-org-setup Prerequisites - Testing Workflows

This document outlines the various testing workflows and combinations for the fast-0-org-setup-prereqs skill to ensure all branches of the skill's logic are covered.

Core Variables & Decision Points

  1. Environment: Standard GCP vs. Google Cloud Dedicated (GCD)
  2. GCD Universe (if applicable): S3NS (France), Berlin (Germany), or Custom
  3. Execution Mode: Auto (Gemini CLI runs commands) vs. Manual (Output commands for user)
  4. Authentication State: Already authenticated vs. Needs authentication
  5. Admin Principal Approach: Group (Approach A) vs. Single User (Approach B)
  6. Billing Account Access Level: Billing Administrator vs. Billing User vs. No Access
  7. Bootstrap Project: Pre-existing (with or without API check) vs. New project created by user
  8. Configuration / Dataset: Standard GCP datasets vs. classic-gcd dataset
  9. Org Policy Import Check: Pre-existing policies found vs. No policies found

Comprehensive Test Scenarios

Scenario 1: Standard GCP - The "Simple User" Flow

  • Environment: Standard GCP
  • Execution Mode: Manual
  • Authentication: Needs authentication (Standard auth flow)
  • Admin Principal: Single User (current authenticated user)
  • Information Gathering: Direct input of Org ID and Billing ID
  • Billing Access: Billing Administrator
  • Bootstrap Project: New project (user creates, CLI outputs full API enable command)
  • Configuration: Default local path, standard dataset, additional context added
  • Org Policies: Some policies exist (to test tfvars update)

Scenario 2: Standard GCP - The "Developer" Flow

  • Environment: Standard GCP
  • Execution Mode: Auto
  • Authentication: Already authenticated
  • Admin Principal: Group (foo-owners@example.com)
  • Information Gathering: Uses "list" to fetch Org ID and Billing ID
  • Billing Access: Billing User (requires disabling billing YAML)
  • Bootstrap Project: Pre-existing project (auto-check and enable missing APIs)
  • Configuration: Custom local path, standard dataset, no additional context
  • Org Policies: Some policies exist (to test tfvars update)

Notes

Scenario 3: GCD (S3NS) - Automated Setup

  • Environment: Google Cloud Dedicated (GCD)
  • Universe: S3NS (France)
  • Execution Mode: Auto
  • Authentication: Needs authentication (WIF login flow)
  • Admin Principal: Group
  • Billing Access: Billing Administrator
  • Bootstrap Project: Pre-existing project (skip API check, enable all)
  • Configuration: classic-gcd dataset (fixed), auto-mapped universe regions, providers.tf generated
  • Org Policies: No policies exist

Scenario 4: GCD (Custom) - Manual Setup with Restrictions

  • Environment: Google Cloud Dedicated (GCD)
  • Universe: Custom (manually input all 5 variables)
  • Execution Mode: Manual
  • Authentication: Already authenticated
  • Admin Principal: Single User
  • Billing Access: No Access (requires warning and manual continuation)
  • Bootstrap Project: New project
  • Configuration: classic-gcd dataset, custom local path
  • Org Policies: Some policies exist