Go Migration Strategy for Tools

authors: Julio Castillo date: Mar 26, 2026

Status

Rejected.

Context

As we evaluate migrating repository tools like tfdoc from Python to Go, the primary objectives are removing external dependencies (like local Python environment setup and pip package management) and improving execution speed. As our repository tooling expands, we need a formalized strategy to determine which existing Python utility scripts (tools/*.py) should be migrated to Go, how they should be structured within a single Go module, and how both environments interact long-term.

Containerization and Contributor Onboarding

A benefit of migrating to natively compiled Go binaries is the ability to easily containerize the toolkit. Because Go generates standalone executables with no runtime dependencies, the resulting tools can be packaged into lightweight, secure container images. This simplifies contributor onboarding by eliminating the need to configure local Python virtual environments or install third-party dependencies.

Decision

The proposed migration to Go for all tools has been rejected. While Go offers performance benefits and easier containerization, we decided to maintain the existing Python-based toolkit for now to avoid the additional maintenance overhead and ensure consistency with the current team's skillset.

Rationale for Rejection (PR #3815 Discussion)

The decision to reject the migration was reached after evaluating a prototype implementation (PR #3815) and discussing the trade-offs:

Maintenance vs. Speed: While the Go implementation of tools like tfdoc was faster, the speed gains on utility scripts were not deemed significant enough to justify the increased maintenance complexity and loss of control compared to the existing Python scripts.
Testing is the Bottleneck: we identified that test execution time is the primary bottleneck in CI, not the utility tools.
Preserving the Pytest Framework: An evaluation of migrating the testing framework to Go concluded it would require high effort for questionable return, as the current declarative YAML/Pytest setup is effective and well-suited for our needs.

Tool Migration Strategy (Original Proposal)

A review of the current tools/*.py codebase identifies three clear partitions:

Low Complexity / Medium Benefit

These scripts provide performance improvements or safety enhancements when written in statically-typed Go.

check_documentation.py: Already fused with the migrated tfdoc as the --check flag.
check_boilerplate.py & check_names.py: These enforce basic naming conventions and monitor file headers (licenses) via directory traversal. Implementing these in native Go provides faster execution times and allows them to share core linter logic.
check_links.py: Validates markdown link anchors across the repository. Go's robust goldmark library and parallelization capabilities make this an excellent candidate.

High Complexity / High Benefit

Migrate tests to Go: Our current framework basically parses Markdown files and runs terraform against the parsed snippets. The resulting plan is compared against a YAML manifest. Very few tests actually rely on Python code. This could potentially be converted to a highly parallel Go implementation removing the complicated python/pytest/tftest layers. Requires careful evaluation (e.g. e2e tests, etc.)
plan_summary.py / tftest_plan_summary.py: These ingest structured terraform plan JSON outputs. Using encoding/json inside Go provides type-safety, ensuring changes in Terraform's internal JSON structure are caught at compile-time when mapped against Go structs. If we migrate the testing framework to Go, the implementation of these tool will be trivial.

Low ROI (Leave in Python)

versions.py: only one external dependency (click) and not used frequently or in CI/CD. It's mostly an standardization tool.

check_yaml_schema.py & schema_docs.py: Standardizing dynamic/untyped schema validation in Go involves heavy boilerplate. Python's native jsonschema library excels at loosely-typed structure checks.
changelog.py & build_service_agents.py: Scripts making REST requests to GitHub APIs or scraping web resources should remain in Python. Dynamic scripting languages are purpose-built for data munging.

Proposed Folder Structure (Original Proposal)

Rather than spawning .mod files per script, all Go tooling operates securely within a monolith module utilizing standard cmd/ entrypoints:

tools/
├── Makefile                     <-- (Builds all binaries into tools/)
├── tfdoc                        <-- (Compiled Go binary)
├── check_boilerplate            <-- (Future compiled Go binary)
├── check_yaml_schema.py         <-- (Retained Python script)
└── src/                         <-- (Single Go module)
    ├── go.mod
    ├── go.sum
    ├── cmd/
    │   ├── tfdoc/
    │   │   └── main.go          <-- (Entrypoint)
    │   └── check_boilerplate/
    │       └── main.go          <-- (Entrypoint)
    └── internal/
        ├── parser/              <-- (Shared HCL parsing)
        ├── linter/              <-- (Shared linting rules)
        └── render/              <-- (Shared Markdown generation)

Risks

More difficult maintenance of Go code: Go can present a slightly higher maintenance burden for teams primarily accustomed to Python scripting. This risk is partly mitigated by the increasing availability of LLMs.
Local usage requires a Go compiler: Contributors running or developing the automated tools directly on their host machines will need to have Go installed, although containerized environments can alleviate this operational setup.

5.8 KiB Raw Permalink Blame History