hunfabric/fast/project-templates/managed-kafka

Managed Kafka Cluster with Topics

This setup allows creating and configuring a managed Kafka cluster using Google Cloud Managed Service for Apache Kafka, with configurable topics, networking, and labels. It is designed to be FAST-compliant and integrates seamlessly with existing Google Cloud infrastructure.

Prerequisites

The project.yaml file describes the project-level configuration needed in terms of API activation and IAM bindings.

If you are deploying this inside a FAST-enabled organization, the file can be lightly edited to match your configuration and then used directly in the project factory.

For non-FAST setups, use the project.yaml file as a reference to configure your project manually:

• Enable the APIs listed under services.
• Grant the permissions listed under iam to the principal running Terraform, either a service account or a human user.

Variable Configuration

Configuration is primarily done via the kafka_config and topics variables. Key considerations:

• The Kafka cluster is deployed in the specified region and subnetworks.
• Topics can be configured with partition count, replication factor, and additional settings.
• Labels can be added for resource organization and management.

Bringing up a cluster and the associated topics from scratch will require approximately 20–30 minutes.

Variables

name	description	type	required	default
kafka_config	Configuration for the Kafka cluster.	object({…})	✓
network_project_ids	List of project IDs where the subnets are located.	list(string)	✓
project_id	The ID of the Google Cloud project where the Kafka cluster will be deployed.	string	✓
labels	Additional labels for the Kafka cluster.	map(string)		{}
topics	The list of topics to create in the Kafka cluster.	list(object({…}))		[]

Outputs

name	description	sensitive
kafka_cluster_id	The ID of the Kafka cluster.
kafka_labels	Labels applied to the Kafka cluster.
kafka_region	The region where the Kafka cluster is deployed.
project_number	The project number of the Kafka cluster.

Test

module "test" {
  source = "./fabric/fast/project-templates/managed-kafka"
  kafka_config = {
    cluster_id   = "test-cluster"
    region       = "us-central1"
    vcpu_count   = 4
    memory_bytes = 21474836480
    subnetworks = [
      "projects/test-project/regions/us-central1/subnetworks/test-subnetwork"
    ]
    rebalance_mode = "AUTO_REBALANCE_ON_SCALE_UP"
  }
  project_id = "my-managed-kafka-project"
  network_project_ids = [
    "test-network-project"
  ]
  topics = [
    {
      topic_id           = "test-topic-1"
      partition_count    = 3
      replication_factor = 2
      configs = {
        "cleanup.policy" = "compact"
      }
    }
  ]
  labels = {
    environment = "test"
  }
}
# tftest skip