Managed Kafka Cluster with Topics

This setup allows creating and configuring a managed Kafka cluster using Google Cloud Managed Service for Apache Kafka, with configurable topics, networking, and labels. It is designed to be FAST-compliant and integrates seamlessly with existing Google Cloud infrastructure.

Prerequisites

The project.yaml file describes the project-level configuration needed in terms of API activation and IAM bindings.

If you are deploying this inside a FAST-enabled organization, the file can be lightly edited to match your configuration and then used directly in the project factory.

For non-FAST setups, use the project.yaml file as a reference to configure your project manually:

Enable the APIs listed under services.
Grant the permissions listed under iam to the principal running Terraform, either a service account or a human user.

Variable Configuration

Configuration is primarily done via the kafka_config and topics variables. Key considerations:

The Kafka cluster is deployed in the specified region and subnetworks.
Topics can be configured with partition count, replication factor, and additional settings.
Labels can be added for resource organization and management.

Bringing up a cluster and the associated topics from scratch will require approximately 20–30 minutes.

Variables

name	description	type	required	default
kafka_config	Configuration for the Kafka cluster.	`object({…})`	✓
network_project_ids	List of project IDs where the subnets are located.	`list(string)`	✓
project_id	The ID of the Google Cloud project where the Kafka cluster will be deployed.	`string`	✓
labels	Additional labels for the Kafka cluster.	`map(string)`		`{}`
topics	The list of topics to create in the Kafka cluster.	`list(object({…}))`		`[]`

Outputs

name	description	sensitive
kafka_cluster_id	The ID of the Kafka cluster.
kafka_labels	Labels applied to the Kafka cluster.
kafka_region	The region where the Kafka cluster is deployed.
project_number	The project number of the Kafka cluster.

Test

module "test" {
  source = "./fabric/fast/project-templates/managed-kafka"
  kafka_config = {
    cluster_id   = "test-cluster"
    region       = "us-central1"
    vcpu_count   = 4
    memory_bytes = 21474836480
    subnetworks = [
      "projects/test-project/regions/us-central1/subnetworks/test-subnetwork"
    ]
    rebalance_mode = "AUTO_REBALANCE_ON_SCALE_UP"
  }
  project_id = "my-managed-kafka-project"
  network_project_ids = [
    "test-network-project"
  ]
  topics = [
    {
      topic_id           = "test-topic-1"
      partition_count    = 3
      replication_factor = 2
      configs = {
        "cleanup.policy" = "compact"
      }
    }
  ]
  labels = {
    environment = "test"
  }
}
# tftest skip

4.0 KiB Raw Blame History Unescape Escape