Deploy Apache Kafka to GKE using Strimzi

The guide shows you how to use the Strimzi operator to deploy Apache Kafka clusters ok GKE.

Objectives

This tutorial covers the following steps:

Create a GKE cluster.
Deploy and configure the Strimzi operator
Configure Apache Kafka using the Strimzi operator

Estimated time:

To get started, click Start.

select/create a project

Create the Autopilot GKE cluster

Change to the autopilot-cluster directory.
```
cd autopilot-cluster 
```
Create a new file terraform.tfvars in that directory.
```
touch terraform.tfvars
```
Open the file for editing.
Paste the following content in the file and update any value as needed.

project_id     = "<walkthrough-project-name/>"
cluster_name   = "gke-patterns-cluster"
cluster_create = {
  deletion_protection = false
  labels = {
    pattern = "kafka"
  }
}
region         = "europe-west1"
vpc_create = {
  enable_cloud_nat = true
}

Initialize the terraform configuration.
```
terraform init
```
Apply the terraform configuration.
```
terraform apply
```

Fetch the cluster credentials.

gcloud container fleet memberships get-credentials gke-patterns-cluster --project "<walkthrough-project-name/>"

Check the nodes are ready.
```
kubectl get pods -n kube-system
```

Install the Kafka Strimzi operator and create associated resources

Change to the patterns/kafka directory.
```
cd ../kafka
```
Create a new file terraform.tfvars in that directory.
```
touch terraform.tfvars
```
Open the file for editing.

Paste the following content in the file.

credentials_config = {
  kubeconfig = {
    path = "~/.kube/config"
  }
}
kafka_config = {
  volume_claim_size = "15Gi"
  replicas          = 4
}
zookeeper_config = {
  volume_claim_size = "15Gi"
}

Initialize the terraform configuration.
```
terraform init
```
Apply the terraform configuration.
```
terraform apply
```
Check that the Redis pods are ready
```
kubectl get pods -n kafka
```
Check that the Redis volumes match the number of replicas
```
kubectl get pv
```
Confirm the Kafka object is running
```
kubectl get kafka -n kafka
```

Destroy resources (optional)

Change to the patterns/autopilot-cluster directory.
```
cd ../autopilot-cluster
```
Destroy the cluster with the following command.
```
terraform destroy
```

Congratulations

You’re all set!

3.1 KiB Raw Blame History Unescape Escape