How to deploy via Terraform¶

This guide describes how to deploy Charmed Apache Kafka using Terraform and the Juju Terraform provider.

For Juju CLI-based deployment, see the Juju CLI deployment guide.

Prerequisites¶

A Juju controller bootstrapped on the target VM or Kubernetes cloud (see Juju CLI deployment guide for setup instructions)
A Juju model created on the controller
Terraform (>= 1.7.3) installed

Terraform configuration¶

Select the module matching the Juju model substrate. The VM module and K8s module have the same deployment profile structure; K8s additionally supports an ingress offer for Kafka UI.

The same main.tf is used for both production and testing deployments — the deployment mode is controlled via a kafka.auto.tfvars file.

See main.tf

terraform {
  required_providers {
    juju = {
      source  = "juju/juju"
      version = ">= 1.0.0"
    }
  }
}

provider "juju" {}

variable "model_name" {
  description = "Name of the Juju model to deploy to"
  type        = string
}

variable "model_owner" {
  description = "Owner of the Juju model"
  type        = string
  default     = "admin"
}

variable "profile" {
  description = "Deployment profile: 'production' or 'testing'"
  type        = string
  default     = "testing"
}

variable "broker" {
  description = "Apache Kafka broker configuration"
  default     = {}
}

variable "controller" {
  description = "Apache Kafka KRaft controller configuration"
  default     = {}
}

variable "integrator" {
  description = "Data Integrator configuration for admin user creation"
  default     = { units = 0 }
}

variable "connect" {
  description = "Kafka Connect configuration"
  default     = { units = 0 }
}

variable "karapace" {
  description = "Karapace Schema Registry configuration"
  default     = { units = 0 }
}

variable "ui" {
  description = "Kafbat Kafka UI configuration"
  default     = { units = 0 }
}

variable "tls_offer" {
  description = "TLS provider cross-model offer URL"
  type        = string
  default     = null
}

variable "cos_offers" {
  description = "COS cross-model offer URLs for observability"
  default     = {}
}

variable "ingress_offer" {
  description = "Ingress provider cross-model offer URL for Kafka UI (Kubernetes)"
  type        = string
  default     = null
}

data "juju_model" "kafka" {
  name  = var.model_name
  owner = var.model_owner
}

module "kafka" {
  # Select exactly one source for the target substrate:
  # VM:  git::https://github.com/canonical/kafka-bundle//terraform?ref=main
  # K8s: git::https://github.com/canonical/kafka-k8s-bundle//terraform?ref=main
  source = "<substrate-module-source>"

  model_uuid = data.juju_model.kafka.uuid
  profile    = var.profile
  broker     = var.broker
  controller = var.controller
  integrator = var.integrator
  connect    = var.connect
  karapace   = var.karapace
  ui         = var.ui
  tls_offer  = var.tls_offer
  cos_offers = var.cos_offers

  # Kubernetes: enable when using the K8s module.
  # ingress_offer = var.ingress_offer
}

When controller includes units > 0, the module deploys separate broker and controller applications. When controller has units = 0, the broker co-locates both the broker and controller roles in a single application.

Deploy for production¶

For production use, deploy separate kafka (broker) and controller (KRaft controller) applications and integrate them. To maintain high availability, 3+ broker units and 3 or 5 controller units are recommended.

Save the following as kafka.auto.tfvars. The application names are arbitrary; the examples use conventional names for each substrate:

model_name = "terraform"
profile    = "production"

broker = {
  app_name = "kafka"
  channel  = "4/stable"
  units    = 3
}

controller = {
  app_name = "controller"
  channel  = "4/stable"
  units    = 3
}

K8s

model_name = "terraform"
profile    = "production"

broker = {
  app_name = "kafka-k8s"
  channel  = "4/stable"
  units    = 3
}

controller = {
  app_name = "controller"
  channel  = "4/stable"
  units    = 3
}

(Alternative) Deploy for testing¶

For non-production testing clusters, co-locate both KRaft controller and broker services in a single application to save resources.

Save the following as kafka.auto.tfvars:

model_name = "terraform"

broker = {
  app_name = "kafka"
  channel  = "4/stable"
  units    = 3
}

controller = {
  units = 0
}

K8s

model_name = "terraform"

broker = {
  app_name = "kafka-k8s"
  channel  = "4/stable"
  units    = 3
}

controller = {
  units = 0
}

With controller.units = 0, the controller role is co-located within the broker application. The profile defaults to "testing".

Deploy¶

Initialise Terraform, then preview and apply the deployment:

terraform init
terraform plan

Review the plan output, then apply:

terraform apply

Terraform automatically loads the .auto.tfvars file in the working directory. See the profile reference for VM or K8s.

Wait for Terraform to finish. Then, monitor the Juju model status with:

watch juju status --color

The deployment is complete once all units show active and idle status.

(Optional) Create an external admin user¶

After deployment, the Apache Kafka cluster does not expose any external listeners by default. To create an admin user, add the following integrator block to your kafka.auto.tfvars file:

integrator = {
  app_name = "data-integrator"
  channel  = "latest/stable"
  config   = {
    topic-name       = "__admin-user"
    extra-user-roles = "admin"
  }
  units = 1
}

The Data Integrator is configured with admin role, granting super.user permissions on the cluster. The bundle automatically integrates it with the Kafka broker.

Apply the changes:

terraform apply

Retrieve authentication credentials with:

juju run data-integrator/leader get-credentials

(Optional) Enable TLS encryption¶

To encrypt client-facing traffic, pass a cross-model offer URL from an existing TLS provider (e.g. self-signed-certificates) to the module. Add the following to your kafka.auto.tfvars file:

tls_offer = "<controller>:<owner>/<model>.certificates"

The module will integrate all Kafka applications with the TLS provider automatically.

Enable ingress for Kafka UI on Kubernetes¶

When deploying Kafka UI on K8s, provide an ingress offer:

ingress_offer = "<controller>:<owner>/<model>.ingress"

Remove the comment marker from the ingress_offer = var.ingress_offer module argument in main.tf. The VM module does not accept or require this argument.

(Optional) Enable observability with COS¶

To connect the cluster to the Canonical Observability Stack (COS), provide the three required cross-model offer URLs. Add the following to your kafka.auto.tfvars file:

cos_offers = {
  dashboard = "<controller>:<owner>/<cos-model>.grafana-dashboards"
  metrics   = "<controller>:<owner>/<cos-model>.prometheus-receive-remote-write"
  logging   = "<controller>:<owner>/<cos-model>.loki-logging"
}

All three fields must be set together — the module validates that either all or none are provided. The metrics offer must be the Prometheus receive-remote-write endpoint: the module deploys an opentelemetry-collector that scrapes Kafka metrics and forwards them to COS via remote write.

Terraform module reference¶

See the Terraform module reference for the full list of input variables and outputs exposed by the Charmed Apache Kafka Terraform module.