How to deploy Charmed Apache Kafka

This guide provides deployment instructions for Charmed Apache Kafka using the Juju CLI, covering both the VM operator and the Kubernetes operator. Use the tabs below to switch between the two substrates – your selection is remembered as you scroll through the rest of the page.

Platform-specific steps are also available:

To deploy a Charmed Apache Kafka cluster on a bare environment, it is necessary to:

  1. Set up a Juju controller

  2. Set up a Juju model

  3. Deploy Charmed Apache Kafka

  4. Create an external admin user

In the next subsections, we cover these steps separately by referring to relevant Juju documentation and providing details on the Charmed Apache Kafka specifics for each substrate. If you already have a Juju controller and/or a Juju model, skip the associated steps.

Juju controller setup

Make sure you have a Juju controller accessible from your local environment using the Juju client snap.

List available controllers:

juju list-controllers

Switch to another controller if needed:

juju switch <controller>

If there are no suitable controllers, create a new one:

juju bootstrap <cloud> <controller>

The command is the same on both substrates; only the backing cloud differs. On VM, the controller’s back-end cloud must not be Kubernetes-based (e.g. localhost for a LXD cloud). On K8s, it must be Kubernetes-based (e.g. microk8s). Retrieve the cloud information with juju list-controllers.

<cloud> – the cloud to deploy the controller to.

For more information on how to set up a new cloud, see the How to manage clouds guide in the Juju documentation. For more controller setup guidance, see the How to manage controllers guide.

Juju model setup

Create a new Juju model using:

juju add-model <model>

Alternatively, switch to any existing Juju model:

juju switch <model-name>

Make sure that the model is of the correct type:

juju show-model | yq '.[]."model-type"'

The model type must be iaas on VM and caas on K8s.

Deploy Charmed Apache Kafka for production

Charmed Apache Kafka for production use-cases is deployed as follows:

juju deploy kafka -n <broker-units> --config roles=broker --channel 4/stable
juju deploy kafka -n <controller-units> --config roles=controller --channel 4/stable controller

  • <broker-units> – the number of units to deploy for Charmed Apache Kafka brokers

  • <controller-units> – the number of units to deploy for KRaft controllers

Charmed Apache Kafka K8s for production use-cases is deployed as follows:

juju deploy kafka-k8s -n <broker-units> --config roles=broker --channel 4/stable --trust kafka
juju deploy kafka-k8s -n <controller-units> --config roles=controller --channel 4/stable controller --trust

  • <broker-units> – the number of units to deploy for Charmed Apache Kafka K8s brokers

  • <controller-units> – the number of units to deploy for KRaft controllers

Note

The --trust flag grants the charm the permissions it needs to manage Kubernetes resources (e.g. Services, StatefulSets). The trailing kafka assigns the same application name as on VM, so the rest of this guide uses kafka on both substrates.

To maintain high-availability of topic partitions, 3+ broker units and 3 or 5 controller units are recommended.

To exchange credentials and endpoints between the two clusters, integrate the broker and controller applications:

juju integrate kafka:peer-cluster-orchestrator controller:peer-cluster

Check the status of the deployment:

juju status

The deployment should be complete once all the units show active and idle status.

(Alternative) Deploy Charmed Apache Kafka for testing

In order to save resources for very-small, non-production test and staging clusters, it is possible to co-locate both the KRaft controller services and the broker services into a single application.

Warning

This is not recommended for any production deployments. Apache Kafka brokers rely on the KRaft controllers to coordinate – if both services go down at the same time, the risk of cluster instability increases.

Charmed Apache Kafka for testing use-cases is deployed as follows:

juju deploy kafka -n <kafka-units> --config roles=broker,controller --channel 4/stable

  • <kafka-units> – the number of units to deploy for Charmed Apache Kafka

Charmed Apache Kafka K8s for testing use-cases is deployed as follows:

juju deploy kafka-k8s -n <kafka-units> --config roles=broker,controller --channel 4/stable --trust kafka

  • <kafka-units> – the number of units to deploy for Charmed Apache Kafka K8s

Check the status of the deployment:

juju status

The deployment should be complete once all the units show active or idle status.

(Optional) Create an external admin user

Charmed Apache Kafka aims to follow the secure by default paradigm. As a consequence, after being deployed the Apache Kafka cluster won’t expose any external listeners – the cluster will be unreachable. Ports are only opened when client applications are integrated.

Note

For more information about the available listeners and protocols, refer to this table.

For most cluster administrators, it may be most helpful to create a user with the admin role, which has super.user permissions on the Apache Kafka cluster.

To create an admin user, deploy the Data Integrator charm with extra-user-roles set to admin:

juju deploy data-integrator --config topic-name="__admin-user" --config extra-user-roles="admin"

Now, integrate it with the Apache Kafka charm:

juju integrate data-integrator kafka

To retrieve authentication information, such as the username and password, use:

juju run data-integrator/leader get-credentials