1. Set up the environment

This is a part of the Charmed Apache Kafka Tutorial.

For this tutorial, we will need to set up the environment with two main components, and extra command-line tooling:

  • A cloud provisioner – LXD for the VM substrate, or MicroK8s for the Kubernetes substrate

  • Juju - enables us to deploy and manage Charmed Apache Kafka and related applications

  • yq - a command-line YAML processor

  • jq - a command-line JSON processor

Prepare the cloud

The fastest, simplest way to get started with Charmed Apache Kafka is to set up a local LXD cloud. LXD is a system container and virtual machine manager; Apache Kafka will be run in one of these containers and managed by Juju. While this tutorial covers the basics of LXD, you can learn more about LXD here.

LXD comes pre-installed on Ubuntu 24.04 LTS. Verify that LXD is installed by entering the command which lxd. This will output /snap/bin/lxd or, for some systems, /usr/sbin/lxd.

Although LXD is already installed, we need to run lxd init to perform post-installation tasks. For this tutorial, the default parameters are preferred and the network bridge should be set to have no IPv6 addresses since Juju does not support IPv6 addresses with LXD:

lxd init --auto
lxc network set lxdbr0 ipv6.address none

You can list all LXD containers by entering the command lxc list into the command line. However, at this point of the tutorial, none should exist and you’ll only see this as output:

+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

For the Kubernetes substrate, the simplest way to get started is a local MicroK8s cloud. Apache Kafka will run in pods on this cluster and be managed by Juju.

Install MicroK8s and add your user to its group:

sudo snap install microk8s --channel 1.32-strict/stable
sudo usermod -a -G snap_microk8s $USER
newgrp snap_microk8s

Enable the add-ons required by Charmed Apache Kafka K8s. The hostpath-storage add-on provides the persistent volumes used by broker storage:

sudo microk8s enable dns hostpath-storage
sudo microk8s status --wait-ready

Caution

hostpath-storage is suitable for this tutorial only. Production deployments require a proper storage class – see the deployment guide.

List all pods to confirm the cluster is running. At this point of the tutorial, only system pods should exist:

sudo microk8s kubectl get pods -A

Install and prepare Juju

Juju is an Operator Lifecycle Manager (OLM) for clouds, bare metal, LXD or Kubernetes. We will be using it to deploy and manage Charmed Apache Kafka. As may be true for LXD, Juju is installed from a snap package:

sudo snap install juju

Install yq, a YAML processor used to parse Juju output in later steps:

sudo snap install yq

Install jq, a JSON processor used in later steps:

sudo snap install jq

Juju already has built-in knowledge of LXD and MicroK8s and how they work, so there is no additional cloud setup or configuration needed. A Juju controller will be deployed, which will in turn manage the operations of Charmed Apache Kafka. All we need to do is bootstrap a Juju controller named overlord. This bootstrapping process can take several minutes depending on the resources available on your machine:

juju bootstrap localhost overlord

The Juju controller exists within an LXD container. To verify this, check the list of containers:

lxc list
Output example 
+---------------+---------+-----------------------+------+-----------+-----------+
|     NAME      |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+---------------+---------+-----------------------+------+-----------+-----------+
| juju-<id>     | RUNNING | 10.105.164.235 (eth0) |      | CONTAINER | 0         |
+---------------+---------+-----------------------+------+-----------+-----------+

where <id> is a unique combination of numbers and letters such as 9d7e4e-0.

juju bootstrap microk8s overlord

The Juju controller runs as a pod in the controller-overlord namespace. To verify this, list the pods:

sudo microk8s kubectl get pods -n controller-overlord
Output example 
NAME           READY   STATUS    RESTARTS   AGE
controller-0   3/3     Running   0          2m

The controller can work with different models; models host applications such as Charmed Apache Kafka. Set up a specific model for Charmed Apache Kafka named tutorial:

juju add-model tutorial

Check the status of the model you created:

juju status
Output example 
Model     Controller  Cloud/Region         Version  SLA          Timestamp
tutorial  overlord    localhost/localhost  3.6.13   unsupported  12:10:54Z

Model "admin/tutorial" is empty.
Output example 
Model     Controller  Cloud/Region         Version  SLA          Timestamp
tutorial  overlord    microk8s/localhost   3.6.13   unsupported  12:10:54Z

Model "admin/tutorial" is empty.