Tutorial

The Charmed Apache Kafka Operator delivers automated operations management from Day 0 to Day 2 on the Apache Kafka event streaming platform. It is an open source, end-to-end, production-ready data platform on top of Juju. As a first step this tutorial shows you how to get Charmed Apache Kafka up and running, but the tutorial does not stop there. Through this tutorial, you will learn a variety of operations, everything from adding replicas to advanced operations such as enabling SSL encryption, cross-cluster asynchronous replication and more.

This tutorial covers both substrates: the VM charm kafka on a local LXD cloud, and the K8s charm kafka-k8s on a local MicroK8s cloud. Each page has VM and K8s tabs, and your selection is remembered as you move between pages. Both substrates deploy the application under the same kafka alias, so the commands are identical unless the substrates genuinely differ.

Note

The example outputs in this tutorial are captured from a VM deployment; the automated tutorial tests exercise this path end-to-end. Outputs on K8s may differ slightly (for example, pod addresses instead of machine addresses).

In this tutorial, we will walk through how to:

  • Set up your local environment using LXD or MicroK8s, and Juju.

  • Deploy Charmed Apache Kafka using only a few commands.

  • Get the admin credentials directly.

  • Add high-availability with replication.

  • Change the admin password.

  • Automatically create Apache Kafka users via Juju relations.

  • Use Cruise Control for cluster rebalancing.

  • Use Apache Kafka Connect for moving data between data applications.

While this tutorial intends to guide and teach you as you deploy Charmed Apache Kafka, it will be most beneficial if you already have a familiarity with:

  • Basic Unix shell commands.

  • General data-intensive application concepts such as partitioning, replication and user management.

Minimum requirements

Before we start, make sure your machine meets the following requirements:

  • Ubuntu 24.04 LTS (Noble) or later.

  • 8 GB of RAM.

  • 2 CPU cores.

  • At least 20 GB of available storage.

  • Access to the internet for downloading the required snaps and charms.

Note

The full tutorial deploys several applications (Apache Kafka, KRaft controllers, Kafka Connect, PostgreSQL and OpenSearch). If you plan to complete the ETL and rebalancing chapters, ensure at least 4 GB more RAM is available than the minimum above, and prefer the testing profile shown in the deployment steps.

Step-by-step guide

Here’s an overview of the steps required with links to our separate tutorials that deal with each individual step: