How to manage units

For general Juju unit management process, see the Juju documentation.

Scaling

Note

Scaling a Charmed Apache Kafka cluster does not automatically rebalance existing topics and partitions. Rebalancing must be performed manually—before scaling in or after scaling out.

Add units

To scale-out Charmed Apache Kafka application, add more units:

juju add-unit kafka -n <num_brokers_to_add>

See the juju add-unit command reference.

juju scale-application kafka <desired-units>

See the juju scale-application command reference.

Make sure to reassign partitions and topics to use newly added units. See below for guidance.

Remove units

Caution

Reassign partitions before scaling in to ensure that decommissioned units do not hold any data. Failing to do so may lead to data loss.

To decrease the number of Apache Kafka brokers, remove some existing units from the Charmed Apache Kafka application:

juju remove-unit kafka/1 kafka/2

See the juju remove-unit command reference.

juju scale-application kafka <desired-units>

Partition reassignment

When brokers are added or removed, Apache Kafka does not automatically rebalance existing topics and partitions across the new set of brokers.

Without reassignment or rebalancing:

  • New storages and new brokers will be used only when new topics and new partitions are created.

  • Removing a broker can result in permanent data loss if the partitions are not replicated on another broker.

Partition reassignment can still be done manually by the admin user with the Apache Kafka reassignment utility.

Use the charmed-kafka.reassign-partitions snap command.

Use /opt/kafka/bin/kafka-reassign-partitions.sh in the kafka workload container.

For more information on the script usage, refer to Apache Kafka documentation.

LinkedIn’s Cruise Control can be used for semi-automatic rebalancing. The partition rebalancing tutorial demonstrates the workflow for VM deployments; use the same charm actions with the kafka application on Kubernetes.

Admin utility scripts

Apache Kafka ships with bin/*.sh commands to do various administrative tasks such as:

  • bin/kafka-configs.sh to update cluster configuration

  • bin/kafka-topics.sh for topic management

  • bin/kafka-acls.sh for management of ACLs of Apache Kafka users

Please refer to the upstream Apache Kafka project and its documentation, for a full list of the bash commands available in Apache Kafka distributions. Additionally, you can use --help argument to print a short summary for a given bash command.

The most important commands are also exposed via the Charmed Apache Kafka snap, accessible via charmed-kafka.<command>. For the mapping between machine snap entry points and Kubernetes executables, see the command-line utilities reference page.

The scripts are available in /opt/kafka/bin inside the kafka workload container. Use juju ssh --container kafka to run them.

Caution

Before running bash scripts, make sure that some listeners have been correctly opened by creating appropriate integrations.

For more information about how listeners are opened based on relations, see the Listeners. For example, to open a SASL/SCRAM listener, integrate a client application using the data integrator, as described in the How to manage client connections guide.

To run most of the scripts, you need to provide:

  1. the Apache Kafka service endpoints, generally referred to as bootstrap servers

  2. authentication information

Endpoints and credentials

For Juju admins of the Apache Kafka deployment, the bootstrap servers information can be obtained using:

Admin client authentication information is stored in the /var/snap/charmed-kafka/current/etc/kafka/client.properties file that is present on every Apache Kafka broker. The content of the file can be accessed using juju ssh command:

juju ssh kafka/leader 'cat /var/snap/charmed-kafka/current/etc/kafka/client.properties'

The file contains the bootstrap.servers entry for the internal listener, which can be extracted as follows:

BOOTSTRAP_SERVERS=$(juju ssh kafka/leader 'grep "^bootstrap.servers=" /var/snap/charmed-kafka/current/etc/kafka/client.properties' | cut -d "=" -f 2)

Admin client authentication information is stored in /etc/kafka/client.properties in each broker’s workload container:

juju ssh --container kafka kafka/leader 'cat /etc/kafka/client.properties'

The file contains the bootstrap.servers entry for the internal listener, which can be extracted as follows:

BOOTSTRAP_SERVERS=$(juju ssh --container kafka kafka/leader 'grep "^bootstrap.servers=" /etc/kafka/client.properties' | cut -d "=" -f 2)

Note

The client.properties file is generated for the internal SASL/SSL listener used by cluster administrators. It is not tied to any external client listener that may be opened through relations. The get-listeners action lists all available listeners, including external ones, but the internal configuration file always targets the internal listener.

This file can be provided to the Apache Kafka bin commands via the --command-config argument. Note that client.properties may also refer to other files (e.g. truststore and keystore for TLS-enabled connections). Those files also need to be accessible and correctly specified.

Commands can also be run within an Apache Kafka broker, since both the authentication file (along with the truststore if needed) and the Apache Kafka utilities are already present. For example, see below.

List topics

To list the current topics on the Apache Kafka cluster, using credentials from inside the cluster, run:

juju ssh kafka/leader "charmed-kafka.topics --bootstrap-server $BOOTSTRAP_SERVERS --list --command-config /var/snap/charmed-kafka/current/etc/kafka/client.properties"

juju ssh --container kafka kafka/leader \
  "/opt/kafka/bin/kafka-topics.sh --bootstrap-server $BOOTSTRAP_SERVERS --list --command-config /etc/kafka/client.properties"

The BOOTSTRAP_SERVERS variable contains the information we retrieved earlier in the previous section.

Juju external users

For external users managed by the Data Integrator Charm, the endpoints and credentials can be fetched using the dedicated action

juju run data-integrator/leader get-credentials --format yaml

Create a new client.properties file for the external user, rather than copying the one from the brokers: the broker file targets the internal listener and references internal TLS key material (peer keystore/truststore) that external clients must not use.

Fetch the information using juju commands:

BOOTSTRAP_SERVERS=$(juju run data-integrator/leader get-credentials --format yaml | yq .kafka.endpoints )
USERNAME=$(juju run data-integrator/leader get-credentials --format yaml | yq .kafka.username )
PASSWORD=$(juju run data-integrator/leader get-credentials --format yaml | yq .kafka.password )

Then write a client.properties file with the following content:

sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="<USERNAME>" password="<PASSWORD>";
sasl.mechanism=SCRAM-SHA-512
security.protocol=SASL_PLAINTEXT
bootstrap.servers=<BOOTSTRAP_SERVERS>

Note

If TLS encryption is enabled for client connections, set security.protocol=SASL_SSL instead, and add the TLS properties (e.g. ssl.truststore.location) pointing to the CA certificate provided through the data-integrator relation — not to the broker’s internal peer keystore/truststore files.