How to deploy on AWS

Amazon Web Services is a popular subsidiary of Amazon that provides on-demand cloud computing platforms on a metered pay-as-you-go basis. Access the AWS web console at console.aws.amazon.com.

Choose the target substrate. The VM procedure uses EC2 through Juju’s AWS cloud; the K8s procedure creates an Amazon Elastic Kubernetes Service (EKS) cluster.

VM

Install AWS and Juju tooling

Install Juju via snap:

sudo snap install juju

Follow the AWS documentation for guidance on how to install the Amazon Web Services CLI.

To check whether both Juju and AWS CLI are correctly installed, run commands to display their versions:

juju version
aws --version

Output example

3.5.4-genericlinux-amd64
aws-cli/2.13.25 Python/3.11.5 Linux/6.2.0-33-generic exe/x86_64.ubuntu.23 prompt/off

Authenticate

Create an IAM account or use legacy user access keys and secret key to operate AWS EC2:

mkdir -p ~/.aws && cat <<- EOF >  ~/.aws/credentials.yaml
credentials:
  aws:
    NAME_OF_YOUR_CREDENTIAL:
      auth-type: access-key
      access-key: SECRET_ACCESS_KEY_ID
      secret-key: SECRET_ACCESS_KEY_VALUE
EOF

Bootstrap Juju controller on AWS EC2

Add AWS credentials to Juju:

juju add-credential aws -f ~/.aws/credentials.yaml

Bootstrap Juju controller (check all supported configuration options):

juju bootstrap aws <CONTROLLER_NAME>

Output example

Creating Juju controller "aws-us-east-1" on aws/us-east-1
Looking for packaged Juju agent version 3.5.4 for amd64
Located Juju agent version 3.5.4-ubuntu-amd64 at https://juju-dist-aws.s3.amazonaws.com/agents/agent/3.5.4/juju-3.5.4-linux-amd64.tgz
Launching controller instance(s) on aws/us-east-1...
 - i-0f4615983d113166d (arch=amd64 mem=8G cores=2)           
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 54.226.221.6:22
Attempting to connect to 172.31.20.34:22
Connected to 54.226.221.6
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 54.226.221.6 to verify accessibility...

Bootstrap complete, controller "aws-us-east-1" is now available
Controller machines are in the "controller" model

Now you can run
	juju add-model <model-name>
to create a new model to deploy workloads.

Deploy charms

Create a new Juju model, if needed:

juju add-model <MODEL_NAME>

(Optional) If you are troubleshooting charms and wish to see DEBUG logs:

juju model-config logging-config='<root>=INFO;unit=DEBUG'

Deploy Charmed Apache Kafka:

juju deploy kafka -n 3 --channel 4/stable --config roles=broker,controller

To use a specific instance type, add --constraints "instance-type=<INSTANCE_TYPE>".

Caution

This co-located broker/controller topology is for evaluation. For a production deployment, use separate broker and controller applications as described in the general deployment guide.

Caution

The smallest AWS instance types may not provide sufficient resources to host an Apache Kafka broker. We recommend choosing an instance type with a minimum of 8 GB of RAM and 4 CPU cores, such as m7i.xlarge.

For more guidance on sizing production environments, see the Requirements page. Additional information about AWS instance types is available in the AWS documentation.

We also recommend to deploy a Data Integrator for creating an admin user to manage the content of the Kafka cluster:

juju deploy data-integrator \
  --config extra-user-roles=admin \
  --config topic-name=__admin-user

And integrate it with the Kafka application:

juju integrate kafka data-integrator

For more information on Data Integrator and how to use it, please refer to the how-to manage client connections guide.

Clean up

Caution

Always clean AWS resources that are no longer necessary! Abandoned resources are tricky to detect and they can become expensive over time.

To list all controllers that have been registered to your local client, use the juju controllers command.

To destroy the Juju controller and remove AWS instance (Warning: all your data will be permanently deleted):

juju destroy-controller <CONTROLLER_NAME> --destroy-all-models --destroy-storage --force

Should the destroying process take a long time or be seemingly stuck, proceed to delete EC2 resources also manually via the AWS portal. See Amazon AWS documentation for more information on how to remove active resources no longer needed.

After destroying the controller, check and manually delete all unnecessary AWS EC2 instances, to show the list of all your EC2 instances run the following command (make sure to use the correct region):

aws ec2 describe-instances --region us-east-1 --query "Reservations[].Instances[*].{InstanceType: InstanceType, InstanceId: InstanceId, State: State.Name}" --output table

Output example

-------------------------------------------------------
|                  DescribeInstances                  |
+---------------------+----------------+--------------+
|     InstanceId      | InstanceType   |    State     |
+---------------------+----------------+--------------+
|  i-0f374435695ffc54c|  m7i.xlarge    |  terminated  |
|  i-0e1e8279f6b2a08e0|  m7i.xlarge    |  terminated  |
|  i-061e0d10d36c8cffe|  m7i.xlarge    |  terminated  |
|  i-0f4615983d113166d|  m7i.xlarge    |  terminated  |
+---------------------+----------------+--------------+

List your Juju credentials with the juju credentials command:

...
Client Credentials:
Cloud        Credentials
aws          NAME_OF_YOUR_CREDENTIAL
...

Remove AWS EC2 CLI credentials from Juju:

juju remove-credential aws NAME_OF_YOUR_CREDENTIAL

Finally, remove AWS CLI user credentials (to avoid forgetting and leaking):

rm -f ~/.aws/credentials.yaml

K8s

Install tooling and authenticate

Install Juju and kubectl, then follow the installation instructions for eksctl and the AWS CLI:

sudo snap install juju --channel 3.6/stable
sudo snap install kubectl --classic
aws configure
aws sts get-caller-identity

Create an EKS cluster

Export a unique cluster name for further use:

export JUJU_NAME=eks-$USER-$RANDOM

Save the following as cluster.yaml. This example enables the EBS CSI driver required for persistent volumes and uses three workers for a non-production evaluation deployment:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${JUJU_NAME}
  region: eu-west-3
iam:
  withOIDC: true
addons:
  - name: aws-ebs-csi-driver
    wellKnownPolicies:
      ebsCSIController: true
nodeGroups:
  - name: kafka
    minSize: 3
    maxSize: 5
    instanceType: m5.xlarge

Create the cluster:

eksctl create cluster -f cluster.yaml

Select a currently supported EKS/Kubernetes version for your region rather than copying an old version from an example.

Bootstrap Juju and deploy

juju add-k8s $JUJU_NAME
juju bootstrap $JUJU_NAME
juju add-model <MODEL_NAME>
juju deploy kafka-k8s -n 3 --channel 4/stable --trust --config roles=broker,controller kafka
juju deploy data-integrator admin --channel stable \
  --config extra-user-roles=admin \
  --config topic-name=admin-topic
juju integrate kafka admin

This co-located broker/controller topology is for testing. For a production deployment, use separate broker and controller applications and size the worker nodes according to the requirements, as described in the general deployment guide.

Inspect the environment with kubectl cluster-info, eksctl get cluster -A, and kubectl get nodes.

Clean up

Caution

Always remove EKS resources that are no longer needed; they can be costly.

juju destroy-controller $JUJU_NAME --yes --destroy-all-models --destroy-storage --force
juju remove-cloud $JUJU_NAME

Before deleting the cluster, check for Services with external load balancers that Juju may not remove automatically — they continue to incur AWS charges while the cluster exists:

kubectl get svc --all-namespaces | grep -v "ClusterIP"

Delete any remaining Services with an EXTERNAL-IP (for example, the Kafka NodePort/LoadBalancer Services created by the charm) before deleting the cluster:

kubectl delete svc <service-name> -n <MODEL_NAME>

Then delete the EKS cluster itself:

eksctl delete cluster $JUJU_NAME --region eu-west-3 --force --disable-nodegroup-eviction

Check for cloud resources that were not removed automatically before deleting local AWS credentials.