How to enable monitoring (COS)
Both Charmed Kafka K8s and Charmed ZooKeeper K8s comes with the JMX exporter.
The metrics can be queried by accessing the http://<kafka-unit-ip>:9101/metrics
and http://<zookeeper-unit-ip>:9998/metrics
endpoints, respectively.
Additionally, the charm provides integration with the Canonical Observability Stack.
Prerequisites
- A deployed Charmed Kafka K8s and Charmed ZooKeeper K8s bundle
- A deployed
cos-lite
bundle in a Kubernetes environment
Offer interfaces via the COS controller
First, we will switch to the COS K8s environment and offer COS interfaces to be cross-model integrated with the Charmed Kafka K8s model.
To switch to the Kubernetes controller for the COS model, run
juju switch <k8s_cos_controller>:<cos_model_name>
To offer the COS interfaces, run
juju offer grafana:grafana-dashboard grafana-dashboards
juju offer loki:logging loki-logging
juju offer prometheus:receive-remote-write prometheus-receive-remote-write
Consume offers via the Kafka model
Next, we will switch to the Charmed Kafka K8s model, find offers, and consume them.
We are currently on the Kubernetes controller for the COS model. To switch to the Kafka model, run
juju switch <k8s_db_controller>:<kafka_model_name>
To find offers, run the following command (make sure not to miss the “:” at the end!):
juju find-offers <k8s_cos_controller>:
The output should be similar to the sample below, where k8s
is the K8s controller name and cos
is the model where cos-lite
has been deployed:
Store URL Access Interfaces
k8s admin/cos.grafana-dashboards admin grafana_dashboard:grafana-dashboard
k8s admin/cos.loki-logging admin loki_push_api:logging
k8s admin/cos.prometheus-receive-remote-write admin prometheus_remote_write:receive-remote-write
...
To consume offers to be reachable in the current model, run
juju consume <k8s_cos_controller>:admin/<cos_model_name>.grafana-dashboards
juju consume <k8s_cos_controller>:admin/<cos_model_name>.loki-logging
juju consume <k8s_cos_controller>:admin/<cos_model_name>.prometheus-receive-remote-write
Deploy and integrate Grafana
First, deploy grafana-agent-k8s:
juju deploy grafana-agent-k8s --trust
Then, integrate grafana-agent-k8s
with consumed COS offers:
juju integrate grafana-agent-k8s grafana-dashboards
juju integrate grafana-agent-k8s loki-logging
juju integrate grafana-agent-k8s prometheus-receive-remote-write
Finally, integrate (previously known as “relate”) it with Charmed Kafka K8s
juju integrate grafana-agent-k8s kafka-k8s:grafana-dashboard
juju integrate grafana-agent-k8s kafka-k8s:logging
juju integrate grafana-agent-k8s kafka-k8s:metrics-endpoint
and Charmed ZooKeeper K8s
juju integrate grafana-agent-k8s zookeeper-k8s:grafana-dashboard
juju integrate grafana-agent-k8s zookeeper-k8s:logging
juju integrate grafana-agent-k8s zookeeper-k8s:metrics-endpoint
Wait for all components to settle down on a active/idle
state on both
models, e.g. <kafka_model_name>
and <cos_model_name>
.
After this is complete, the monitoring COS stack should be up and running and ready to be used.
Connect Grafana web interface
To connect to the Grafana web interface, follow the Browse dashboards section of the MicroK8s “Getting started” guide.
juju run grafana/leader get-admin-password --model <k8s_cos_controller>:<cos_model_name>
Tune server logging level
To tune the level of the server logs for Kafka and ZooKeeper, configure the log-level
and log_level
properties accordingly.
Kafka
juju config kafka log_level=<LOG_LEVEL>
Possible values are ERROR
, WARNING
, INFO
, DEBUG
.
ZooKeeper
juju config kafka log-level=<LOG_LEVEL>
Possible values are ERROR
, WARNING
, INFO
, DEBUG
.