How to integrate with the Canonical Observability Stack¶

gopkg-k8s exposes Prometheus metrics, forwards its structured logs, and ships a Grafana dashboard and alert rules. Integrate its metrics-endpoint, logging, and grafana-dashboard endpoints with the Canonical Observability Stack (COS) to use them. For what each endpoint carries and the metrics and alert rules the charm provides, see Integrations.

These steps assume that gopkg-k8s and nginx-ingress-integrator are deployed and integrated, as they are after the deployment steps of Deploy and verify gopkg-k8s on Kubernetes and before its clean-up section. They deploy the three COS charms into the same model, which is enough to see the integrations work locally. A production deployment keeps COS in its own model and integrates through cross-model offers; the COS documentation describes that layout.

Deploy the observability charms¶

juju deploy prometheus-k8s --channel=2/stable --trust
juju deploy loki-k8s --channel=2/stable --trust
juju deploy grafana-k8s --channel=2/stable --trust

Integrate the endpoints¶

juju integrate gopkg-k8s:metrics-endpoint prometheus-k8s:metrics-endpoint
juju integrate gopkg-k8s:logging loki-k8s:logging
juju integrate gopkg-k8s:grafana-dashboard grafana-k8s:grafana-dashboard

Wait until every application is active:

juju status --relations --watch 2s

The COS charms take a few minutes, and each restarts its own pod once after it first reports active. If one of them stays blocked with Kubernetes resources patch failed: Unauthorized for more than a couple of minutes, it has hit a known race and will not recover by itself: follow COS charm blocked on a Kubernetes patch to redeploy that charm, then continue here.

Verify that Prometheus scrapes the service¶

Prometheus scrapes /metrics on the application port of every unit. Ask its API for the up series of the application; a value of 1 means the last scrape succeeded. Reach the API through the Kubernetes Service that Juju maintains for prometheus-k8s. Its address is the one juju status shows for the application, and it survives pod replacement, which the COS charms trigger shortly after they first report active, when they set resource limits on their own pods. The loop retries until the first scrape completes and gives up after ten minutes:

export PROMETHEUS_IP=$(microk8s kubectl -n gopkg-k8s get service \
  prometheus-k8s -o jsonpath='{.spec.clusterIP}')
echo "${PROMETHEUS_IP}"
timeout 600 bash -c '
  until curl --silent --show-error --get \
      "http://${PROMETHEUS_IP}:9090/api/v1/query" \
      --data-urlencode "query=up{juju_application=\"gopkg-k8s\"}" \
      | grep -F "\"1\"]"; do
    sleep 10
  done
' || { echo "Prometheus has not scraped gopkg-k8s successfully" >&2; false; }

echo prints the Service address, such as 10.152.183.148. The loop prints nothing while it waits; when the scrape succeeds, its output is a JSON document whose result entry ends in "1". If it gives up after ten minutes, it prints the message on the last line instead: follow Prometheus does not report the service as up before going on.

Generate traffic and view the dashboards¶

The dashboards show nothing until the service has handled requests. Send two minutes of mixed traffic through ingress: package pages, go-get queries, a package that does not exist, and health checks, so that every panel has data. The loop stops by itself and leaves nothing behind; stop it early with Ctrl-C:

export INGRESS_HOST=gopkg.example.com
SECONDS=0
while [ "${SECONDS}" -lt 120 ]; do
  for path in /yaml.v2 "/yaml.v2?go-get=1" /mgo.v2 /check.v1 \
      /does-not-exist.v9 /health-check; do
    curl --silent --output /dev/null "http://${INGRESS_HOST}${path}" \
      --resolve "${INGRESS_HOST}:80:127.0.0.1"
  done
  sleep 1
done

Package requests make the service look up references on GitHub, so the loop also exercises the upstream and cache panels.

Grafana has no route out of the VM, so fetch its administrator password and then, in a second terminal on the VM, forward its port to every interface of the VM:

juju run grafana-k8s/0 get-admin-password
microk8s kubectl -n gopkg-k8s port-forward --address 0.0.0.0 \
  svc/grafana-k8s 3000:3000

In a browser on your workstation, open http://<vm-address>:3000, where <vm-address> is the VM’s address from multipass info charm-dev on the host, and log in as admin with that password. Under Dashboards, gopkg Overview shows the traffic you just sent as request rate by route, HTTP error rate, latency, upstream failure rate and refs cache hit ratio; only its git upload-pack panel stays empty, because that needs a real git clone. Go Operator is the framework’s dashboard. In Explore, the Loki data source shows one JSON log record per request other than a health check when filtered by juju_application="gopkg-k8s". Press Ctrl-C in the second terminal to stop the port forward; nothing else needs cleaning up.

Keep the metrics endpoint off the public hostname¶

By default the metrics endpoint shares the application port, so ingress publishes it at /metrics on the public hostname. Move it to a port that ingress does not route. The charm passes the new port to the service, which opens a second listener for the metrics path only, and updates the scrape job:

juju config gopkg-k8s metrics-port=9102

Confirm that Prometheus scrapes the new port, then that the public hostname no longer serves metrics. The up series cannot tell the ports apart, because the Prometheus charm rewrites its instance label to the Juju topology, so ask the targets API instead: it lists the scrape URL and health of every active target. The loop waits until the target whose URL ends in :9102/metrics reports up:

timeout 600 bash -c '
  until curl --silent --show-error \
      "http://${PROMETHEUS_IP}:9090/api/v1/targets?state=active" \
      | grep --only-matching "\"scrapeUrl\":\"[^\"]*:9102/metrics\"[^}]*\"health\":\"up\""; do
    sleep 10
  done
' || { echo "Prometheus has no healthy target on port 9102" >&2; false; }
export INGRESS_HOST=gopkg.example.com
curl --silent --output /dev/null --write-out '%{http_code}\n' \
  http://${INGRESS_HOST}/metrics \
  --resolve ${INGRESS_HOST}:80:127.0.0.1 | grep -Fx 404

The first command prints the matching part of the target entry, from its scrape URL to "health":"up", or the message on its last line if no such target appears within ten minutes (see Prometheus does not report the service as up). The second prints 404, because the application answers its own not-found page for that path.