<a id="integrate-with-cos"></a>

# How to integrate with COS Lite

`gopkg-k8s` exposes Prometheus metrics, forwards its structured logs,
and ships a Grafana dashboard and alert rules. Integrate its
`metrics-endpoint`, `logging`, and `grafana-dashboard` endpoints with
COS Lite, the lightweight Canonical Observability Stack (COS) built on
Prometheus, Loki, and Grafana, to use them. For what each endpoint
carries and the metrics and alert rules the charm provides, see
[Integrations](https://canonical.com/juju/docs/gopkg-charm/latest/reference/integrations/index.html.md#integrations).

These steps assume that `gopkg-k8s` and `nginx-ingress-integrator`
are deployed and integrated, as they are after the deployment steps of
[Deploy and verify gopkg-k8s on Kubernetes](https://canonical.com/juju/docs/gopkg-charm/latest/tutorials/deploy-and-verify-on-kubernetes/index.html.md#deploy-and-verify-on-kubernetes) and before its clean-up section. They
deploy the three COS Lite charms into the same model, which is enough to see
the integrations work locally. A production deployment keeps COS Lite in its
own model and integrates through cross-model offers; the [COS documentation](https://documentation.ubuntu.com/observability/) describes that layout.
The full COS keeps metrics in Mimir, which receives them from an
OpenTelemetry Collector instead of scraping them, so these steps do not apply
to it as written.

## Deploy the observability charms

```bash
juju deploy prometheus-k8s --channel=2/stable --trust
juju deploy loki-k8s --channel=2/stable --trust
juju deploy grafana-k8s --channel=2/stable --trust
```

## Integrate the endpoints

```bash
juju integrate gopkg-k8s:metrics-endpoint prometheus-k8s:metrics-endpoint
juju integrate gopkg-k8s:logging loki-k8s:logging
juju integrate gopkg-k8s:grafana-dashboard grafana-k8s:grafana-dashboard
```

Wait until every application is active:

<!-- SPREAD SKIP -->
```bash
juju status --relations --watch 2s
```

<!-- SPREAD SKIP END -->
<!-- SPREAD
for application in gopkg-k8s prometheus-k8s loki-k8s grafana-k8s; do
  juju wait-for application "${application}" \
    --query='status=="active"' --timeout=20m
done -->
<!-- SPREAD END -->

The COS Lite charms take a few minutes, and each restarts its own pod once
after it first reports active. If one of them stays `blocked` with
`Kubernetes resources patch failed: Unauthorized` for more than a couple of
minutes, it has hit a known race and will not recover by itself: follow
[COS Lite charm blocked on a Kubernetes patch](https://canonical.com/juju/docs/gopkg-charm/latest/how-to/troubleshoot-deployment/index.html.md#cos-charm-blocked-patch-unauthorized) to redeploy that charm, then
continue here.

## Verify that Prometheus scrapes the service

Prometheus scrapes `/metrics` on the application port of every unit. Ask its
API for the `up` series of the application; a value of `1` means the last
scrape succeeded. Reach the API through the Kubernetes Service that Juju
maintains for `prometheus-k8s`. Its address is the one `juju status` shows
for the application, and it survives pod replacement, which the COS Lite
charms trigger shortly after they first report active, when they set resource
limits on their own pods. The loop retries until the first scrape completes
and gives up after ten minutes:

```bash
export PROMETHEUS_IP=$(microk8s kubectl -n gopkg-k8s get service \
  prometheus-k8s -o jsonpath='{.spec.clusterIP}')
echo "${PROMETHEUS_IP}"
timeout 600 bash -c '
  until curl --silent --show-error --get \
      "http://${PROMETHEUS_IP}:9090/api/v1/query" \
      --data-urlencode "query=up{juju_application=\"gopkg-k8s\"}" \
      | grep -F "\"1\"]"; do
    sleep 10
  done
' || { echo "Prometheus has not scraped gopkg-k8s successfully" >&2; false; }
```

`echo` prints the Service address, such as `10.152.183.148`. The loop
prints nothing while it waits; when the scrape succeeds, its output is a
JSON document whose `result` entry ends in `"1"`. If it gives up after
ten minutes, it prints the message on the last line instead: follow
[Prometheus does not report the service as up](https://canonical.com/juju/docs/gopkg-charm/latest/how-to/troubleshoot-deployment/index.html.md#prometheus-does-not-scrape) before going on.

## Generate traffic and view the dashboards

The dashboards show nothing until the service has handled requests. Send
two minutes of mixed traffic through ingress: package pages, `go-get`
queries, a package that does not exist, and health checks, so that every
panel has data. The loop stops by itself and leaves nothing behind; stop it
early with Ctrl-C:

```bash
export INGRESS_HOST=gopkg.example.com
SECONDS=0
while [ "${SECONDS}" -lt 120 ]; do
  for path in /yaml.v2 "/yaml.v2?go-get=1" /mgo.v2 /check.v1 \
      /does-not-exist.v9 /health-check; do
    curl --silent --output /dev/null "http://${INGRESS_HOST}${path}" \
      --resolve "${INGRESS_HOST}:80:127.0.0.1"
  done
  sleep 1
done
```

Package requests make the service look up references on GitHub, so the
loop also exercises the upstream and cache panels.

Grafana has no route out of the VM, so fetch its administrator password and
then, in a second terminal on the VM, forward its port to every interface
of the VM:

<!-- SPREAD SKIP -->
```bash
juju run grafana-k8s/0 get-admin-password
microk8s kubectl -n gopkg-k8s port-forward --address 0.0.0.0 \
  svc/grafana-k8s 3000:3000
```

<!-- SPREAD SKIP END -->

In a browser on your workstation, open `http://<vm-address>:3000`, where
`<vm-address>` is the VM’s address from `multipass info charm-dev` on
the host, and log in as `admin` with that password. Under **Dashboards**,
**gopkg Overview** shows the traffic you just sent as request rate by route,
HTTP error rate, latency, upstream failure rate and refs cache hit ratio;
only its git upload-pack panel stays empty, because that needs a real
`git clone`. **Go Operator** is the framework’s dashboard. In
**Explore**, the Loki data source shows one
JSON log record per request other than a health check when filtered by
`juju_application="gopkg-k8s"`. Press Ctrl-C in the second terminal to
stop the port forward; nothing else needs cleaning up.

<a id="cos-metrics-port"></a>

## Keep the metrics endpoint off the public hostname

By default the metrics endpoint shares the application port, so ingress
publishes it at `/metrics` on the public hostname. Move it to a port that
ingress does not route. The charm passes the new port to the service, which
opens a second listener for the metrics path only, and updates the scrape
job:

```bash
juju config gopkg-k8s metrics-port=9102
```

Confirm that Prometheus scrapes the new port, then that the public hostname
no longer serves metrics. The `up` series cannot tell the ports apart,
because the Prometheus charm rewrites its `instance` label to the Juju
topology, so ask the targets API instead: it lists the scrape URL and health
of every active target. The loop waits until the target whose URL ends in
`:9102/metrics` reports `up`:

```bash
timeout 600 bash -c '
  until curl --silent --show-error \
      "http://${PROMETHEUS_IP}:9090/api/v1/targets?state=active" \
      | grep --only-matching "\"scrapeUrl\":\"[^\"]*:9102/metrics\"[^}]*\"health\":\"up\""; do
    sleep 10
  done
' || { echo "Prometheus has no healthy target on port 9102" >&2; false; }
export INGRESS_HOST=gopkg.example.com
curl --silent --output /dev/null --write-out '%{http_code}\n' \
  http://${INGRESS_HOST}/metrics \
  --resolve ${INGRESS_HOST}:80:127.0.0.1 | grep -Fx 404
```

The first command prints the matching part of the target entry, from its
scrape URL to `"health":"up"`, or the message on its last line if no such
target appears within ten minutes (see [Prometheus does not report the service as up](https://canonical.com/juju/docs/gopkg-charm/latest/how-to/troubleshoot-deployment/index.html.md#prometheus-does-not-scrape)).
The second prints `404`, because the application answers its own not-found
page for that path.