# How to manage logs > See also: [Log](https://documentation.ubuntu.com/juju/2.9/reference/log.md#log) #### TIP For an integrated solution consider [the Loki charm](https://charmhub.io/loki-k8s). ## Manage the logs ### Stream the logs To stream the logs in the current model (including agent logs and logs from charm code that uses the Python [`logging`](https://docs.python.org/3/library/logging.html) module), run the `debug-log` command: ```text juju debug-log ``` In a machine deployment, this will output logs for all the Juju machine and unit agents, starting with the last 10 lines, in the following format: ```text ``` The command has various options that allow you to control the length, appearance, amount and type of detail. > See more: [juju debug-log](https://documentation.ubuntu.com/juju/2.9/reference/juju-cli/list-of-juju-cli-commands/debug-log.md#command-juju-debug-log) ### Tips for Kubernetes View logs from the charm and workload containers with `kubectl logs`: ```text # Get logs from the charm container kubectl -n logs pods/prometheus-0 -c charm # Get logs from the workload container kubectl -n logs pods/prometheus-0 -c prometheus ``` > See more: [Kubernetes | `kubectl logs`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/) View logs from a particular Pebble service with `pebble logs`: ```text juju ssh --container prometheus prometheus/0 \ /charm/bin/pebble logs prometheus ``` > See more: [Pebble | `pebble logs`](https://documentation.ubuntu.com/pebble/reference/cli-commands/#reference-pebble-logs-command) View logs for a Pebble service that failed to start with `pebble services` and `pebble tasks`: ```text # get the ID of the failed change with pebble services: $ juju ssh --container prometheus prometheus/0 /charm/bin/pebble changes ID Status Spawn Ready Summary 30 Error yesterday at 21:31 UTC yesterday at 21:31 UTC Replan service "prometheus" 31 Done yesterday at 21:38 UTC yesterday at 21:38 UTC Execute command "/usr/bin/promtool" 32 Done yesterday at 21:38 UTC yesterday at 21:38 UTC Replan service "prometheus" # query for the logs with pebble tasks: $ juju ssh --container prometheus prometheus/0 /charm/bin/pebble tasks 30 Status Spawn Ready Summary Error yesterday at 21:31 UTC yesterday at 21:31 UTC Start service "prometheus" ...................................................................... Start service "prometheus" 2023-03-07T21:31:39Z INFO Most recent service output: (...) ts=2023-03-07T21:31:39.309Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 ts=2023-03-07T21:31:39.309Z caller=main.go:807 level=error msg="Unable to start web listener" err="listen tcp 0.0.0.0:9090: bind: address already in use" 2023-03-07T21:31:39Z ERROR cannot start service: exited quickly with code 1 ``` > See more: [Pebble | `pebble services`](https://documentation.ubuntu.com/pebble/reference/cli-commands/#reference-pebble-services-command), [Pebble | `pebble tasks`](https://documentation.ubuntu.com/pebble/reference/cli-commands/#reference-pebble-tasks-command) ### Configure the logging level Juju saves or discards logs according to the value of the model config key `logging-config`. Therefore, `logging-config` needs to be set before the events you want to collect logs for (i.e. before attempting to reproduce a bug). **Set values.** - **To change the logging configuration for machine and unit agents**:
Run the `model-config` command with the `logging-config` key set to a `"`-enclosed, semi-colon-separated list of `=` pairs. ### Examples Set machine agent logs to `WARNING` and unit agent logs to `TRACE`: ```text juju model-config logging-config="=WARNING;unit=TRACE" ``` Set unit agent logs for unit `0` of `mysql` to `DEBUG`: ```text juju model-config logging-config="unit.mysql/0=DEBUG" ``` > See more: [Configure a model](https://documentation.ubuntu.com/juju/2.9/howto/manage-models.md#configure-a-model), [logging-config](https://documentation.ubuntu.com/juju/2.9/reference/configuration/list-of-model-configuration-keys.md#model-config-logging-config) #### CAUTION **To avoid filling up the database unnecessarily:**
When verbose logging is no longer needed, return logging to normal levels! - **To change the logging configuration on a per-unit-agent basis:** 1. SSH into the unit’s machine. E.g., for `mysql/0`: ```text juju ssh mysql/0 ``` 1. Open the unit’s agent configuration file. For our example, it will be `/var/lib/juju/agents/unit-mysql-0/agent.conf`/ Then, find the `values` section, and add a line with the field `LOGGING_OVERRIDE` set to `juju=`, below `TRACE`. The bottom of the file should now look as below: ```text loggingconfig: =WARNING;unit=DEBUG values: CONTAINER_TYPE: "" NAMESPACE: "" LOGGING_OVERRIDE: juju=trace mongoversion: "0.0" ``` 1. Restart the affected agent: ```text sudo systemctl restart jujud-unit-mysql-0.service ``` **Get values.** To verify the current logging configuration for machine and unit agents, run `model-config` followed by the `logging-config` key: ```text juju model-config logging-config ``` Sample output: ```text =WARNING;unit=DEBUG;#http=TRACE ``` which means that the machine agent (``) log level is set to `WARNING`, the unit agent (`unit`) log level is set at `DEBUG`, and the `http` label is set to `TRACE`. > See more: [Configure a model](https://documentation.ubuntu.com/juju/2.9/howto/manage-models.md#configure-a-model), [logging-config](https://documentation.ubuntu.com/juju/2.9/reference/configuration/list-of-model-configuration-keys.md#model-config-logging-config) ### Forward logs to an external logsink You can optionally forward log messages to a remote syslog server over a secure TLS connection, on a per-model basis, as below: > See [Rsyslog documentation](http://www.rsyslog.com/doc/v8-stable/tutorials/tls_cert_summary.html) for help with security-related files (certificates, keys) and the configuration of the remote syslog server. 1. Configure the controller for remote logging by configuring it during controller creation as below: ```bash juju bootstrap --config mylogconfig.yaml ``` where the YAML file is as below: ```text syslog-host: : syslog-ca-cert: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- syslog-client-cert: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- syslog-client-key: | -----BEGIN PRIVATE KEY----- -----END PRIVATE KEY----- ``` > See more: [Configure a controller](https://documentation.ubuntu.com/juju/2.9/howto/manage-controllers.md#configure-a-controller) 1. Enable log forwarding for a model by configuring it as below: ```text juju model-config -m logforward-enabled=True ``` An initial 100 (maximum) existing log lines will be forwarded. > See more: [Configure a model](https://documentation.ubuntu.com/juju/2.9/howto/manage-models.md#configure-a-model) #### TIP You can configure remote logging *and* enable log forwarding on *all* the controller’s models in one step by running ```text juju bootstrap --config mylogconfig.yaml --config logforward-enabled=True ``` ## Manage the log files #### CAUTION Only applicable for machines – for Kubernetes logs are written directly to `stdout` of the container and can be retrieved with native Kubernetes methods, e.g., `kubectl logs -c -n ` . ### View the log files To view the Juju log files in a Juju machine: 1. Open a shell into the machine: (1a) If Juju can connect to the machine (i.e., the output contains `Connected to `) and the machine is fully provisioned, use `juju ssh`. For example, to connect to machine 0: ```text juju ssh 0 ``` (1b) If Juju can connect to the machine (i.e., the output contains `Connected to `) but the machine is not fully provisioned (e.g., command hangs at `Running machine configuration script...`), use the `ssh` command followed by the address of the machine and the path to the place where Juju stores your SSH keys (including the ones it generates automatically for you): ```text ssh ubuntu@ -i /ssh/juju_id_rsa ``` Here, `` defaults to `~/.local/share/juju`, but if you’ve set the `JUJU_DATA` environment variable, it will be equal to that instead. (1c) If Juju *cannot* connect to the machine (i.e., the command never reaches `Connected to `), use cloud-specific tools. For example, for the LXD cloud: ```text lxc exec bash ``` or, for the MicroK8s cloud: ```text microk8s kubectl exec controller-0 -itc api-server -n [namespace] -- bash ``` > See more: [Access a machine via SSH](https://documentation.ubuntu.com/juju/2.9/howto/manage-machines.md#access-a-machine-via-ssh) 1. Examine the log files under `/var/log` with commands such as `cat`, `less`, or `tail -f`, for example: ```text cat /var/log/juju ``` #### IMPORTANT Which log to look at depends on the type of failure, but generally speaking, `syslog`, `cloud-init.log`, `cloud-init-output.log`, and `/var/log/juju` are good ones to look at. ### Example for a controller machine ```text # SSH into machine 0 of the controller model: juju ssh -m controller 0 # Navigate to the logs directory: cd /var/log/juju # List the contents: ls # View, e.g., the audit log file: cat audit.log ``` ### Control the log file rotation > See also: [List of controller configuration keys](https://documentation.ubuntu.com/juju/2.9/reference/configuration/list-of-controller-configuration-keys.md#list-of-controller-configuration-keys) Juju has settings to control the rotation of the various log files it produces. There are two settings for each log file type: maximum size and number of backups. When the current log file of a particular type reaches the maximum size, Juju renames the log file to include a timestamp and gzips it, producing a “backup” log file. Here’s an example of the controller’s machine agent logs with the maximum size set to 1MB, showing two timestamped backups as well as the current log file: ```text $ juju bootstrap localhost --config agent-logfile-max-size=1MB $ lxc exec juju-6bf629-0 -- ls -l /var/log/juju ... -rw-r----- 1 syslog adm 3577 Jan 12 02:01 machine-0-2022-01-12T02-01-07.995.log.gz -rw-r----- 1 syslog adm 3578 Jan 12 02:02 machine-0-2022-01-12T02-02-08.011.log.gz -rw-r----- 1 syslog adm 600000 Jan 12 02:02 machine-0.log ``` The full list of the controller settings that configure log file rotation is shown below. Normally these are set at bootstrap time with the `--config` option (see [Configure a controller](https://documentation.ubuntu.com/juju/2.9/howto/manage-controllers.md#configure-a-controller)). - The following config settings configure agent log files, including the API server “log sink”, the machine agent logs on controller and unit machines, and the unit agent logs: * `agent-logfile-max-backups` * `agent-logfile-max-size` - The following config settings configure the audit log files (note the missing “file” in the key name compared to the agent log file settings): * `audit-log-max-backups` * `audit-log-max-size` - The following config settings configure the model log files: * `model-logfile-max-backups` * `model-logfile-max-size`