--- title: Using an external etcd with MicroK8s description: Canonical makes open source secure, reliable and easy to use, providing support for Ubuntu and a portfolio of enterprise-grade technologies. Founded in 2004, Canonical operates globally with team members in over 80 countries. url: https://canonical.com/microk8s/docs/external-etcd?format=md --- *Submit* # Using an external etcd with MicroK8s This guide will take you through the process of using an external etcd as the underlying data store for MicroK8s. This makes sense depending on your use case, for example: * In production environments, you want your data store to run in designated nodes (for example, nodes with SSD disks). * You want to manage the data store yourself, instead of depending on MicroK8s for backup and restore functionality. It is important to note that **while migrating a running MicroK8s cluster from dqlite to an external etcd should be theoretically possible, it is highly discouraged. The instructions below are meant to be followed in a fresh MicroK8s installation**. Further, any existing resources and workloads that might be running in the cluster will be lost. ### [Deploy an external etcd cluster](https://canonical.com/microk8s/docs/external-etcd?format=md#p-31444-deploy-an-external-etcd-cluster) The first step is to deploy the etcd cluster you wish to use. If you are installing it manually or using an existing installation, the section below details the information you will need to gather from the cluster. Alternatively, you can use Juju to deploy etcd, also detailed below. #### Deploy manually Deploy your etcd cluster using your method of choice. There are no limitations other than making sure that your etcd members are reachable from MicroK8s. After deploying, you will need the following: * List of addresses of the etcd members, for example `https://10.10.0.1:2379,https://10.10.0.2:2379,https://10.10.0.3:2379`. * CA certificate for the etcd cluster, we will refer to this as `ca.crt`. * Client certificate for connecting to the etcd cluster, we will refer to this as `client.crt`. * Client key for connecting to the etcd cluster, we will refer to this as `client.key`. #### Deploy etcd in local LXD containers using Juju Use the following commands to [deploy a simple etcd cluster using Juju](https://ubuntu.com/kubernetes/docs/charm-etcd), running in LXD containers in a local machine: If not already installed, you should install and configure LXD: ``` sudo snap install lxd sudo lxd init --auto lxc network set lxdbr0 ipv6.address none ``` Similarly, install Juju via snap if it isn’t already installed: ``` sudo snap install juju --classic ``` Juju can then be used to deploy an etcd cluster ``` juju bootstrap lxd juju deploy containers-easyrsa juju deploy etcd -n 3 juju integrate easyrsa etcd ``` You can then check the status of the install using: ``` watch -d -c juju status etcd --color ``` After etcd is deployed successfully, create client credentials for use in MicroK8s: ``` juju run etcd/leader package-client-credentials --wait=2m juju scp etcd/leader:etcd_credentials.tar.gz ./ tar xvzf etcd_credentials.tar.gz ``` This leaves us with a directory named `etcd_credentials`, with the following contents: ``` etcd_credentials/ etcd_credentials/client.key etcd_credentials/client.crt etcd_credentials/ca.crt etcd_credentials/README.txt ``` Also, note the IPs of the etcd cluster members. These can be found with the `juju status etcd` command. For example, in the output below … ``` Unit Workload Agent Machine Public address Ports Message etcd/0* active idle 1 10.139.219.7 2379/tcp Healthy with 3 known peers etcd/1 active idle 2 10.139.219.148 2379/tcp Healthy with 3 known peers etcd/2 active idle 3 10.139.219.100 2379/tcp Healthy with 3 known peers ``` … the IP addresses of the etcd cluster would be `10.139.219.7`, `10.139.219.148`, and `10.139.219.100`. ### [Install MicroK8s](https://canonical.com/microk8s/docs/external-etcd?format=md#p-31444-install-microk8s) Install a new MicroK8s cluster with: ``` sudo snap install microk8s --classic ``` Do not add any extra nodes at this point. ### [Switch MicroK8s to external etcd](https://canonical.com/microk8s/docs/external-etcd?format=md#p-31444-switch-microk8s-to-external-etcd) 1. Copy the etcd certificates from the previous step to the machine where MicroK8s is running if required (e.g. via `scp`). Move the cert files into the MicroK8s certificates directory. **Make sure to prefix them with `etcd-` to avoid overwriting existing files**: ``` mv etcd_credentials/client.key /var/snap/microk8s/current/certs/etcd-client.key mv etcd_credentials/client.crt /var/snap/microk8s/current/certs/etcd-client.crt mv etcd_credentials/ca.crt /var/snap/microk8s/current/certs/etcd-ca.crt ``` 2. Edit `/var/snap/microk8s/current/args/kube-apiserver` to **remove** any existing `--etcd-servers` arguments (for MicroK8s 1.23+), or `--storage-backend=dqlite` (MicroK8s 1.22 and earlier). 3. Append the following lines in `/var/snap/microk8s/current/args/kube-apiserver`. Make sure to replace the IP addresses with the IP addresses of your etcd cluster members. ``` --etcd-servers=https://10.139.219.7:2379,https://10.139.219.148:2379,https://10.139.219.100:2379 --etcd-cafile=${SNAP_DATA}/certs/etcd-ca.crt --etcd-certfile=${SNAP_DATA}/certs/etcd-client.crt --etcd-keyfile=${SNAP_DATA}/certs/etcd-client.key ``` 4. Restart MicroK8s ``` sudo snap restart microk8s ``` 5. Re-apply the Calico CNI: ``` microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml ``` ### [That’s all!](https://canonical.com/microk8s/docs/external-etcd?format=md#p-31444-thats-all) Congratulations! Your MicroK8s installation is now running using your external etcd as a data store! ### [Notes for MicroK8s cluster](https://canonical.com/microk8s/docs/external-etcd?format=md#p-31444-notes-for-microk8s-cluster) #### Adding worker only nodes (MicroK8s 1.23+) When using an external data store, adding new worker-only nodes works as is: 1. In the existing MicroK8s node, run the `add-node` command: ``` microk8s add-node ``` 2. In the new node, join as worker: ``` microk8s join 10.0.1.1:25000/XXXXXXXXXXXXXXXXXXXXXXXXXXX/YYYYYYYYYY --worker ``` #### Adding more control plane nodes When adding new control plane nodes to the cluster, an extra manual step is required: 1. In an existing MicroK8s node, run the `add-node` command: ``` microk8s add-node ``` 2. In the new node, join the cluster: ``` microk8s join 10.0.1.1:25000/XXXXXXXXXXXXXXXXXXXXXXXXXXX/YYYYYYYYYY ``` 3. Copy the etcd certificates to the new MicroK8s node (e.g. via `scp`). Then, move them into the MicroK8s certificates directory. Make sure to prefix them with `etcd-` to avoid overwriting existing files: ``` mv etcd_credentials/client.key /var/snap/microk8s/current/certs/etcd-client.key mv etcd_credentials/client.crt /var/snap/microk8s/current/certs/etcd-client.crt mv etcd_credentials/ca.crt /var/snap/microk8s/current/certs/etcd-ca.crt ``` 4. Edit `/var/snap/microk8s/current/args/kube-apiserver` and **remove** any existing `--etcd-servers` arguments (for MicroK8s 1.23+), or `--storage-backend=dqlite` (MicroK8s 1.22 and earlier). 5. Append the following lines in `/var/snap/microk8s/current/args/kube-apiserver`. Make sure to replace the IP addresses with the IP addresses of your etcd cluster members. ``` --etcd-servers=https://10.139.219.7:2379,https://10.139.219.148:2379,https://10.139.219.100:2379 --etcd-cafile=${SNAP_DATA}/certs/etcd-ca.crt --etcd-certfile=${SNAP_DATA}/certs/etcd-client.crt --etcd-keyfile=${SNAP_DATA}/certs/etcd-client.key ``` 6. Restart MicroK8s ``` sudo snap restart microk8s ``` Last updated 3 years ago. [Help improve this document in the forum](https://discuss.kubernetes.io/t/using-an-external-etcd-with-microk8s/18351).