# How to perform disaster recovery with replicators Active-passive disaster recovery with replicators requires advance preparation: you must [set up replicators](https://canonical.com/lxd/docs/latest/howto/replicators_create/index.html.md#howto-replicators-setup) on the primary cluster to regularly copy instances to a secondary cluster. If the primary cluster becomes unavailable, you can promote your secondary cluster to take over workloads. Then, after the primary cluster comes back online, you can synchronize the clusters and resume the original replication setup. #### IMPORTANT Changing replica modes does not redirect application traffic. You must manage traffic separately from LXD. ## Fail over to the secondary cluster If the primary cluster becomes unavailable, you can manually fail over to the secondary cluster. On the secondary cluster, promote the replica project to `leader` mode: CLI ```bash lxc project promote-replica ``` If the primary cluster is unreachable, promotion proceeds automatically without requiring validation. Use `--force` only to promote a standby project when the primary cluster is still reachable and its project is still in `leader` mode: ```bash lxc project promote-replica --force ``` For a planned switchover rather than a failover, do not use `--force`; instead, demote the leader first, then promote the standby, as described in [Replicators](https://canonical.com/lxd/docs/latest/explanation/disaster_recovery/index.html.md#exp-replicators). UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Promote to leader. If the primary cluster is unreachable, promotion proceeds automatically without requiring validation. Click Promote in the confirmation modal. If the primary cluster is still reachable and its project has not been demoted to `standby`, checking Force before clicking Promote skips validation. For a planned switchover, demote the leader first and then promote without Force. #### CAUTION Forced promotion of a standby project can create a split-brain risk: if the target of replication is still in `leader` mode, then both projects are writable. Any instances created, started, or modified independently on each side during that window will diverge and cannot be automatically reconciled by the next replicator run. Only force-promote in this situation when you understand and accept that risk; for a planned switchover, demote the leader first instead. After promotion, the project on the secondary cluster becomes writable. Start the instances to resume your workloads: CLI ```bash lxc start --all --project ``` UI Select Instances in the navigation sidebar. Click the checkbox in the header row to select all instances, then click Start in the page header. In the confirmation modal, click Start. You can then verify that the instances start successfully. CLI Run this command to confirm that all instances have state `RUNNING`: ```bash lxc list --project ``` You can also run the following command to check that custom volumes have been mounted: ```bash lxc exec -- df -h ``` UI On the Instances page, verify that the status for each instance changes from Stopped to Running. Instances may fail to boot for application-related reasons, or may require a strict boot order, which LXD does not orchestrate automatically. If a virtual machine fails to boot, you can attach its root volume to another virtual machine to investigate its contents. Refer to the [troubleshooting guides](https://canonical.com/lxd/docs/latest/howto/troubleshoot/index.html.md#troubleshoot) if you encounter any issues. #### IMPORTANT Verifying that an instance is `RUNNING` does not confirm application readiness. You must manage application validation separately from LXD. ## Fail back to the primary cluster After failover, the secondary cluster manages workloads with a project in `leader` mode. As a result, when the primary cluster comes back online, the replica projects will be out of sync. Scheduled replicator runs on the primary cluster will fail because both projects are in `leader` mode. To fail back to the primary cluster, you must synchronize the projects, restore the original replica modes, and resume the original replication direction. ### 1. Verify cluster restoration Once the primary cluster comes back online, you can confirm the health of the cluster. CLI Run this command to ensure that all cluster members are back online: ```bash lxc cluster list ``` UI Click Clustering in the navigation sidebar, select Members from the expanded drop-down list, and confirm that all members have status Online. ### 2. Synchronize projects You can run the replicator on the primary cluster in “restore” mode to synchronize the project on the primary cluster with the project on the secondary cluster. The replicator only runs in “restore” mode if all instances in the original source project are stopped, to prevent partial restoration of instances. CLI Run this command to stop running instances: ```bash lxc stop [...] --force --project ``` UI Select Instances in the navigation sidebar. Click the checkbox in the header row to select all instances, then click Stop in the page header. In the confirmation modal, check Force stop then click Stop. Demote the project on the primary cluster to `standby` mode: CLI ```bash lxc project demote-replica ``` Though demotion does not require contact with the other project, a project can only be demoted to `standby` if [`replica.cluster`](https://canonical.com/lxd/docs/latest/reference/projects/index.html.md#project-replica:replica.cluster) is set, to specify the source of replication data. If this key is unset, use `--force` to demote the project: ```bash lxc project demote-replica --force ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Demote to standby. A project can only be demoted if [`replica.cluster`](https://canonical.com/lxd/docs/latest/reference/projects/index.html.md#project-replica:replica.cluster) is set, to specify the source of replication data. If this key is unset, check Force before clicking Demote. On the primary cluster, run the replicator in “restore” mode to pull data from the secondary cluster: CLI ```bash lxc replicator run --restore ``` UI Click Clustering in the navigation sidebar, then select Replicators from the expanded drop-down list. Click on the run button at the end of the replicator’s row. Alternatively, click on a replicator name to view its detail page, then click on the Restore button in the header. In the confirmation modal, check Overwrite local data, then click Restore. Restore mode uses the instance list from the secondary cluster as the authoritative source. Any instances created on the secondary cluster during the failover period are also created on the primary cluster automatically. The project on the primary cluster is now a standby replica of the project on the secondary cluster (the replicator remains on the primary cluster). ### 3. Resume original replication direction To restore the original setup, in which the primary cluster replicates to the secondary cluster, stop any running instances in the project on the secondary cluster. Next, demote the project on the secondary cluster back to `standby` mode: CLI ```bash lxc project demote-replica ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Demote to standby. Finally, promote the project on the primary cluster back to `leader` mode. When a project is promoted from `standby` to `leader`, LXD uses the [`replica.cluster`](https://canonical.com/lxd/docs/latest/reference/projects/index.html.md#project-replica:replica.cluster) key to identify the corresponding replica project and confirm that it is no longer in `leader` mode. If this key is unset, set it to the name of the cluster link used for replication; then, promote the project: CLI ```bash lxc project promote-replica ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Promote to leader. Your original active-passive disaster recovery setup is now restored. You can restart your instances on the primary cluster and resume your scheduled replicator runs. ## Related topics How-to guides: * [How to set up replicators](https://canonical.com/lxd/docs/latest/howto/replicators_create/index.html.md#howto-replicators-setup) * [How to manage replicators](https://canonical.com/lxd/docs/latest/howto/replicators_manage/index.html.md#howto-replicators-manage) * [How to perform disaster recovery with storage replication](https://canonical.com/lxd/docs/latest/howto/disaster_recovery_replication/index.html.md#disaster-recovery-replication)