# How to perform disaster recovery with replicators Once you have [set up replicators](https://canonical.com/lxd/docs/latest/howto/replicators_create/index.html.md#howto-replicators-setup) for active-passive replication, you can use them to fail over to the standby cluster if the leader cluster becomes unavailable, and to restore the original replication direction when the leader comes back online. ## Failover process If the leader cluster becomes unavailable, you can manually fail over to the standby cluster. On the standby cluster, promote the replica project to become the leader: CLI ```bash lxc project promote-replica ``` If the leader cluster is unreachable, promotion proceeds automatically without requiring validation. Use `--force` to skip validation when the leader cluster is still reachable but you want to promote anyway (for example, during a planned takeover before demoting the leader): ```bash lxc project promote-replica --force ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Promote to leader. If the leader cluster is unreachable, promotion proceeds automatically without requiring validation. Click Promote in the confirmation modal. If the leader cluster is still reachable but you want to promote the replica project anyway (for example, during a planned takeover before demoting the leader), then check Force and click Promote to skip validation. After promoting the project on the standby cluster, the project becomes writable. Start the instances to resume your workloads: CLI ```bash lxc start --all --project ``` UI Select Instances in the navigation sidebar. Click the checkbox in the header row to select all instances, then click Start in the page header. In the confirmation modal, click Start. ## Recovering the original leader cluster When the original leader cluster comes back online, it will be out of sync with the new leader (the former standby). Scheduled replicator runs on the original leader cluster will fail because both projects are in leader mode. A replicator run requires the source project to be in leader mode and the target project to be in standby mode. To restore the original leader cluster and resume the original replication direction: ### 1. Sync from the new leader back to the original leader On the original leader cluster, stop all running instances in the project before running restore. The “restore” action is rejected if any local instance is running, to prevent partial restores. CLI ```bash lxc stop [...] --force ``` UI Select Instances in the navigation sidebar. Click the checkbox in the header row to select all instances, then click Stop in the page header. In the confirmation modal, check Force stop then click Stop. Demote the project on the original leader cluster to standby mode: CLI ```bash lxc project demote-replica ``` If the new leader cluster is unreachable, use `--force` to skip the validation: ```bash lxc project demote-replica --force ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Demote to standby. If the new leader is reachable, click Demote. If the new leader is unreachable, check Force to skip the validation, then click Demote. On the original leader cluster, run the replicator in restore mode to pull data from the new leader: CLI ```bash lxc replicator run --restore ``` UI Click Clustering in the navigation sidebar, then select Replicators from the expanded drop-down list. Click on the run button at the end of the replicator’s row. Alternatively, click on a replicator name to view its detail page, then click on the Restore button in the header. In the confirmation modal, check Overwrite local data, then click Restore. Restore mode uses the new leader’s instance list as the authoritative source. Any instances created on the new leader during the failover period are also created on the recovering cluster automatically. The original leader cluster is now a standby replica of the new leader cluster. ### 2. Resume original replication direction To return to the original setup where the original leader cluster replicates to the standby, stop any running instances in the project on the new leader cluster (former standby). Next, demote the project on the new leader cluster back to standby mode: CLI ```bash lxc project demote-replica ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Demote to standby. Finally, promote the project on the original leader cluster back to leader mode: CLI ```bash lxc project promote-replica ``` UI Select the project from the Project drop-down menu, then click Configuration in the navigation sidebar. Select the Replication tab, then, under Replica mode, click Promote to leader. Your original active-passive disaster recovery setup is now restored. You can restart your instances on the leader cluster and resume your scheduled replicator runs. ## Related topics How-to guides: * [How to set up replicators](https://canonical.com/lxd/docs/latest/howto/replicators_create/index.html.md#howto-replicators-setup) * [How to manage replicators](https://canonical.com/lxd/docs/latest/howto/replicators_manage/index.html.md#howto-replicators-manage) * [How to perform disaster recovery with storage replication](https://canonical.com/lxd/docs/latest/howto/disaster_recovery_replication/index.html.md#disaster-recovery-replication)