How to track changed blocks on virtual machine volumes

Changed block tracking records the blocks of a block volume that a guest writes to after a snapshot. This enables a backup tool to copy only those blocks instead of the whole volume.

LXD implements changed block tracking with QEMU dirty bitmaps. LXD creates a bitmap together with an instance snapshot and names the bitmap after the snapshot. When you take a later snapshot with a bitmap, LXD copies every bitmap of the volume into that snapshot. LXD then serves the snapshot and its bitmap copies to a Network Block Device (NBD) client through the LXD API.

LXD keeps a bitmap across a stop, a reboot and a forced stop of the virtual machine. LXD deletes a bitmap in the following cases:

• You delete or rename its snapshot.

• The QEMU process exits before LXD stores the bitmap, for example on a crash or a host power loss.

• Something other than the running virtual machine writes the volume, for example a restore or a read-write NBD export.

After LXD deletes a bitmap, the next backup must be a full one.

When you take the first snapshot with a bitmap of a block disk, LXD creates a small qcow2 image on the config volume of the virtual machine. This image stores the bitmaps of the volume. The guest reads and writes the volume directly. LXD merges the bitmaps into the image before the QEMU process ends. When the guest shuts down or reboots, the QEMU process pauses until LXD has merged the bitmaps.

Requirements

Changed block tracking has the following requirements on the LXD server and client:

• Enable the changed_block_tracking feature preview on both the server and the client (see Configure feature previews). While the preview is disabled, the server does not register the endpoints and does not advertise the storage_volume_block_tracking API extension (see storage_volume_block_tracking). The server also rejects a snapshot with a bitmap, and the client hides the lxc bitmap, lxc nbd and lxc storage volume nbd commands.

• Install an NBD client on the machine that runs the LXD client. For example, use nbdinfo and nbdcopy from libnbd, or qemu-img.

LXD creates a bitmap on the root volume of the virtual machine. If you take the snapshot with --disk-volumes all-exclusive, LXD also creates a bitmap on every attached custom volume of content type block. LXD does not create a bitmap on a volume with security.shared enabled. Several virtual machines can write to such a volume at once, but a bitmap records the writes of one virtual machine only.

Each action has its own requirements (see Permissions for the entitlements):

• To take a snapshot with a bitmap, start the virtual machine. You need the can_manage_snapshots entitlement on the instance.

• To list the bitmaps of a snapshot, you need the can_view entitlement on the instance.

• To read a snapshot over NBD, you need the can_connect_nbd entitlement on the instance.

• To write a volume over NBD, stop the virtual machine that uses the volume. You need the can_connect_nbd entitlement on the storage volume.

Take a snapshot with a bitmap

LXD creates a bitmap together with an instance snapshot and names the bitmap after the snapshot. The bitmap starts recording guest writes to the volume when you take the snapshot. It continues recording until LXD deletes it.

LXD also copies every existing bitmap of the volume into the new snapshot. Each copy records exactly the blocks that the guest wrote between the creation of its bitmap and the new snapshot.

CLIAPI

Use the following command to snapshot a virtual machine and create a bitmap on its root volume:

lxc snapshot <instance_name> <snapshot_name> --bitmap

To also snapshot the attached block volumes and create a bitmap on each of them, add --disk-volumes all-exclusive.

LXD rejects the request if the instance is not a running virtual machine, or if the snapshot is stateful.

Every bitmap has a name and a UUID. The UUID of a bitmap is the UUID of its snapshot, which is the volatile.uuid of the root volume snapshot. If you delete or rename a snapshot and then take a new snapshot with the same name, the new snapshot has a different UUID.

A backup tool stores the UUID together with the name. It passes the UUID to the export of the next snapshot. This ensures that the backup tool reads only the changes recorded by the bitmap it stored.

List the bitmaps of a snapshot

CLIAPI

Use the following command to list the bitmap copies that an instance snapshot keeps:

lxc bitmap list <instance_name>/<snapshot_name>

The command prints one row per bitmap and volume. Each row shows the name and the UUID of the bitmap, the disk device, the pool, the type and the name of the volume, the granularity in bytes and whether the bitmap is recording writes. The granularity is the size of the block that one bit covers.

Use the following command to show one bitmap with its name, its UUID and the volumes it exists on:

lxc bitmap show <instance_name>/<snapshot_name> <bitmap_name>

The bitmap copies in a snapshot do not record writes, and LXD never modifies a snapshot. Deleting a snapshot removes the bitmap with its name from every volume of the virtual machine. Renaming a snapshot removes the bitmaps with its old name and its new name. The other snapshots keep their bitmap copies.

Read a snapshot

LXD serves an instance snapshot with a bitmap read-only over NBD. The export publishes the bitmap copies of the snapshot as qemu:dirty-bitmap:<bitmap_name> metadata contexts, next to base:allocation.

You can limit the export to the bitmap copy of one snapshot by giving the UUID of that snapshot. If no bitmap copy has that UUID, the export publishes no bitmap copy. This happens, for example, when you delete a snapshot and take a new one with the same name before you take the snapshot that you export. In that case, the backup tool takes a full backup.

The virtual machine keeps running during the export.

CLIAPI

Use the following command to serve an instance snapshot to a local NBD client:

lxc nbd <instance_name>/<snapshot_name>

The command opens a local listener and prints the listening address, for example:

NBD listening on 127.0.0.1:41337

The command waits for one NBD client to connect and forwards that connection to LXD. It exits when the client disconnects. Use the --address flag to specify the listening address. Without it, the command listens on a random port on the loopback interface.

Each run of the command serves one client, and each client opens its own session. You can therefore run the command several times to serve the same snapshot to several clients.

LXD serves each volume snapshot as a separate NBD export, named after its disk device. To select a volume, add the device name to the NBD URL. In a second terminal, point an NBD client at the printed address.

To list the blocks that a bitmap recorded up to the snapshot, use the following command:

nbdinfo --map=qemu:dirty-bitmap:<bitmap_name> nbd://127.0.0.1:41337/root

To copy the whole root volume snapshot to a file, use the following command:

nbdcopy --connections=1 nbd://127.0.0.1:41337/root <file_path>

To serve only some of the volumes, add the --devices flag with a comma separated list of disk device names. To serve only the bitmap copy of the previous snapshot, add the --previous-snapshot-uuid flag with the UUID of that snapshot.

To take incremental backups, take every snapshot with a bitmap. Copy the first snapshot in full, because it has no copy of an earlier bitmap. For every later snapshot, follow these steps:

1. Open the export of the new snapshot with the UUID of the previous snapshot.

2. Read the map of the bitmap named after the previous snapshot.

3. Copy the blocks that the map marks.

4. After you store the backup, delete the previous snapshot.

Deleting the previous snapshot removes its bitmap from the virtual machine. The new snapshot keeps its own copy of that bitmap.

Write a volume while the virtual machine is stopped

To restore a backup, write it into the volume through a read-write NBD export. Before you open the export, stop the virtual machine that uses the volume. To restore into a new virtual machine, first create it without an image:

lxc init <instance_name> --empty --vm

CLIAPI

Use the following command to serve a volume read-write to a local NBD client:

lxc storage volume nbd <pool_name> [<volume_type>/]<volume_name> --writable

The default volume type is custom. The command requires the --writable flag to confirm that you want to overwrite the volume.

The command prints the listening address, for example NBD listening on 127.0.0.1:41337. It serves the volume under the default export. In a second terminal, write the backup into the export with either of the following commands:

qemu-img convert -n -f raw -O raw <file_path> nbd://127.0.0.1:41337
nbdcopy <file_path> nbd://127.0.0.1:41337

After the client disconnects, start the virtual machine.

The bitmaps of the volume do not record the writes of the export. Therefore, LXD deletes them before the export starts. The snapshots keep their bitmap copies.

List and cancel NBD sessions

LXD represents every NBD session with an operation on the cluster member that serves the session. This applies to both snapshot exports and volume exports. The operation runs while the client stays connected. Cancelling the operation closes the connection.

Use the following command to list the open sessions:

lxc operation list

Use the following command to end a session:

lxc operation delete <operation_id>

The Location header of the 101 Switching Protocols response contains the URL of the operation.

Limitations

The following events delete the bitmaps of a volume. The copies kept by the snapshots are not affected, but the next snapshot records no bitmaps and the backup following it must be a full one.

• The QEMU process crashes, or the host loses power.

• The virtual machine is live migrated or moved to another cluster member.

• The volume is restored from a snapshot, copied, refreshed or imported.

• The volume is written through a read-write NBD export.

• The volume is resized.

• A custom volume is detached from the virtual machine.

• The security.shared option is enabled on the volume.

Restoring an instance snapshot deletes the bitmaps of every volume of the virtual machine.

Bitmaps are named after the instance snapshots they were created with:

• Deleting a custom volume snapshot removes the bitmap created with it from that volume. The instance snapshot that recorded it then has no snapshot of that volume, and its export skips the volume.

• Deleting an instance snapshot removes the bitmap with its name from every volume.

• Renaming an instance snapshot removes the bitmaps with its old name and its new name.

• Renaming a custom volume snapshot does not affect the bitmaps.

While a snapshot with a bitmap is in progress:

• The virtual machine cannot be stopped or restarted.

• Its volumes cannot be resized or have security.shared enabled.

• A disk detach, and a power off or reboot from the guest, take effect when the snapshot ends.

While an NBD export is open:

• The virtual machine whose volume is exported cannot be started.

• The instance snapshot cannot be deleted or renamed.

• The custom volume snapshots that the exported instance snapshot recorded cannot be deleted or renamed.

After the first snapshot with a bitmap, if the guest powers off while LXD is not running, the virtual machine is reported as STOPPING. It stays STOPPING until LXD starts and stops it.