combines both physical volume management and a file system.
A ZFS installation can span across a series of storage devices and is very scalable, allowing you to add disks to expand the available space in the storage pool immediately.
ZFS is a block-based file system that protects against data corruption by using checksums to verify, confirm and correct every operation.
To run at a sufficient speed, this mechanism requires a powerful environment with a lot of RAM.
In addition, ZFS offers snapshots and replication, RAID management, copy-on-write clones, compression and other features.
To use ZFS, make sure you have `zfsutils-linux` installed on your machine.
## Terminology
ZFS creates logical units based on physical storage devices.
These logical units are called *ZFS pools* or *zpools*.
Each zpool is then divided into a number of *datasets*.
These datasets can be of different types:
- A *ZFS filesystem* can be seen as a partition or a mounted file system.
- A *ZFS volume* represents a block device.
- A *ZFS snapshot* captures a specific state of either a ZFS filesystem or a ZFS volume.
ZFS snapshots are read-only.
- A *ZFS clone* is a writable copy of a ZFS snapshot.
## `zfs` driver in LXD
The `zfs` driver in LXD uses ZFS filesystems and ZFS volumes for images and custom storage volumes, and ZFS snapshots and clones to create instances from images and for instance and custom volume snapshots.
By default, LXD enables compression when creating a ZFS pool.
LXD assumes that it has full control over the ZFS pool and dataset.
Therefore, you should never maintain any datasets or file system entities that are not owned by LXD in a ZFS pool or dataset, because LXD might delete them.
Due to the way copy-on-write works in ZFS, parent ZFS filesystems can’t be removed until all children are gone.
As a result, LXD automatically renames any objects that are removed but still referenced.
Such objects are kept at a random `deleted/` path until all references are gone and the object can safely be removed.
Note that this method might have ramifications for restoring snapshots.
See [Limitations](#storage-zfs-limitations) below.
LXD automatically enables trimming support on all newly created pools on ZFS 0.8 or later.
This increases the lifetime of SSDs by allowing better block re-use by the controller, and it also allows to free space on the root file system when using a loop-backed ZFS pool.
If you are running a ZFS version earlier than 0.8 and want to enable trimming, upgrade to at least version 0.8.
Then use the following commands to make sure that trimming is automatically enabled for the ZFS pool in the future and trim all currently unused space:
```none
zpool upgrade ZPOOL-NAME
zpool set autotrim=on ZPOOL-NAME
zpool trim ZPOOL-NAME
```
### Image variants and optimized instance creation
The ZFS driver supports optimized image volumes that significantly reduce instance creation time.
If an image is unpacked into an optimized image volume, when LXD creates an instance from that image, it can clone the instance from that volume rather than unpacking from scratch.
This speeds up instance creation.
The ZFS driver can maintain multiple variants of the same optimized image volume to support fast instance creation when using instance-specific initial root disk settings (see [Initial volume configuration for instance root disk devices](https://canonical.com/lxd/docs/latest/reference/devices_disk/index.html.md#devices-disk-initial-config)).
#### Variant types
LXD can maintain multiple optimized image volumes for the same image, representing different volume configurations:
- **Dataset variant** (dataset mode): A `ZFS filesystem` dataset created when [`zfs.block_mode`](#storage-zfs-volume-conf:zfs.block_mode) is `false`.
- **Block-backed variants**: ZFS volumes with specific filesystems (ext4, btrfs, xfs) created when [`zfs.block_mode`](#storage-zfs-volume-conf:zfs.block_mode) is `true`.
#### Variant lifecycle
Variants are created lazily on demand when an instance requests a configuration that doesn’t match an existing variant.
A variant is deleted when one of the following events occurs:
- The last instance that uses the variant is removed.
- The variant is not used by any instance and the image is deleted.
- The variant is not used by any instance and the pool configuration is changed such that the variant no longer matches the new pool defaults (for example, a change from dataset mode to block mode).
Variants with active instances are always preserved regardless of image deletion or configuration changes.
#### Configuration
Changing pool configuration frequently can reduce optimization benefits, as variants may need to be recreated.
To optimize instance creation, set up default pool configurations and create instances that use those defaults (you can still override those settings for specific instances as needed):
- **Pool default**: Set [`zfs.block_mode`](#storage-zfs-volume-conf:zfs.block_mode) and [`block.filesystem`](#storage-zfs-volume-conf:block.filesystem) at the pool level during pool creation by using the `volume.` prefix.
- **Instance override**: Use `initial.zfs.block_mode` and `initial.block.filesystem` [device configuration](https://canonical.com/lxd/docs/latest/reference/devices_disk/index.html.md#devices-disk-initial-config) for instances that require different settings.
#### Performance benefits
Using image variants provides:
- Faster instance creation (the time to clone a volume with ZFS is unrelated to the volume size).
- Minimal disk I/O during instance creation (metadata operations only).
- Copy-on-write space efficiency (instances initially share data with the image variant).
### Limitations
The `zfs` driver has the following limitations:
Restoring from older snapshots
: ZFS doesn’t support restoring from snapshots other than the latest one.
You can, however, create new instances from older snapshots.
This method makes it possible to confirm whether a specific snapshot contains what you need.
After determining the correct snapshot, you can [remove the newer snapshots](https://canonical.com/lxd/docs/latest/howto/storage_backup_volume/index.html.md#storage-edit-snapshots) so that the snapshot you need is the latest one and you can restore it.
Alternatively, you can configure LXD to automatically discard the newer snapshots during restore.
To do so, set the [`zfs.remove_snapshots`](#storage-zfs-volume-conf:zfs.remove_snapshots) configuration for the volume (or the corresponding `volume.zfs.remove_snapshots` configuration on the storage pool for all volumes in the pool).
Note, however, that if [`zfs.clone_copy`](#storage-zfs-pool-conf:zfs.clone_copy) is set to `true`, instance copies use ZFS snapshots too.
In that case, you cannot restore an instance to a snapshot taken before the last copy without having to also delete all its descendants.
If this is not an option, you can copy the wanted snapshot into a new instance and then delete the old instance.
You will, however, lose any other snapshots the instance might have had.
Observing I/O quotas
: I/O quotas are unlikely to affect ZFS filesystems very much.
That’s because ZFS is a port of a Solaris module (using SPL) and not a native Linux file system using the Linux VFS API, which is where I/O limits are applied.
Feature support in ZFS
: Some features, like the use of idmaps or delegation of a ZFS dataset, require ZFS 2.2 or higher and are therefore not widely available yet.
### Quotas
ZFS provides two different quota properties: `quota` and `refquota`.
`quota` restricts the total size of a dataset, including its snapshots and clones.
`refquota` restricts only the size of the data in the dataset, not its snapshots and clones.
By default, LXD uses the `quota` property when you set up a size/quota for your storage volume.
If you want to use the `refquota` property instead, set the [`zfs.use_refquota`](#storage-zfs-volume-conf:zfs.use_refquota) configuration for the volume (or the corresponding `volume.zfs.use_refquota` configuration on the storage pool for all volumes in the pool).
You can also set the [`zfs.reserve_space`](#storage-zfs-volume-conf:zfs.reserve_space) (or `volume.zfs.reserve_space`) configuration to use ZFS `reservation` or `refreservation` along with `quota` or `refquota`.
## Configuration options
The following configuration options are available for storage pools that use the `zfs` driver and for storage volumes in these pools.
### Storage pool configuration
`rsync.bwlimit`
Upper limit on the socket I/O for `rsync`
[](#storage-zfs-pool-conf:rsync.bwlimit)
| **Key:** | `rsync.bwlimit` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | `0` (no limit) |
| **Scope:** | global |
When `rsync` must be used to transfer storage entities, this option specifies the upper limit
to be placed on the socket I/O.
`rsync.compression`
Whether to use compression while migrating storage pools
[](#storage-zfs-pool-conf:rsync.compression)
| **Key:** | `rsync.compression` |
|--------------|-----------------------|
| **Type:** | bool |
| **Default:** | `true` |
| **Scope:** | global |
`size`
Size of the storage pool (for loop-based pools)
[](#storage-zfs-pool-conf:size)
| **Key:** | `size` |
|--------------|-------------------------------------------------------|
| **Type:** | string |
| **Default:** | auto (20% of free disk space, >= 5 GiB and <= 30 GiB) |
| **Scope:** | local |
When creating loop-based pools, specify the size in bytes ([suffixes](https://canonical.com/lxd/docs/latest/reference/instance_units/index.html.md#instances-limit-units) are supported).
You can increase the size to grow the storage pool.
The default (`auto`) creates a storage pool that uses 20% of the free disk space,
with a minimum of 5 GiB and a maximum of 30 GiB.
`source`
Path to an existing block device, loop file, or ZFS dataset/pool
[](#storage-zfs-pool-conf:source)
| **Key:** | `source` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | local |
`source.recover`
Whether to recover an existing `source`
[](#storage-zfs-pool-conf:source.recover)
| **Key:** | `source.recover` |
|--------------|--------------------|
| **Type:** | bool |
| **Default:** | `false` |
| **Scope:** | local |
Set this option to true to recover an existing source which was previously created by LXD.
`source.wipe`
Whether to wipe the block device before creating the pool
[](#storage-zfs-pool-conf:source.wipe)
| **Key:** | `source.wipe` |
|--------------|-----------------|
| **Type:** | bool |
| **Default:** | `false` |
| **Scope:** | local |
Set this option to `true` to wipe the block device specified in `source`
prior to creating the storage pool.
`user.*`
User-provided free-form key/value pairs
[](#storage-zfs-pool-conf:user.*)
| **Key:** | `user.*` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | global |
`zfs.clone_copy`
Whether to use ZFS lightweight clones
[](#storage-zfs-pool-conf:zfs.clone_copy)
| **Key:** | `zfs.clone_copy` |
|--------------|--------------------|
| **Type:** | string |
| **Default:** | `true` |
| **Scope:** | global |
Set this option to `true` or `false` to enable or disable using ZFS lightweight clones rather
than full dataset copies.
Set the option to `rebase` to copy based on the initial image.
`zfs.export`
Whether to export the zpool when an unmount is being performed
[](#storage-zfs-pool-conf:zfs.export)
| **Key:** | `zfs.export` |
|--------------|----------------|
| **Type:** | bool |
| **Default:** | `true` |
| **Scope:** | global |
`zfs.pool_name`
Name of the zpool
[](#storage-zfs-pool-conf:zfs.pool_name)
| **Key:** | `zfs.pool_name` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | name of the pool |
| **Scope:** | local |
#### TIP
In addition to these configurations, you can also set default values for the storage volume configurations. See storage-configure-vol-default.
### Storage volume configuration
`block.filesystem`
File system of the storage volume
[](#storage-zfs-volume-conf:block.filesystem)
| **Key:** | `block.filesystem` |
|----------------|------------------------------------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.block.filesystem` |
| **Condition:** | block-based volume with content type `filesystem` (`zfs.block_mode` enabled) |
| **Scope:** | global |
Valid options: `btrfs`, `ext4`, `xfs`
If not set, `ext4` is assumed.
`block.mount_options`
Mount options for block-backed file system volumes
[](#storage-zfs-volume-conf:block.mount_options)
| **Key:** | `block.mount_options` |
|----------------|------------------------------------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.block.mount_options` |
| **Condition:** | block-based volume with content type `filesystem` (`zfs.block_mode` enabled) |
| **Scope:** | global |
`security.shared`
Enable volume sharing
[](#storage-zfs-volume-conf:security.shared)
| **Key:** | `security.shared` |
|----------------|---------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.shared` or `false` |
| **Condition:** | virtual-machine or custom block volume |
| **Scope:** | global |
Enable this option to allow the volume to be shared across multiple instances despite the possibility of data loss.
`security.shifted`
Enable ID shifting overlay
[](#storage-zfs-volume-conf:security.shifted)
| **Key:** | `security.shifted` |
|----------------|----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.shifted` or `false` |
| **Condition:** | custom volume |
| **Scope:** | global |
Enable this option to allow the volume to be attached to multiple isolated instances.
`security.unmapped`
Disable ID mapping for the volume
[](#storage-zfs-volume-conf:security.unmapped)
| **Key:** | `security.unmapped` |
|----------------|-----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.unmapped` or `false` |
| **Condition:** | custom volume |
| **Scope:** | global |
`size`
Size/quota of the storage volume
[](#storage-zfs-volume-conf:size)
| **Key:** | `size` |
|----------------|-----------------------|
| **Type:** | string |
| **Default:** | same as `volume.size` |
| **Condition:** | appropriate driver |
| **Scope:** | global |
`snapshots.expiry`
Time until snapshots are deleted
[](#storage-zfs-volume-conf:snapshots.expiry)
| **Key:** | `snapshots.expiry` |
|----------------|-----------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.snapshots.expiry` |
| **Condition:** | custom volume |
| **Scope:** | global |
Specify an expression like `1M 2H 3d 4w 5m 6y`.
`snapshots.pattern`
Template for the snapshot name
[](#storage-zfs-volume-conf:snapshots.pattern)
| **Key:** | `snapshots.pattern` |
|----------------|------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.snapshots.pattern` or `snap%d` |
| **Condition:** | custom volume |
| **Scope:** | global |
You can specify a naming template for scheduled snapshots and unnamed snapshots.
The `snapshots.pattern` option takes a Pongo2 template string to format the snapshot name.
To add a time stamp to the snapshot name, use the Pongo2 context variable `creation_date`.
Make sure to format the date in your template string to avoid forbidden characters in the snapshot name.
For example, set `snapshots.pattern` to `{{ creation_date|date:'2006-01-02_15-04-05' }}` to name the snapshots after their time of creation, down to the precision of a second.
Another way to avoid name collisions is to use the placeholder `%d` in the pattern.
If no matching snapshots exist, the placeholder is replaced with `0`.
Otherwise, it is replaced with the next snapshot index, which is one higher than the highest existing matching snapshot index.
`snapshots.schedule`
Schedule for automatic volume snapshots
[](#storage-zfs-volume-conf:snapshots.schedule)
| **Key:** | `snapshots.schedule` |
|----------------|------------------------------|
| **Type:** | string |
| **Default:** | same as `snapshots.schedule` |
| **Condition:** | custom volume |
| **Scope:** | global |
Specify either a cron expression (``), a comma-separated list of schedule aliases (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@annually`, `@yearly`), or leave empty to disable automatic snapshots (the default).
`user.*`
User-provided free-form key/value pairs
[](#storage-zfs-volume-conf:user.*)
| **Key:** | `user.*` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | global |
`volatile.devlxd.owner`
ID of the DevLXD identity that owns the volume
[](#storage-zfs-volume-conf:volatile.devlxd.owner)
| **Key:** | `volatile.devlxd.owner` |
|--------------|---------------------------|
| **Type:** | string |
| **Default:** | DevLXD owner identity ID |
| **Scope:** | global |
`volatile.idmap.last`
JSON-serialized UID/GID map that has been applied to the volume
[](#storage-zfs-volume-conf:volatile.idmap.last)
| **Key:** | `volatile.idmap.last` |
|----------------|-------------------------|
| **Type:** | string |
| **Condition:** | filesystem |
`volatile.idmap.next`
JSON-serialized UID/GID map that has been applied to the volume
[](#storage-zfs-volume-conf:volatile.idmap.next)
| **Key:** | `volatile.idmap.next` |
|----------------|-------------------------|
| **Type:** | string |
| **Condition:** | filesystem |
`volatile.uuid`
Volume UUID
[](#storage-zfs-volume-conf:volatile.uuid)
| **Key:** | `volatile.uuid` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | random UUID |
| **Scope:** | global |
`zfs.block_mode`
Whether to use a formatted `zvol` rather than a dataset
[](#storage-zfs-volume-conf:zfs.block_mode)
| **Key:** | `zfs.block_mode` |
|--------------|---------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.zfs.block_mode` |
| **Scope:** | global |
`zfs.block_mode` can be set only for custom storage volumes.
To enable ZFS block mode for all storage volumes in the pool, including instance volumes,
use `volume.zfs.block_mode`.
`zfs.blocksize`
Size of the ZFS block
[](#storage-zfs-volume-conf:zfs.blocksize)
| **Key:** | `zfs.blocksize` |
|--------------|--------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.zfs.blocksize` |
| **Scope:** | global |
The size must be between 512 bytes and 16 MiB and must be a power of 2.
For a block volume, a maximum value of 128 KiB will be used even if a higher value is set.
Depending on the value of [`zfs.block_mode`](#storage-zfs-volume-conf:zfs.block_mode),
the specified size is used to set either `volblocksize` or `recordsize` in ZFS.
`zfs.delegate`
Whether to delegate the ZFS dataset
[](#storage-zfs-volume-conf:zfs.delegate)
| **Key:** | `zfs.delegate` |
|----------------|-------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.zfs.delegate` |
| **Condition:** | ZFS 2.2 or higher |
| **Scope:** | global |
This option controls whether to delegate the ZFS dataset and anything underneath it to the
container or containers that use it. When used in conjunction with
[`security.nesting`](https://canonical.com/lxd/docs/latest/reference/instance_options/index.html.md#instance-security:security.nesting), this allows
using the `zfs` command in the container.
`zfs.promote`
Whether to promote the ZFS dataset
[](#storage-zfs-volume-conf:zfs.promote)
| **Key:** | `zfs.promote` |
|--------------|-----------------|
| **Type:** | bool |
| **Default:** | false |
| **Scope:** | global |
This option controls whether to promote the ZFS dataset at volume create/refresh time.
When enabled, if the source volume is a clone, the new volume will be promoted to be a parent.
`zfs.remove_snapshots`
Remove snapshots as needed
[](#storage-zfs-volume-conf:zfs.remove_snapshots)
| **Key:** | `zfs.remove_snapshots` |
|--------------|--------------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.zfs.remove_snapshots` or `false` |
| **Scope:** | global |
`zfs.reserve_space`
Use `reservation`/`refreservation` along with `quota`/`refquota`
[](#storage-zfs-volume-conf:zfs.reserve_space)
| **Key:** | `zfs.reserve_space` |
|--------------|-----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.zfs.reserve_space` or `false` |
| **Scope:** | global |
`zfs.use_refquota`
Use `refquota` instead of `quota` for space
[](#storage-zfs-volume-conf:zfs.use_refquota)
| **Key:** | `zfs.use_refquota` |
|--------------|----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.zfs.use_refquota` or `false` |
| **Scope:** | global |