[Ceph](https://ceph.io/en/) is an open-source storage platform that stores its data in a storage cluster based on .
It is highly scalable and, as a distributed system without a single point of failure, very reliable.
#### TIP
If you want to quickly set up a basic Ceph cluster, check out [MicroCeph](https://canonical.com/microcloud).
Ceph provides different components for block storage and for file systems.
is Ceph’s file system component that provides a robust, fully-featured POSIX-compliant distributed file system.
Internally, it maps files to Ceph objects and stores file metadata (for example, file ownership, directory paths, access permissions) in a separate data pool.
## Terminology
Ceph uses the term *object* for the data that it stores.
The daemon that is responsible for storing and managing data is the *Ceph* .
Ceph’s storage is divided into *pools*, which are logical partitions for storing objects.
They are also referred to as *data pools*, *storage pools* or *OSD pools*.
A *CephFS file system* consists of two OSD storage pools, one for the actual data and one for the file metadata.
## `cephfs` driver in LXD
#### NOTE
The `cephfs` driver can only be used for custom storage volumes with content type `filesystem`.
For other storage volumes, use the [Ceph](https://canonical.com/lxd/docs/latest/reference/storage_ceph/index.html.md#storage-ceph) driver.
That driver can also be used for custom storage volumes with content type `filesystem`, but it implements them through Ceph RBD images.
Unlike other storage drivers, this driver does not set up the storage system but assumes that you already have a Ceph cluster installed.
You can either create the CephFS file system that you want to use beforehand and specify it through the [`cephfs.path`](#storage-cephfs-pool-conf:cephfs.path) option, or specify the [`cephfs.create_missing`](#storage-cephfs-pool-conf:cephfs.create_missing) option to automatically create the file system and the data and metadata OSD pools (with the names given in [`cephfs.data_pool`](#storage-cephfs-pool-conf:cephfs.data_pool) and [`cephfs.meta_pool`](#storage-cephfs-pool-conf:cephfs.meta_pool)).
This driver also behaves differently than other drivers in that it provides remote storage.
As a result and depending on the internal network, storage access might be a bit slower than for local storage.
On the other hand, using remote storage has big advantages in a cluster setup, because all cluster members have access to the same storage pools with the exact same contents, without the need to synchronize storage pools.
LXD assumes that it has full control over the OSD storage pool.
Therefore, you should never maintain any file system entities that are not owned by LXD in a LXD OSD storage pool, because LXD might delete them.
The `cephfs` driver in LXD supports snapshots if snapshots are enabled on the server side.
## Configuration options
The following configuration options are available for storage pools that use the `cephfs` driver and for storage volumes in these pools.
### Storage pool configuration
`cephfs.cluster_name`
Name of the Ceph cluster that contains the CephFS file system
[](#storage-cephfs-pool-conf:cephfs.cluster_name)
| **Key:** | `cephfs.cluster_name` |
|--------------|-------------------------|
| **Type:** | string |
| **Default:** | `ceph` |
| **Scope:** | global |
`cephfs.create_missing`
Automatically create the CephFS file system
[](#storage-cephfs-pool-conf:cephfs.create_missing)
| **Key:** | `cephfs.create_missing` |
|--------------|---------------------------|
| **Type:** | bool |
| **Default:** | `false` |
| **Scope:** | global |
Use this option if the CephFS file system does not exist yet.
LXD will then automatically create the file system and the missing data and metadata OSD pools.
`cephfs.data_pool`
Data OSD pool name
[](#storage-cephfs-pool-conf:cephfs.data_pool)
| **Key:** | `cephfs.data_pool` |
|-------------|----------------------|
| **Type:** | string |
| **Scope:** | global |
This option specifies the name for the data OSD pool that should be used when creating
a file system automatically.
`cephfs.fscache`
Enable use of kernel `fscache` and `cachefilesd`
[](#storage-cephfs-pool-conf:cephfs.fscache)
| **Key:** | `cephfs.fscache` |
|--------------|--------------------|
| **Type:** | bool |
| **Default:** | `false` |
| **Scope:** | global |
`cephfs.meta_pool`
Metadata OSD pool name
[](#storage-cephfs-pool-conf:cephfs.meta_pool)
| **Key:** | `cephfs.meta_pool` |
|-------------|----------------------|
| **Type:** | string |
| **Scope:** | global |
This option specifies the name for the file metadata OSD pool that should be used when
creating a file system automatically.
`cephfs.osd_pg_num`
Number of placement groups when creating missing OSD pools
[](#storage-cephfs-pool-conf:cephfs.osd_pg_num)
| **Key:** | `cephfs.osd_pg_num` |
|-------------|-----------------------|
| **Type:** | string |
| **Scope:** | global |
This option specifies the number of OSD pool placement groups (`pg_num`) to use
when creating a missing OSD pool.
`cephfs.osd_pool_size`
Number of RADOS object replicas. Set to 1 for no replication.
[](#storage-cephfs-pool-conf:cephfs.osd_pool_size)
| **Key:** | `cephfs.osd_pool_size` |
|--------------|--------------------------|
| **Type:** | string |
| **Default:** | `3` |
This option specifies the number of OSD pool replicas to use
when creating an OSD pool.
`cephfs.path`
The base path for the CephFS mount
[](#storage-cephfs-pool-conf:cephfs.path)
| **Key:** | `cephfs.path` |
|--------------|-----------------|
| **Type:** | string |
| **Default:** | `/` |
| **Scope:** | global |
This option specifies the base path for the CephFS mount.
The path gets created if missing.
`cephfs.user.name`
The Ceph user to use
[](#storage-cephfs-pool-conf:cephfs.user.name)
| **Key:** | `cephfs.user.name` |
|--------------|----------------------|
| **Type:** | string |
| **Default:** | `admin` |
| **Scope:** | global |
`rsync.bwlimit`
Upper limit on the socket I/O for `rsync`
[](#storage-cephfs-pool-conf:rsync.bwlimit)
| **Key:** | `rsync.bwlimit` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | `0` (no limit) |
| **Scope:** | global |
When `rsync` must be used to transfer storage entities, this option specifies the upper limit
to be placed on the socket I/O.
`rsync.compression`
Whether to use compression while migrating storage pools
[](#storage-cephfs-pool-conf:rsync.compression)
| **Key:** | `rsync.compression` |
|--------------|-----------------------|
| **Type:** | bool |
| **Default:** | `true` |
| **Scope:** | global |
`source.recover`
Whether to recover an existing `source`
[](#storage-cephfs-pool-conf:source.recover)
| **Key:** | `source.recover` |
|--------------|--------------------|
| **Type:** | bool |
| **Default:** | `false` |
| **Scope:** | local |
Set this option to true to recover an existing source which was previously created by LXD.
`user.*`
User-provided free-form key/value pairs
[](#storage-cephfs-pool-conf:user.*)
| **Key:** | `user.*` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | global |
#### TIP
In addition to these configurations, you can also set default values for the storage volume configurations. See storage-configure-vol-default.
### Storage volume configuration
`security.shifted`
Enable ID shifting overlay
[](#storage-cephfs-volume-conf:security.shifted)
| **Key:** | `security.shifted` |
|----------------|----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.shifted` or `false` |
| **Condition:** | custom volume |
| **Scope:** | global |
Enable this option to allow the volume to be attached to multiple isolated instances.
`security.unmapped`
Disable ID mapping for the volume
[](#storage-cephfs-volume-conf:security.unmapped)
| **Key:** | `security.unmapped` |
|----------------|-----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.unmapped` or `false` |
| **Condition:** | custom volume |
| **Scope:** | global |
`size`
Size/quota of the storage volume
[](#storage-cephfs-volume-conf:size)
| **Key:** | `size` |
|----------------|-----------------------|
| **Type:** | string |
| **Default:** | same as `volume.size` |
| **Condition:** | appropriate driver |
| **Scope:** | global |
`snapshots.expiry`
Time until snapshots are deleted
[](#storage-cephfs-volume-conf:snapshots.expiry)
| **Key:** | `snapshots.expiry` |
|----------------|-----------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.snapshots.expiry` |
| **Condition:** | custom volume |
| **Scope:** | global |
Specify an expression like `1M 2H 3d 4w 5m 6y`.
`snapshots.pattern`
Template for the snapshot name
[](#storage-cephfs-volume-conf:snapshots.pattern)
| **Key:** | `snapshots.pattern` |
|----------------|------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.snapshots.pattern` or `snap%d` |
| **Condition:** | custom volume |
| **Scope:** | global |
You can specify a naming template for scheduled snapshots and unnamed snapshots.
The `snapshots.pattern` option takes a Pongo2 template string to format the snapshot name.
To add a time stamp to the snapshot name, use the Pongo2 context variable `creation_date`.
Make sure to format the date in your template string to avoid forbidden characters in the snapshot name.
For example, set `snapshots.pattern` to `{{ creation_date|date:'2006-01-02_15-04-05' }}` to name the snapshots after their time of creation, down to the precision of a second.
Another way to avoid name collisions is to use the placeholder `%d` in the pattern.
If no matching snapshots exist, the placeholder is replaced with `0`.
Otherwise, it is replaced with the next snapshot index, which is one higher than the highest existing matching snapshot index.
`snapshots.schedule`
Schedule for automatic volume snapshots
[](#storage-cephfs-volume-conf:snapshots.schedule)
| **Key:** | `snapshots.schedule` |
|----------------|------------------------------|
| **Type:** | string |
| **Default:** | same as `snapshots.schedule` |
| **Condition:** | custom volume |
| **Scope:** | global |
Specify either a cron expression (``), a comma-separated list of schedule aliases (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@annually`, `@yearly`), or leave empty to disable automatic snapshots (the default).
`user.*`
User-provided free-form key/value pairs
[](#storage-cephfs-volume-conf:user.*)
| **Key:** | `user.*` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | global |
`volatile.devlxd.owner`
ID of the DevLXD identity that owns the volume
[](#storage-cephfs-volume-conf:volatile.devlxd.owner)
| **Key:** | `volatile.devlxd.owner` |
|--------------|---------------------------|
| **Type:** | string |
| **Default:** | DevLXD owner identity ID |
| **Scope:** | global |
`volatile.idmap.last`
JSON-serialized UID/GID map that has been applied to the volume
[](#storage-cephfs-volume-conf:volatile.idmap.last)
| **Key:** | `volatile.idmap.last` |
|----------------|-------------------------|
| **Type:** | string |
| **Condition:** | filesystem |
`volatile.idmap.next`
JSON-serialized UID/GID map that has been applied to the volume
[](#storage-cephfs-volume-conf:volatile.idmap.next)
| **Key:** | `volatile.idmap.next` |
|----------------|-------------------------|
| **Type:** | string |
| **Condition:** | filesystem |
`volatile.uuid`
Volume UUID
[](#storage-cephfs-volume-conf:volatile.uuid)
| **Key:** | `volatile.uuid` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | random UUID |
| **Scope:** | global |