=,,`.
#### IMPORTANT
The order of the arguments does not actually matter – they are identified based on a regex (pool names must start with a letter and sizes must end with a unit suffix).
`` is a string taken from the charmed operator itself. It encapsulates a specific storage option/feature. Sometimes it is also called a *store*.
The values are as follows:
* ``: the storage pool. See more: [Storage pool](#storage-pool).
* ``: number of volumes
* ``: size of each volume
If at least one constraint is specified the following default values come into effect:
* ``: the default storage pool. See more: [How to manage storage pools](https://documentation.ubuntu.com/juju/2.9/howto/manage-storage-pools.md#manage-storage-pools).
* ``: the minimum number required by the charm, or ‘1’ if the storage is optional
* ``: determined from the charm’s minimum storage size, or 1GiB if the charmed operator does not specify a minimum
### Expand to see an example of a partial specification and its fully-specified equivalent
Suppose you want to deploy PostgreSQL with one instance (count) of 100GiB, via the charm’s ‘pgdata’ storage label, using the default storage pool:
```text
juju deploy postgresql --storage pgdata=100G
```
Assuming an AWS model, where the default storage pool is `ebs`, this is equivalent to:
```text
juju deploy postgresql --storage pgdata=ebs,100G,1
```
In the absence of any storage specification, the storage will be put on the root filesystem (`rootfs`).
`--storage` may be specified multiple times, to support multiple charm labels.
`. Editing a profile will affect all of the containers using it, so you can add loop devices to all LXD containers by editing the ‘default’ profile, or you can scope it to a model.
To add loop devices to your container, add entries to the ‘default’, or model-specific, profile, with `lxc profile edit `:
```yaml
...
devices:
loop-control:
major: "10"
minor: "237"
path: /dev/loop-control
type: unix-char
loop0:
major: "7"
minor: "0"
path: /dev/loop0
type: unix-block
loop1:
major: "7"
minor: "1"
path: /dev/loop1
type: unix-block
...
loop9:
major: "7"
minor: "9"
path: /dev/loop9
type: unix-block
```
Doing so will expose the loop devices so the container can acquire them via the `losetup` command. However, it is not sufficient to enable the container to mount filesystems onto the loop devices. One way to achieve that is to make the container “privileged” by adding:
```yaml
config:
security.privileged: "true"
```
### `maas`
MAAS has support for discovering information about machine disks, and an API for acquiring nodes with specified disk parameters. Juju’s MAAS provider has an integrated ‘maas’ storage provider. This storage provider is static-only; it is only possible to deploy charmed operators using ‘maas’ storage to a new machine in MAAS, and not to an existing machine, as described in the section on dynamic storage.
The MAAS provider currently has a single configuration attribute:
**tags**
* A comma-separated list of tags to match on the disks in MAAS. For example, you might tag some disks as ‘fast’; you can then create a storage pool in Juju that will draw from the disks with those tags.
### `oracle`
Oracle-based models have access to the ‘oracle’ storage provider. The Oracle provider currently supports a single pool configuration attribute:
**volume-type**
* Volume type, a value of ‘default’ or ‘latency’. Use ‘latency’ for low-latency, high IOPS requirements, and ‘default’ otherwise.
For convenience, the Oracle provider registers two predefined pools:
* ‘oracle’ (volume type is ‘default’)
* ‘oracle-latency’ (volume type is ‘latency’).
/
# ...
```
The above snippet will ensure that both the `important-app` container and charm container inside each Pod has the `logs` store mounted. Under the hood, the `storage` map is translated into a series of [`PersistentVolume`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)s, mounted into Pods with [`PersistentVolumeClaim`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)s.
#### NOTE
The `location` attribute *must* be specified when mounting a storage into a workload container as shown above - this will dictate the mount point for the specific container.
Optionally, developers can specify the `location` attribute on the storage itself, which will specify the mount point in the charm container. If left unset, the charm container will have the storage volume mounted at a predictable path at `/var/lib/juju/storage//`, where `` is the index of the storage. This defaults to `0`.
For the above `metadata.yaml`, the charm container would have the storage available at: `/var/lib/juju/storage/logs/0`.
## Storage events
There are two key events associated with storage:
| Event name | Event Type | Description |
|----------------------------|------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `_storage_attached` | [`StorageAttachedEvents`](https://ops.readthedocs.io/en/latest/index.html#ops.StorageAttachedEvent) | This event is triggered when new storage is available for the charm to use. Callback methods bound to this event allow the charm to run code when storage has been added._storage_detaching` | [`StorageDetachingEvent`](https://ops.readthedocs.io/en/latest/index.html#ops.StorageDetachingEvent) | Callback methods bound to this event allow the charm to run code before storage is removed./` where `num`
is zero for “normal”/singular storages or integer id for storages that support `multiple`
attachments.
The operator framework provides the `Model.storages` dict-like member that maps storage names to a
list of storages mounted under that name. It is a list in order to handle the case of storage
configured for multiple instances. For the basic singular case, you will simply access the
first/only element of this list.
Charm developers should *not* directly assume a location/path for mounted storage. To access
mounted storage resources, retrieve the desired storage’s mount location from within your charm
code - e.g.:
```python
def _my_hook_function(self, event):
...
storage = self.model.storages['my-storage'][0]
root = storage.location
fname = 'foo.txt'
fpath = os.path.join(root, fname)
with open(fpath, 'w') as f:
f.write('super important config info')
...
```
This example utilizes the framework’s representation of juju storage - i.e. `self.model.storages`
which returns a [`mapping`](https://ops.readthedocs.io/en/latest/index.html#ops.StorageMapping) of
`` to [`Storage`](https://ops.readthedocs.io/en/latest/index.html#ops.Storage)
objects, which exposes the `name`, `id` and `location` of each storage to the charm developer,
where `id` is the underlying storage provider ID.
If you have also mounted storage in a container, that storage will be located directly at the
specified mount location. For example with the following content in your metadata.yaml:
```default
containers:
foo:
resource: foo-image
mounts:
- storage: data
location: /foo-data
```
storage for the “foo” container will be mounted directly at `/foo-data`. There are no storage name
or integer-indexed subdirectories. Juju does not currently support multiple storage instances for
charms using “containers” functionality. If you are writing a container-based charm (e.g. for
kubernetes clouds) it is best to have your charm code communicate the storage location to the
workload rather than hard-coding the storage path in the container itself. This can be
accomplished by various means. One method is passing the mount path via a file using the
`Container` API:
```python
def _on_mystorage_storage_attached(self, event):
# get the mount path from the charm metadata
container_meta = self.framework.meta.containers['my-container']
storage_path = container_meta.mounts['my-storage'].location
# push the path to the workload container
c = self.model.unit.get_container('my-container')
c.push('/my-app-config/storage-path.cfg', storage_path)
... # tell workload service to reload config/restart, etc.
```
### Scaling storage
While juju provides an `add-storage` command, this does not “grow” existing storage
instances/mounts like you might expect. Rather it works by increasing the number of storage
instances available/mounted for storages configured with the `multiple` parameter. For charm
development, handling storage scaling (add/detach) amounts to handling `_storage_attached`
and ` my-storage=32G,2` will add two additional instances to this storage - e.g.:
`/var/lib/juju/storage/my-storage/2` and `/var/lib/juju/storage/my-storage/3`. “Adding” storage
does not modify or affect existing storage mounts. This would generate two separate
storage-attached events that should be handled.
In addition to juju client requests for adding storage, the [`StorageMapping`](https://ops.readthedocs.io/en/latest/index.html#ops.StorageMapping)
returned by `self.model.storages` also exposes a
[`request`](https://ops.readthedocs.io/en/latest/index.html#ops.StorageMapping.request)
method (e.g. `self.model.storages.request()`) which provides an expedient method for the developer
to invoke the underlying
[`storage-add`](https://discourse.charmhub.io/t/hook-tools/1163#heading--storage-add) hook tool in
the charm to request additional storage. On success, this will fire a
`-storage-attached` event.
kubernetes storage-class= storage-provisioner=kubernetes.io/no-provisioner`
*classname* is the base storage class name assigned to each volume.
*poolname* will be used when deploying the charm.
Kubernetes will pick an available available volume each time it needs to provide storage to a new pod. Once a volume is used, it is never re-used, even if the unit/pod is terminated and the volume is released. Just as volumes are manually created, they must also be manually deleted.
This approach is useful for testing/protyping. If you deploy the kubernetes-core bundle, you can create one or more “host path” persistent volumes on the worker node (each mounted to a different directory). Here’s an example YAML config file to use with kubectl to create 1 volume:
```default
kind: PersistentVolume
apiVersion: v1
metadata:
name: mariadb-data
spec:
capacity:
storage: 100Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: -mariadb-unit-storage
hostPath:
path: "/mnt/data"
```
You’d tweak the host path and volume name to create a selection of persistent volumes to test with - remember, each manually created volume can only be used once.
**Note:** the storage class name in the PV YAML above has the model name prepended to it. This is because storage classes are global to the cluster and so Juju will prepend the model name to disambiguate. So you will need to know the model name when setting up static PVs. Or you can create them and edit the storage class attribute later using `kubectl edit`.
Then create the Juju storage pool:
```text
juju create-storage-pool test-storage kubernetes storage-class=mariadb-unit-storage storage-provisioner=kubernetes.io/no-provisioner
```
Now deploy the charm:
```text
juju deploy cs:~juju/mariadb-k8s --storage database=10M,test-storage
```
Juju will create a suitably named Kubernetes storage class with the relevant provisioner type to enable the use of the statically created volumes.
#### Dynamic Persistent Volumes
To allow for Kubernetes to create persistent volumes on demand, a Kubernetes storage class is used. This is done automatically by Juju if you create a storage pool. As with vm based clouds, a Juju storage pool configures different classes of storage which are available to use with deployed charm.
It’s also possible to set up a Kubernetes storage class manually and have finer grained control over how things tie together, but that’s beyond the scope of this topic.
Before deploying your charm which requires storage, create a Juju storage pool which defines what backing provider will be used to provision the dynamic persistent volumes. The backing provider is specific to the underlying cloud and more details are available in the Kubernetes [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) documentation.
The example below is for a Kubernetes cluster deployed on AWS requiring EBS persistent volumes of type gp2. The name of the pool is arbitrary - in this case *k8s-eb2*. Note that the Kubernetes cluster needs be deployed with the cloud specific integrator charm as described earlier.
`juju create-storage-pool k8s-ebs kubernetes storage-class=juju-ebs storage-provisioner=kubernetes.io/aws-ebs parameters.type=gp2`
You can see what storage pools have been set up in Juju.
`juju storage-pools`
**Note**: only pools of type “kubernetes” are currently supported. rootfs, tmpfs and loop are unsupported.
Once a storage pool is set up, to define how Kubernetes should be configured to provide dynamic volumes, you can go ahead a deploy a charm using the standard Juju storage directives.
`juju deploy cs:~juju/mariadb-k8s --storage database=10M,k8s-ebs`
Use `juju storage` command (and its variants) to see the state of the storage.
If you scale up
`juju scale-application mariadb 3`
you will see that 2 need EBS volumes are created and become attached.
If you scale down
`juju scale-application mariadb 2`
you will see that one of the EBS volumes becomes detached but is still associated wih the model.
Scaling up again
`juju scale-application mariadb 3`
will result in this detached storage being reused and attached to the new unit.
Destroying the entire model will result in all persistent volumes for that model also being deleted.
#### What’s next?
[https://discourse.jujucharms.com/t/advanced-storage-support/204](https://discourse.jujucharms.com/t/advanced-storage-support/204)