How to migrate to the charmlibs monorepo

This guide will walk you through migrating an existing Charmhub-hosted library to the charmlibs monorepo.

Tip

This guide is for library authors. If you’re a library user trying to figure out how to switch to the charmlibs version of a library, all you need to do is:

  1. Delete your vendored copy of the Charmhub-hosted library, remove any references to it in your charmcraft.yaml, and remove any transitive dependencies you added.

  2. Add the library to your charm as a regular Python dependency (with the appropriate version constraints).

  3. Update any imports to refer to the new library – the library’s reference docs should explain this.

Get started

The first thing to check is whether the library you’re migrating is an interface library. That is, is it responsible for abstracting away the management of a specific interface’s databag contents for charms? If it is, it will be distributed under the charmlibs.interfaces namespace instead of the charmlibs namespace, and you’ll want to think about interface definitions and tests too.

After cloning your fork of the charmlibs monorepo, run the following command to add a new directory for your library with an appropriately structured Python package and tests.

just init

just init --interface

You’ll be prompted for information about your package, with default values shown in brackets.

The most important piece of information to get right initially is the project name. This determines the name of your library’s project directory, which most of the just commands run later in this guide require. We’ll refer to this as <library path> in examples.

  • For a general library, the project name is the import package name (without the charmlibs. namespace).

  • For an interface library, the project name is interfaces/<interface name>.

Read on for more details about naming and generating your project, depending on whether you’re migrating a general library or an interface library.

General libraries

The project name must be the name of the package as users would type it when they import with:

from charmlibs import <your package name>

Typically this should be the <module name> component of the Charmhub-hosted lib’s name: <charm name>.v<n>.<module name>. For example, operator_libs_linux.v2.snap is now available as charmlibs.snap.

Important

The package name must be unique across the charmlibs namespace packages defined in the charmlibs monorepo. This is different from Charmhub, where the name only needed to be unique across the libraries defined by a specific charm.

If the name you want to use for your library is already taken, consider the following:

  1. Is the functionality you’re looking for available from the existing library with the same name?

    1. If not, would it make sense to add it to that library?

    2. If so, perhaps the library you’re looking at migrating could be deprecated without migration.

  2. Is there another logical name your library could use?

Interface libraries

The project name must be the canonical name of the interface from the charms’ perspective – exactly as it is spelled in charmcraft.yaml files.

This means it might be a hyphenated name or an underscored name. The important thing is that it exactly matches the actual interface name.

The interface name must be unique across the charming ecosystem – including the interfaces/ directory of the monorepo.

If init fails because the directory already exists, take a look at the directory. It may be that the interface definitions are already hosted in the repo under interfaces/<interface name>/interface. In this case:

  • Temporarily move the <interface name> directory.

  • Re-run just init --interface.

  • Then add the interface subdirectory to your newly generated project.

You’ll also want to check for any config files under the old <interface name> directory (for example, a ruff.toml file), and incorporate any applicable settings into your project’s pyproject.toml.

Migrate your library’s code

This is the easy bit, since Charmhub-hosted libs are only a single module. Download a copy of the latest release of your library, and add it to your new package as a private module, alongside the __init__.py file.

<library path>/src/charmlibs/<name>/_<name>.py

<library path>/src/charmlibs/interfaces/<name>/_<name>.py

Tip

Add dependencies with the just add <library path> <args...> command. This will automatically respect any repo-level version constraints imposed by the tool versions used in CI. This uses uv add under the hood – any arguments after <library path> are passed to it. For example:

just add pathops 'pydantic>=2' 'requests~=2.3'
just add interfaces/tls-certificates --requirements my-requirements.txt

Now follow these steps to migrate your library’s source code:

  1. Copy the copyright header from __init__.py to _<name>.py to satisfy the linter.

  2. Move the docstring from _<name>.py to __init__.py so that it’s included in your library’s automatically built reference docs.

  3. Document in the _<name>.py docstring the API and patch version of the source code that you’re migrating. This will be helpful for future maintainers and users if they need to debug issues.

  4. Delete LIB_ID, LIB_API, and LIB_PATCH from _<name>.py – unless they’re used internally by the library, then you’ll need to keep them for now.

  5. Move the contents of PYDEPS to the dependencies entry in your pyproject.toml (using just add), and delete the PYDEPS variable. You’ll also need to add any additional dependencies that were assumed to be provided by the charm, like ops or pydantic. Consider adding version constraints to your dependencies too.

  6. Import the public API of your library to __init__.py and add the imported names to __all__, like this:

# immediately before or after from ._version
# (imports are sorted alphabetically)
from ._<name> import (
    # your library's public API
)

...

__all__ = [
    # the names we imported, as strings
]

You can now test that your library can be built and imported by running the simple unit tests that your project was initialized with. From anywhere in the repo, run the following command:

just unit <library path>

Tip

Commit your code now if you haven’t already!

To be merged, your library will need to comply with the repo’s linting and static type checking. Check how you’re doing by running:

just lint <library path>

Consider running just format to handle any automatically fixable errors.

You can also check if your docstrings are compatible with the format that Sphinx expects when building the reference docs. From anywhere in the repo, run just docs. This builds the reference docs for all the libraries. To speed things up, only build the reference docs for your library:

just docs html <library path>

Migrate your library’s tests

This part is a bit trickier. With any luck, your library was previously developed in a placeholder charm that exists purely for library distribution. If your library’s development and testing was tightly coupled to a real charm, this step will be more involved. You’ll need to consider which tests can live alongside the library, and which only make sense with the charm. You might want to add a simplified placeholder charm to run some of the tests against.

Warning

Don’t add pytest to your pyproject.toml.

just unit <library path> will install and run a specific version of pytest, which may clash with the version added in your dependencies. Instead, use just to run tests – any extra arguments will be passed to pytest. You can point your IDE to uptime/.venv after running any of the test commands to have it use the correct virtual environment.

Unit tests

If your library wasn’t tightly coupled to a real charm, these steps should be sufficient:

  1. Add any unit test dependencies to the unit dependency group in your pyproject.toml (using just add).

  2. Copy any relevant contents of your conftest.py to tests/unit/conftest.py.

  3. Copy your library’s existing unit test files to tests/unit/, along with any data files, placeholder charms, and so on.

  4. Correct the imports in those files.

Replace imports like this:

from charms.<charm>.v<n> import <name>
from charms.<charm>.v<n>.<name> import ...

With imports like this:

from charmlibs import <name>
from charmlibs.<name> import ...

from charmlibs.interfaces import <name>
from charmlibs.interfaces.<name> import ...

There’s now a good chance that the following command will successfully run your unit tests!

just unit <library path>

Functional tests

While unit tests are run across a selection of the Python versions that your library supports, functional tests are run on different Ubuntu bases using the system Python. They’re intended for tests that interact with the real world, but don’t require a real Juju deployment.

The process for migrating them is exactly the same as for unit tests.

Read more: Add functional tests, Functional tests

Integration tests

Integration tests involve packing your library into a charm and deploying it on a real Juju model.

Read more: Add integration tests, Integration tests

If you take a look at your <library path>/tests/integration directory, you’ll see a pack.sh script. Currently it packs a simple k8s or machine charm, depending on the CHARMLIBS_SUBSTRATE variable that is set in CI. In CI, the script is executed by just pack-k8s or just pack-machine. The integration tests provided by the template use jubilant to deploy and test the packed charm. They’re executed by just integration-k8s or just integration-machine.

The simple k8s and machine charms are defined in the <library path>/tests/integration/charms directory. You’re more than welcome to fit your existing integration tests into this structure. However, the use of the pack.sh script is completely optional – you’re free to remove it entirely, in which case that step is skipped in CI. This is especially useful if your integration tests used pytest-operator to pack and deploy charms from the tests themselves.

In CI, integration tests are run (separately) with a Juju machine cloud and a Juju K8s cloud. The charmlibs CI is aware of two special pytest marks: k8s_only and machine_only. If there are no tests compatible with a substrate, then it’s skipped completely. By default each test is treated as compatible with both substrates.

Migrate your library’s docs

Your library’s reference documentation is automatically built from its docstrings and source code. On top of that, you can add tutorials, how-to guides, and explanations under <library path>/docs, and they’ll be included in this documentation site under the respective categories.

Read more: How to add docs to a library

Note

There is no reference docs category – reference documentation is generated from your library’s docstrings. If your existing docs include hand-written reference material, fold the relevant content into your docstrings, or rework it into an explanation or how-to guide.

If your library already has documentation hosted on Charmhub, here’s how to bring it across.

Find the source topics

Charmhub renders a charm’s documentation from a set of topics on Discourse. To find them:

  1. Start from the charm’s Charmhub page – https://charmhub.io/<charm>.

  2. Follow the Help improve this document in the forum link at the bottom of the page. This link points to the Discourse root topic for the charm’s docs.

The root topic contains a table of contents linking each child topic. This usually already encodes a Diátaxis-like layout (tutorial / how-to / explanation / reference) that you can carry over.

Tip

Some charms host their docs on a Read the Docs / Sphinx site instead of Discourse. That content isn’t available through the Discourse API, but it’s straightforward to migrate by hand: copy the doc site’s source files into your library’s docs/ directory and adapt them.

Download the topics

Download each Discourse topic into your library’s docs/ directory:

.scripts/import_discourse_docs.py <discourse-url> <library path>/docs/<category>/<page>.md

For example:

.scripts/import_discourse_docs.py \
    https://discourse.charmhub.io/t/tls-certificates-interface/15539 \
    interfaces/tls-certificates/docs/explanation/tls-certificates-interface.md

The script downloads the topic, extracts its Markdown source, and resolves any embedded images.

Repeat for each topic, choosing a Diátaxis category for each.

Finish up

Edit the imported pages to follow the conventions described in How to add docs to a library: give each page a title and meta description, fix up cross-references, and confirm the categorisation.

Update the library metadata

The interface library listing and general library listing are generated from .docs/reference/libs.yaml.

There should already be an entry for the old Charmhub-hosted library. If there isn’t, add one with a deprecated status and a description noting it has been superseded by the new charmlibs package.

Add a new entry for the charmlibs package with the new package name, source URL, status, description, and any relevant tags. See .docs/reference/tags.yaml for the full list of available tags and their assignment criteria. Don’t invent new tags, only use tags defined in tags.yaml.

Deprecate the old library

When migrating an existing Charmhub-hosted library, our recommendation is to do a bug-for-bug migration of the latest release. The new charmlibs package should be released as version 1.0.0, indicating that the API is stable. This will make it as easy as possible for users to migrate.

You will need to provide critical security and bug fixes for the Charmhub-hosted library for some time, but you should immediately mark it as deprecated by adding a prominent comment to the docstring and releasing a new patch version of the library. You should also announce the deprecation in your team’s usual communication channels. You’re free to continue to provide feature updates, but users should not expect them. You should encourage users to migrate to get feature updates.

Don’t add deprecation warnings to the code – we don’t want to flood the Juju logs with warnings. Likewise, don’t remove the library code – we want old charms using charmcraft fetch-libs in their build process to continue to work.