--- title: From docs to Schema description: 'This article was originally featured on Chad Smith’s blog Is there an echo in here? When looking through cloud-config modules it seemed there was a lot of boilerplate documentation and logic in each module to document and validate accepted configuration keys for the module. Houston, we have a problem Problem 1: Doc rot Cloud-init has […]' url: https://canonical.com/blog/from-docs-to-schema?format=md --- 1. [Blog](https://canonical.com/blog) 2. Article --- [Canonical](https://canonical.com/blog/author/canonical "More about Canonical") 24 August 2017 # From docs to Schema --- Share the article *This article was originally featured [on Chad Smith’s blog](https://blackboxsw.github.io/)* Is there an echo in here? When looking through cloud-config modules it seemed there was a lot of boilerplate documentation and logic in each module to document and validate accepted configuration keys for the module. ## Houston, we have a problem ### Problem 1: Doc rot Cloud-init has 51 python modules which define the configuration functions for cloud-config features. Each module has a set of supported YAML configuration options which are documented @ http://cloudinit.readthedocs.io. Documentation of new configuration options need to be updated with changes to module supported options. We’re all (mostly) human, and here’s where our friend “doc rot” enters our project. It is easy to forget to update documentation to match changed features. ### Problem 2: Repetitive docs and configuration option parsing Each cloud-config module has a boilerplate reStructured text docstring describing all configuration option for the module. Most modules also check presence of a top-level configuration key before parsing or skipping a given config. This key definition could be encoded in a simple structure which can be sourced for both documentation and initial config parse. Let’s observe a DRY approach to docs and module configuration definitions. ### Problem 3: Absent config validation Most cloud-config modules do little validation on the configuration options provided to each module. While appearing flexible, the lack of validation ultimately costs the user time and clarity due to terse KeyError or ValueError tracebacks which could be better handled if more strict validation were performed. ## Solution: One schema to rule them all Performing validation using a strict declarative schema has the following benefits: * a declared schema is an **explicit API contract** between the module and the configuration making it easier to consume due to reduced ambiguity * a schema definition **improves automated testing coverage** by describing all supported options which can be exercised * **avoid stale docs** by tightly coupling documentation to our config validation * strict validation versus permissive acceptance **reduces cost of failures** by addressing invalid configuration errors exlicitly and early instead of at deployment time. * performing upfront schema validation on the entire config allows for **reporting multiple errors in one pass** instead of individually hitting them at runtime. ### Step 1: Add JSONchema definitions for each cloud-config module from which documentation can also be generated. The ntp module, which supports optional **servers** and **pools** keys, shows and easy schema which codifies each property name, type and expected format: ``` schema = { 'id': 'cc_ntp', 'name': 'NTP', 'title': 'enable and configure ntp', 'description': 'Something with ntp', 'distros': ['centos', 'ubuntu',...], 'examples': [...], 'properties': { 'ntp': { 'properties': { 'pools': { 'type': 'array', 'items': { 'type': 'string', 'format': 'hostname' }, 'uniqueItems': True, }, 'servers': { 'type': 'array', 'items': { 'type': 'string', 'format': 'hostname' }, 'uniqueItems': True, } }, 'required': [], # No required properties 'additionalProperties': False # Error on unregistered properties } } } ``` ### Step 2: Add simple helper functions to generate sphinx docs from schema dict instead of module docstrings The magic in sphinx doc generation is overriding the default module-level docstring behavior to make use of docs rendered from schema definition. This docstring generating callback needs to be added to your conf.py in the directory where you run sphinx: ``` def generate_docstring_from_schema(app, what, name, obj, options, lines): """Override module docs from schema when present.""" if what == 'module' and hasattr(obj, "schema"): del lines[:] lines.extend(get_schema_doc(obj.schema).split('\n')) def setup(app): app.connect('autodoc-process-docstring', generate_docstring_from_schema) ``` And the simple doc-generation from schema function is below: ``` SCHEMA_DOC_TMPL = """ {name} --- **Summary:** {title} {description} **Internal name:** ``{id}`` **Module frequency:** {frequency} **Supported distros:** {distros} **Config schema**: {property_doc} {examples} """ def get_schema_doc(schema): """Return reStructured text rendering the provided jsonschema. @param schema: Dict of jsonschema to render. @raise KeyError: If schema lacks an expected key. """ schema['property_doc'] = _get_property_doc(schema) schema['examples'] = _get_schema_examples(schema) schema['distros'] = ', '.join(schema['distros']) return SCHEMA_DOC_TMPL.format(**schema) ``` ### Step 3: The module handler function will iterate over schema errors with jsonschema.Validator and log collected warnings for all schema infractions. ``` from jsonschema import Draft4Validator, FormatChecker validator = Draft4Validator(schema, format_checker=FormatChecker()) for error in sorted(validator.iter_errors(config), key=lambda e: e.path): path = '.'.join([str(p) for p in error.path]) errors += ((path, error.message),) if errors: raise SchemaValidationError(errors) ``` ### Step 4: Simple cmdline tools to validate cloud-config files against known schema to avoid costly errors during instance deployment. Already included with cloud-init 0.7.9 is a minimal schema validation development tool: ``` python3 -m cloudinit.config.schema --help ``` [Get in touch Interested in running Ubuntu in your organization?](https://ubuntu.com/about/contact-us/form) ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical.com/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### Canonical joins the Open Secure AI Alliance](https://canonical.com/blog/open-secure-ai-alliance) Canonical is now part of the Open Secure AI Alliance, announced by NVIDIA with partners across cloud computing, cybersecurity, enterprise software, open source foundations, and... [Canonical](https://canonical.com/blog/author/canonical) 28 August 2026 [### Arduino® VENTUNO™ Q is available for pre-order with Ubuntu pre-installed](https://canonical.com/blog/arduino-ventuno-q-is-available-for-pre-order-with-ubuntu-pre-installed) London, UK – August 25, 2026 – Following our initial collaboration announcement in March 2026, Canonical and Arduino (a subsidiary of Qualcomm Technologies, Inc.) are excited... [Canonical](https://canonical.com/blog/author/canonical) 25 August 2026 [### Advantech AOM-2721 is now Ubuntu Certified](https://canonical.com/blog/advantech-aom-2721-ubuntu-certified) Canonical announces that the Advantech AOM-2721 is officially joining the list of Ubuntu Certified Hardware. [Mikhail Khazov](https://canonical.com/blog/author/khazov-m) 13 August 2026 [### Canonical integrates NVIDIA Nemotron 3.5 Lightning with Ubuntu for always-on AI agents](https://canonical.com/blog/nvidia-nemotron-3-5-lightning) Canonical is pleased to announce that NVIDIA’s newly introduced NVIDIA Nemotron 3.5 Lightning, an open, customizable model built for always-on AI agents, is now available on... [Canonical](https://canonical.com/blog/author/canonical) 11 August 2026