--- title: Leveling up snapd integration tests description: Over the last several months there has been noticeable and growing pain associated with the evolving integration tests around snapd, and given the project goal of being a cross-distribution platform, we are very keen on solving this problem appropriately so that stability is guaranteed everywhere.With that mindset a more focused effort wa […] url: https://canonical.com/blog/leveling-up-snapd-integration-tests?format=md --- 1. [Blog](https://canonical.com/blog) 2. Article --- [Gustavo Niemeyer](https://canonical.com/blog/author/niemeyer "More about Gustavo Niemeyer") 16 June 2016 # Leveling up snapd integration tests [containers](https://canonical.com/blog/tag/containers) [snapcraft](https://canonical.com/blog/tag/snapcraft) [snapd](https://canonical.com/blog/tag/snapd) [Snaps](https://canonical.com/blog/tag/snaps) --- Share the article Over the last several months there has been noticeable and growing pain associated with the evolving integration tests around [snapd](https://github.com/snapcore/snapd), and given the project goal of being a [cross-distribution platform](https://insights.ubuntu.com/2016/06/14/universal-snap-packages-launch-on-multiple-linux-distros/), we are very keen on solving this problem appropriately so that stability is guaranteed everywhere. With that mindset a more focused effort was made over the last few weeks to produce a tool that can get the project out of those problems, and onto a runway of more pleasant stability. Despite the short amount of time, I’m very happy about the *Spread* project which resulted from this effort. Spread is not Jenkins or Travis, and is not a language or library either. Spread is a tool that will very conveniently ship your code to one or more systems, in parallel, and then offer the right set of options so you can run whatever you need to run to make sure the logic is working, and drive it all from the local system. That implies you can run Spread *inside* Travis, Jenkins, or your terminal, in a similar way to how your unit tests work. Here is a short list of interesting facts about Spread: * Full-system tests with on demand machine allocation. * Multi-backend with Linode and LXD (for local runs) out of the box for now. * Multi-language since it can run arbitrary remote code. * Agent-less and driven via embedded ssh (kudos to [Go team](http://golang.org/)). * Convenient harness with project+backend+suite+test prepare and restore scripts. * Variants feature for test duplication without copy & paste. * Great debugging support – add *-debug* and stop with a shell inside every failure. * Reuse of servers – server allocation is fast, but not allocating is faster. * Reasonable test outputs with the shell’s +x mode on failures. * … and so forth. This is all [well documented](https://github.com/snapcore/spread), so I’ll just provide one example here to offer a real taste of how the system feels like. This is `spread.yaml`, put in the project root to define the basics: ``` project: spread backends: lxd: systems: - ubuntu-16.04 - ubuntu-14.04 path: /home/test prepare: | echo Entering project... restore: | echo Leaving project... suites: tests/: summary: Integration tests prepare: | echo Entering suite... restore: | echo Leaving suite... ``` The suite name is also the path under which the tests are found. Then, this is `tests/hello/task.yaml`: ``` summary: Greet the world prepare: | echo "Entering task..." restore: | echo "Leaving task..." environment: FOO/a: one FOO/b: two execute: | echo "Hello world!" [ $FOO = one ] || exit 1 ``` The outcome should be almost obvious (intended feature :-). The one curious detail here is the *FOO/a* and *FOO/b* environment variables. This is how to introduce *variants*, which means this one test will in fact become two: first with *FOO=one*, and then with *FOO=two*. Now consider that such environment variables can be defined at any level – project, backend, suite, and task – and imagine how easy it is to test small variations without any copy & paste. After cascading takes place *(project→backend→suite→task)* all environment variables using a given variant key will be present at once on the same execution. Now let’s try to run this configuration, including the `-debug` flag so we get a shell on the failures. Note how with a single test we get four different jobs, two variants over two systems, with the variant *b* failing as instructed: ``` $ spread -debug 2016/06/11 19:09:27 Allocating lxd:ubuntu-14.04... 2016/06/11 19:09:27 Allocating lxd:ubuntu-16.04... 2016/06/11 19:09:41 Waiting for LXD container to have an address... 2016/06/11 19:09:43 Waiting for LXD container to have an address... 2016/06/11 19:09:44 Allocated lxd:ubuntu-14.04. 2016/06/11 19:09:44 Connecting to lxd:ubuntu-14.04... 2016/06/11 19:09:48 Allocated lxd:ubuntu-16.04. 2016/06/11 19:09:48 Connecting to lxd:ubuntu-16.04... 2016/06/11 19:09:52 Connected to lxd:ubuntu-14.04. 2016/06/11 19:09:52 Sending project data to lxd:ubuntu-14.04... 2016/06/11 19:09:53 Connected to lxd:ubuntu-16.04. 2016/06/11 19:09:53 Sending project data to lxd:ubuntu-16.04... 2016/06/11 19:09:54 Error executing lxd:ubuntu-14.04:tests/hello:b : ----- + echo Hello world! Hello world! + [ two = one ] + exit 1 ----- 2016/06/11 19:09:54 Starting shell to debug... lxd:ubuntu-14.04 ~/tests/hello# echo $FOO two lxd:ubuntu-14.04 ~/tests/hello# cat /etc/os-release | grep ^PRETTY PRETTY_NAME="Ubuntu 14.04.4 LTS" lxd:ubuntu-14.04 ~/tests/hello# exit exit 2016/06/11 19:09:55 Error executing lxd:ubuntu-16.04:tests/hello:b : ----- + echo Hello world! Hello world! + [ two = one ] + exit 1 ----- 2016/06/11 19:09:55 Starting shell to debug... lxd:ubuntu-16.04 ~/tests/hello# echo $FOO two lxd:ubuntu-16.04 ~/tests/hello# cat /etc/os-release | grep ^PRETTY PRETTY_NAME="Ubuntu 16.04 LTS" lxd:ubuntu-16.04 ~/tests/hello# exit exit 2016/06/11 19:10:33 Discarding lxd:ubuntu-14.04 (spread-129)... 2016/06/11 19:11:04 Discarding lxd:ubuntu-16.04 (spread-130)... 2016/06/11 19:11:05 Successful tasks 2016/06/11 19:11:05 Aborted tasks: 0 2016/06/11 19:11:05 Failed tasks: 2 - lxd:ubuntu-14.04:tests/hello:b - lxd:ubuntu-16.04:tests/hello:b error: unsuccessful run ``` This demonstrates many of the stated goals (parallelism, clarity, convenience, debugging, …) while running on a local system. Running on a remote system is just as easy by using an appropriate backend. The snapd [project on GitHub](https://github.com/snapcore/snapd), for example, is hooked up on Travis to run Spread and then ship its tests over to Linode. Here is a [real run output](https://travis-ci.org/snapcore/snapd/jobs/138065072) with the initial tests being ported, and a [basic smoke test](https://github.com/snapcore/snapd/blob/master/tests/basics/task.yaml). If you like what you see, by all means please [go ahead](https://github.com/snapcore/spread) and make good use of it. We’re all for more stability and sanity everywhere. [Original article](http://blog.labix.org/2016/06/16/leveling-up-snapd-integration-tests "Original article") ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical.com/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### TurtleBot3 OpenCR firmware update from a snap](https://canonical.com/blog/turtlebot3-opencr-firmware-update-from-a-snap) The TurtleBot3 robot is a standard platform robot in the ROS community, and it’s a reference that Canonical knows well, since we’ve used it in our tutorials. As a matter of... [gbeuzeboc](https://canonical.com/blog/author/gbeuzeboc) 25 September 2024 [### Canonical partners with Snyk for scanning chiseled Ubuntu containers](https://canonical.com/blog/canonical-partners-with-snyk-for-scanning-chiseled-ubuntu-containers) Canonical, the publisher of Ubuntu, is pleased to announce a new partnership with developer-focused cybersecurity company Snyk. Snyk Container, Snyk’s container security... [Canonical](https://canonical.com/blog/author/canonical) 20 March 2026 [### Source to production: Spring Boot containers made easy](https://canonical.com/blog/spring-boot-containers-made-easy) This blog is contributed by Pushkar Kulkarni, a Software Engineer at Canonical. Building on the rise in popularity of Spring Boot and the 12 factor paradigm, our Java offering... [Massimiliano Gori](https://canonical.com/blog/author/massigori) 2 July 2025 [### Spring support available on Ubuntu](https://canonical.com/blog/devpack-spring-support-ubuntu) This blog is contributed by Vladimir Petko, a Software Engineer at Canonical. The release of Plucky Puffin earlier this year introduced the availability of the devpack for... [Massimiliano Gori](https://canonical.com/blog/author/massigori) 2 July 2025