--- title: Minimising latency in your edge cloud with real-time kernel description: Learn how to enable Ubuntu real-time kernel in MicroClouds - a great combination for your low latency edge cloud needs. Performance benchmarks included. url: https://canonical.com/blog/minimising-latency-in-your-edge-cloud-with-real-time-kernel?format=md --- 1. [Blog](https://canonical.com/blog) 2. Article --- [Miona Aleksic](https://canonical.com/blog/author/mionaalex "More about Miona Aleksic") 12 June 2023 # Minimising latency in your edge cloud with real-time kernel [edge](https://canonical.com/blog/tag/edge) [microcloud](https://canonical.com/blog/tag/microcloud) [real-time Linux](https://canonical.com/blog/tag/real-time-linux) [realtime kernel](https://canonical.com/blog/tag/realtime-kernel) --- Share the article *Written by Juan Pablo Noreña* From applications in telecommunications to edge cloud and industrial digital twins, experimenting with real-time capabilities in cloud technologies is a trend in the industry. Applications for the edge often have an additional requirement as they interact with real-time systems: they need to run deterministically. It means that time constraints their execution and interaction within the system. Canonical’s [MicroClouds](https://microcloud.is/) are a small-scale cloud deployment that hosts scalable and highly available applications, providing compute, secure networking, and resilient storage closer to the data sources. They are especially suitable for use cases where workloads need to be served near where they are needed, as with edge cloud deployments. Coupled with the [Real-time kernel feature in Ubuntu](https://ubuntu.com/blog/real-time-ubuntu-is-now-generally-available), MicroClouds are a great solution for applications at the edge with stringent low-latency requirements. This blog presents how to enable the Real-time kernel in MicroCloud hosts. It will also cover our real-time testing, comparing the performance of virtualised and containerised applications leveraging the real-time versus the generic Ubuntu kernel. ## Deploy your MicroCloud If you’re not yet familiar with MicroCloud, it is easy to get started. All you need is to install the necessary components with: ``` sudo snap install lxd microcloud microceph microovn ``` and follow the [initialisation process](https://canonical-microcloud.readthedocs-hosted.com/en/latest/how-to/initialise/) as outlined. ## Enable real-time nodes After [launching your MicroCloud](https://canonical.com/blog/one_command_micro_cloud), which can look like this: ``` ubuntu@ob76-node0:~$ lxc cluster listubuntu@ob76-node0:~$ lxc cluster list ``` ```       NAME          |            URL            |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      | +-----------------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ | ob76-node2.microcloud | https://172.27.77.19:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational | |                       |                           | database         |              |                |             |        |                   | +-----------------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ | ob76-node3.microcloud | https://172.27.77.12:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational | +-----------------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ | ob76-node5.microcloud | https://172.27.77.20:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational | +-----------------------+---------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ | ob76-node6.microcloud | https://172.27.77.11:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational ``` You can [attach an Ubuntu Pro subscription](https://ubuntu.com/server/docs/install/subscription) to the hosts and enable the real-time kernel feature. In this blog, we enable one of the nodes as a real-time node: ``` ubuntu@ob76-node6:~$ sudo pro attach ubuntu@ob76-node6:~$ sudo pro enable realtime-kernel ``` Then, edit the line beginning with “GRUB\_CMDLINE\_LINUX\_DEFAULT” in  **/etc/default/grub**and add the following set of parameters to tune the real-time host: ``` GRUB_CMDLINE_LINUX_DEFAULT="quiet splash skew_tick=1 isolcpus=managed_irq,domain,2-3 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2-3 rcu_nocbs=2-3" ``` Finally, update the grub command line with sudo update-grub. At this point, rebooting is required to apply the changes to the kernel and verify them: ``` ubuntu@ob76-node6:~$ uname -a Linux ob76-node6 5.15.0-1034-realtime #37-Ubuntu SMP PREEMPT_RT Wed Mar 1 20:50:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ob76-node6:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.15.0-1034-realtime root=UUID=449c88ff-8e2e-470e-9f04-4f758114573f ro quiet splash skew_tick=1 isolcpus=managed_irq,domain,2-3 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=2-3 rcu_nocbs=2-3 vt.handoff=7 ``` ### Launch a real-time system container In the case of system containers, the guests share the host kernel and operating system, so no additional configuration is required inside the container. You can launch and pin a system container to one or multiple of the isolated cores and start running your real-time application. ``` ubuntu@ob76-node0:~$ lxc launch ubuntu:22.04 c-rt -c limits.cpu=2 -c security.privileged=true \ -c limits.kernel.rtprio=99 --target ob76-node6.microcloud ``` ### Launch a real-time virtual machine In the case of virtual machines, it is possible to attach the same Ubuntu Pro subscription of the host to the guest and enable the real-time kernel inside the virtual machine: ``` ubuntu@ob76-node0:~$ lxc launch ubuntu:22.04 vm-rt --vm -c limits.cpu=2 \ -c limits.kernel.rtprio=99 --target ob76-node6.microcloud ``` ``` ubuntu@ob76-node0:~$ lxc exec vm-rt -- pro attach ubuntu@ob76-node0:~$ lxc exec vm-rt -- pro enable realtime-kernel ``` Again edit **/etc/default/grub** and modify the line beginning with “GRUB\_CMDLINE\_LINUX\_DEFAULT”: ``` GRUB_CMDLINE_LINUX_DEFAULT="quiet splash skew_tick=1 isolcpus=managed_irq,domain,0 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=0 rcu_nocbs=0" ``` Update the grub with sudo update-grub and reboot the virtual machine. ## Testing and results We used the [cyclictest](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start) from the [rt-tests suite](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/rt-tests) to measure the latency experienced inside a system container and a virtual machine as workloads. Along the instances, we simulated load on the host as a [Worst Case Latency Test Scenario](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/worstcaselatency) to obtain accurate results for stressful environments. Generate heavy CPU load on the host node under test using backbench: ``` ubuntu@ob76-node6:~$ sudo apt update -y && sudo apt install rt-tests -y ubuntu@ob76-node6:~$ while true; do /bin/dd if=/dev/zero of=bigfile bs=1024000 count=1024; done & \ while true; do /usr/bin/killall hackbench; sleep 5; done & \ while true; do hackbench 20; done ``` hackbench in a node with a generic kernel. ### Testing Instances Get a console to the real-time virtual machine and install rt-tests: ``` ubuntu@ob76-node0:~$ lxc exec vm-rt -- /bin/bashroot@vm-rt:~# apt update -y && apt install rt-tests -y ``` Finally, run a cyclictest: ``` root@vm-rt:~# cyclictest -t1 -p 99 -i 1000 -l 1000000000000 -m -a0 -c1 -h 15000 > \ rt-test.output ``` It can be done similarly in the real-time container: ``` ubuntu@ob76-node0:~$ lxc exec c-rt -- /bin/bashroot@c-rt:~# apt update -y && apt install rt-tests -y ``` ``` root@c-rt:~# cyclictest -t1 -p 99 -i 1000 -l 1000000000000 -m -a0 -c1 -h 15000 > \ rt-test.output ``` ### Performance comparison We performed the tests in a MicroCloud built over Intel(R) Nucs. | | | | --- | --- | | **Processor** | Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz | | **OS** | Ubuntu 22.04.2 Jammy | | **Generic (gen) kernel** | Linux 5.15.0-69-generic #76-Ubuntu SMP | | **Real-time (rt) kernel** | Linux 5.15.0-1034-realtime #37-Ubuntu SMP PREEMT\_RT | The following table shows the expected latency between system containers and virtual machines in generic and real-time environments under the stress conditions simulated using hackbench: | | | | | | | --- | --- | --- | --- | --- | | **Kernel** | **Instance type** | **Min latency** | **Avg latency** | **Max latency****(Worst Case Scenario)** | | Host w/ gen kernel | bare-metal | 3 μs | 12 μs | 1496 μs | | Host w/ gen kernel | container | 3 μs | 20 μs | 2118 μs | | Host w/ gen kernel – Guest w/ gen kernel | virtual machine | 4 μs | 105 μs | 13579 μs | | Host w/ rt kernel | bare-metal | 3 μs | 4 μs | 17 μs | | Host w/ rt kernel | container | 3 μs | 7 μs | 21 μs | | Host w/ rt kernel – Guest w/ gen kernel | virtual machine | 4 μs | 15 μs | 4472 μs | | Host w/ rt kernel – Guest w/ rt kernel | virtual machine | 4  μs | 13 μs | 2282 μs | ## Conclusion Real-time applications in cloud technologies can sound like an oxymoron, however, when Canonical MicroClouds meet the real-time kernel feature in Ubuntu, it is possible to get a deterministic behaviour in virtualised workloads. In other words, this allows the cloud in question to accurately deliver on its latency requirements, even if the underlying host is under stress, making it an effective solution for edge deployments with strict latency expectations. MicroCloud offers two alternatives to hosting workloads, [system containers and virtual machines](https://canonical.com/blog/containerization-vs-virtualization), offering different strengths depending on the use case. In the case of real-time applications, system containers with the enabled real-time feature showed the lowest average and lowest max latency, making it the recommended choice for your workload. However, running a privileged container needs to be considered carefully. It requires additional security considerations, like ensuring that you are running trusted workloads or that no untrusted task is running as root in the container. As the benchmark shows, MicroClouds coupled with real-time kernel can leverage critical applications in industries with real-time constraints deployed with all the essential features the industry seeks in cloud technologies. [This whitepaper](https://ubuntu.com/engage/guide-to-micro-clouds) is a good starting point if you’d like to learn more about MicroClouds in edge cloud environments. If you’re curious about the real-time kernel, this [blog series](https://ubuntu.com/blog/what-is-real-time-linux-i) or the [webinar introducing real-time Linux](https://ubuntu.com/engage/an-introduction-to-real-time-linux-part-i) is for you. ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical.com/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### Introducing MicroCloud Cluster Manager](https://canonical.com/blog/introducing-microcloud-cluster-manager) Canonical introduces the beta release of MicroCloud Cluster Manager, a new way to discover, organize, and operate your MicroCloud environments from a single, unified interface. [Miona Aleksic](https://canonical.com/blog/author/mionaalex) 20 March 2026 [### How to set up a micro lab: four principles for a reliable homelab](https://canonical.com/blog/how-to-set-up-a-reliable-homelab) After over a decade of running a homelab, I have learned a few difficult lessons. Although it begins as a “lab,” you inevitably end up with something you want to keep. If a... [Jake Nabasny](https://canonical.com/blog/author/slapcat) 17 March 2026 [### Native integration available between Canonical LXD and HPE Alletra MP B10000](https://canonical.com/blog/canonical_lxd_hpe_alletra) Native integration available between Canonical LXD and HPE Alletra MP B10000. The integration combines efficient open source virtualization with high performance,... [Miona Aleksic](https://canonical.com/blog/author/mionaalex) 15 December 2025 [### Native integration now available for Pure Storage and Canonical LXD](https://canonical.com/blog/lxd-pure-storage) Canonical, the company behind Ubuntu, and Pure Storage, the IT pioneer delivering enterprise-grade all-flash storage, have partnered to introduce a native integration between... [Canonical](https://canonical.com/blog/author/canonical) 25 June 2025