Ubuntu Explained: How to ensure security and stability in cloud instances—part 2

Ubuntu updates: best practices for updating your instance

You probably know that it is important to apply security updates. You may not be as clear on the details of how to do that. We are going to explain best practices for applying Ubuntu updates to single instances and what the built-in unattended-upgrades tool does and does not do.

This blog is the second in a three-part series by John Chittum (Engineering Manager, CPC) and Aaron Whitehouse (Senior Public Cloud Enablement Director). In the first part we covered the philosophy of Ubuntu’s releases, the archive’s structure and how packages are updated. That background will help you understand what we cover below, so please read that first before continuing. The next blog will extend the concepts in this blog to give best practices for applying updates to multiple Ubuntu instances. Update: Part 3 is now online here.

Ubuntu is a Debian-based Linux distribution which uses two primary package types: snaps and Debian (.deb) packages. Snaps have their own update mechanisms, so this blog series focuses only on the Debian package updates.

At its core, updating packages on an Ubuntu server is straightforward. You simply need to achieve the following:

1. download the latest package indexes, showing what has been updated;
2. download the updated packages;
3. install the updated packages; and 
4. restart anything that is still using the old versions of the packages.

Updating packages interactively

If you are logged into your Ubuntu server and want to update the packages to the latest version, you can do this with APT. This would look something like (as root or an elevated user):

apt update   # Downloads the latest package indexes
apt upgrade  # Downloads and installs any updated packages or new dependencies

This will download the package indexes, then download any updated packages and install them.

This will install updated packages from any enabled repositories (see Part 1 of this series for more information), not just security updates. Installing all updates makes sense, as you are running commands interactively and are able to see what is changing and watch for any ill effects. You can view or change your APT repository configuration in /etc/apt/sources.list and /etc/apt/sources.list.d/.

Running commands manually is fine, but most people want to apply security updates regularly. That requires setting up your server in a way that will install security updates automatically, without you sitting at the keyboard.

Updating packages automatically with unattended-upgrades

By default, we configure Ubuntu server to download and install any new security updates (and only security updates) every day. We also do this for public cloud instances. We do this using a tool called unattended-upgrades.

You can see an instance’s current unattended-upgrade status using recent versions of the pro client by running pro api u.unattended_upgrades.status.v1:

{
 "apt_periodic_job_enabled": true,
 "package_lists_refresh_frequency_days": 1,
 "systemd_apt_timer_enabled": true,
 "unattended_upgrades_allowed_origins": [
   "${distro_id}:${distro_codename}",
   "${distro_id}:${distro_codename}-security",
   "${distro_id}ESMApps:${distro_codename}-apps-security",
   "${distro_id}ESM:${distro_codename}-infra-security"
 ],
 "unattended_upgrades_disabled_reason": null,
 "unattended_upgrades_frequency_days": 1,
 "unattended_upgrades_last_run": null,
 "unattended_upgrades_running": true
}

This summarises information from a number of different places on the system. Some key points are as follows:

1. Settings are contained in /etc/apt/apt.conf.d/50unattended-upgrades and /etc/apt/apt.conf.d/20auto-upgrades on an Ubuntu 22.04 instance.
2. A systemd.daily unit runs unattended-upgrades each day using these settings.
3. By default, unattended-upgrades only checks the release (the frozen component), security, and the security portion of ESM. ESM stands for Expanded Security Maintenance and is included in Ubuntu Pro.

You can see more detail on why certain pockets are enabled by default if you read /etc/apt/apt.conf.d/50unattended-upgrades.

Users may disable unattended-upgrades by setting APT::Periodic::Unattended-Upgrade “0”; in /etc/apt/apt.conf.d/20auto-upgrades.

Unattended-upgrades is relatively simple, but provides a good default position for single-node Ubuntu instances from a security perspective. It ensures the vast number of Ubuntu instances in the world receive security fixes automatically in a very timely manner. However, the unattended-upgrades tool runs independently on each single node. That may mean it is not the best tool for the job if you are administering an entire fleet. We will cover best practices for updating multiple nodes in the next part of this blog series. There is no monitoring built into unattended-upgrades or APT. If APT ends up in a broken state, then unattended-upgrades will stop working. You should therefore either add APT tests into your monitoring or run apt update and apt upgrade manually on a periodic basis.

Required reboots and restarting services

Even if you install a new, secure version of a package onto the disk, you can still be running the old, insecure version in memory. The easiest way to ensure that you are only running the updated versions of packages is to reboot the instance, but this of course means disrupting your workloads.

You can tell whether or not any of the updates you have installed have signalled that an update is required by running pro system reboot-required (or use the API endpoint if doing so programmatically). If so, you can normally see which packages signalled the reboot by looking in /var/run/reboot-required.pkgs (it will often be packages like the Linux kernel, libc or dbus).

If a restart is inconvenient, in many cases you can restart individual services instead of the entire system. On recent versions of Ubuntu, a tool called needrestart will run after APT to tell you about these cases and help you do so.

Livepatching your kernel until the next reboot

The Linux kernel is at the heart of your Ubuntu system and it is not really feasible to restart it without restarting your whole system. It also receives frequent security updates. The traditional approach to staying secure is to apply kernel security updates as soon as they are available and immediately reboot the system. This has a significant impact on service availability, as it will often mean a reboot outside a scheduled maintenance window.

Canonical offers a service called Livepatch (as part of Ubuntu Pro) that can help. Livepatch security-patches the version of the kernel running in memory to keep you secure until you reboot in your next scheduled maintenance window. When using Livepatch you still should be applying any available security updates to the kernel packages on disk whenever they become available as normal. The benefit is that you are taking less of a security risk to continue running the old version of the kernel in memory until your next reboot.

Cloud-optimised rolling kernels

Ubuntu improves the user experience on public clouds by maintaining cloud-optimised kernels for our public cloud partners (see a blog on this here). This includes boot time, security and enablement of the latest cloud features. By default, kernels on Ubuntu are normally installed via metapackages, such as linux-generic, linux-generic-hwe, linux-azure, linux-aws, etc. These metapackages point to specific kernel images.

You can find further information on the kernel release cycle and hardware enablement (HWE) kernels here. Importantly, the default cloud-optimised kernels on most of our public cloud partners’ platforms are based on HWE kernels. This means that they will “roll” to a new kernel version approximately every six months for the first two years of the release. It will then stay on the version matching the initial kernel in the next Ubuntu LTS release for the remainder of the release’s support lifetime. As an example, Ubuntu 20.04 LTS started with 5.4, rolled through 5.9 and 5.11 and settled on 5.15.

We use HWE kernels where we have consulted with the public cloud and believe this default to be the best balance for most of their customers. Rolling the kernel, however, can cause issues for people using tools that rely on very specific kernel versions. If you do not wish to use the HWE kernels on a cloud image, you must downgrade your kernel. You can do this by installing the linux-${CLOUD}-lts-${UBUNTU_VERSION} metapackage, e.g. linux-azure-lts-22.04, and removing the more recent specific kernel packages. Your kernel will then stay on the GA version of the kernel that was released with that LTS release (e.g. 5.4 in Ubuntu 20.04 LTS). These kernels receive security maintenance for the life of the release (including ESM for Ubuntu Pro customers), but they may not have enablement for the latest cloud features or instance types. 

Conclusion

If you are running Ubuntu in production, it is crucial that you apply available security updates to your instances. This is straightforward to do manually on demand. In our regular Ubuntu server images, we set unattended-upgrades to download and install available security updates every day, which is a sensible default to ensure that the Ubuntu instances in the world stay secure. While they can never be infallible, we set out in the first part of this blog series some of the steps that we take to minimise the risk that a security update will have negative impacts on our users. 

Even if you are automatically downloading and installing updates, it is possible that you are still using the old, insecure version in memory. The foolproof way to ensure you are running updated versions is to reboot, but this impacts availability. You can use tools like the reboot-required API endpoint, needrestart and Livepatch to reduce unplanned downtime.

Finally, by default many of our public cloud images use rolling kernels. This means that the kernel rolls to a later kernel version approximately every six months for the first two years. This gives you access to the latest cloud features, but can cause issues with tools that require specific kernel versions. In those cases, you can downgrade to the GA kernel.

There is more ability to balance stability and security in your specific environment when you are administering multiple Ubuntu instances. We will cover some of those in the final part of this blog series, so stay tuned for future posts. Update: Part 3 is now online here.

Further reading:

• Ubuntu Explained: How to ensure security and stability in cloud instances—part 1
• Enhancing the Ubuntu Experience on Azure: Introducing Ubuntu Pro Updates Awareness
• Securing open source software dependencies in the public cloud