Your submission was sent successfully! Close

Thank you for signing up for our newsletter!
In these regular emails you will find the latest updates from Canonical and upcoming events where you can meet our team.Close

Thank you for contacting our team. We will be in touch shortly.Close

  1. Blog
  2. Article

Aaron Whitehouse
on 21 November 2023


Ubuntu updates: best practices for updating your instance

You probably know that it is important to apply security updates. You may not be as clear on the details of how to do that. We are going to explain best practices for applying Ubuntu updates to single instances and what the built-in unattended-upgrades tool does and does not do.

This blog is the second in a three-part series by John Chittum (Engineering Manager, CPC) and Aaron Whitehouse (Senior Public Cloud Enablement Director). In the first part we covered the philosophy of Ubuntu’s releases, the archive’s structure and how packages are updated. That background will help you understand what we cover below, so please read that first before continuing. The next blog will extend the concepts in this blog to give best practices for applying updates to multiple Ubuntu instances. Update: Part 3 is now online here.

Ubuntu is a Debian-based Linux distribution which uses two primary package types: snaps and Debian (.deb) packages. Snaps have their own update mechanisms, so this blog series focuses only on the Debian package updates.

At its core, updating packages on an Ubuntu server is straightforward. You simply need to achieve the following:

  1. download the latest package indexes, showing what has been updated;
  2. download the updated packages;
  3. install the updated packages; and 
  4. restart anything that is still using the old versions of the packages.

Updating packages interactively

If you are logged into your Ubuntu server and want to update the packages to the latest version, you can do this with APT. This would look something like (as root or an elevated user):

apt update   # Downloads the latest package indexes
apt upgrade  # Downloads and installs any updated packages or new dependencies

This will download the package indexes, then download any updated packages and install them.

Terminal window showing updating an Ubuntu instance with APT

This will install updated packages from any enabled repositories (see Part 1 of this series for more information), not just security updates. Installing all updates makes sense, as you are running commands interactively and are able to see what is changing and watch for any ill effects. You can view or change your APT repository configuration in /etc/apt/sources.list and /etc/apt/sources.list.d/.

Running commands manually is fine, but most people want to apply security updates regularly. That requires setting up your server in a way that will install security updates automatically, without you sitting at the keyboard.

Updating packages automatically with unattended-upgrades

By default, we configure Ubuntu server to download and install any new security updates (and only security updates) every day. We also do this for public cloud instances. We do this using a tool called unattended-upgrades.

You can see an instance’s current unattended-upgrade status using recent versions of the pro client by running pro api u.unattended_upgrades.status.v1:

{
"apt_periodic_job_enabled": true,
"package_lists_refresh_frequency_days": 1,
"systemd_apt_timer_enabled": true,
"unattended_upgrades_allowed_origins": [
   "${distro_id}:${distro_codename}",
   "${distro_id}:${distro_codename}-security",
   "${distro_id}ESMApps:${distro_codename}-apps-security",
   "${distro_id}ESM:${distro_codename}-infra-security"
],
"unattended_upgrades_disabled_reason": null,
"unattended_upgrades_frequency_days": 1,
"unattended_upgrades_last_run": null,
"unattended_upgrades_running": true
}

This summarises information from a number of different places on the system. Some key points are as follows:

  1. Settings are contained in /etc/apt/apt.conf.d/50unattended-upgrades and /etc/apt/apt.conf.d/20auto-upgrades on an Ubuntu 22.04 instance.
  2. A systemd.daily unit runs unattended-upgrades each day using these settings.
  3. By default, unattended-upgrades only checks the release (the frozen component), security, and the security portion of ESM. ESM stands for Expanded Security Maintenance and is included in Ubuntu Pro.

You can see more detail on why certain pockets are enabled by default if you read /etc/apt/apt.conf.d/50unattended-upgrades.

Users may disable unattended-upgrades by setting APT::Periodic::Unattended-Upgrade “0”; in /etc/apt/apt.conf.d/20auto-upgrades.

Unattended-upgrades is relatively simple, but provides a good default position for single-node Ubuntu instances from a security perspective. It ensures the vast number of Ubuntu instances in the world receive security fixes automatically in a very timely manner. However, the unattended-upgrades tool runs independently on each single node. That may mean it is not the best tool for the job if you are administering an entire fleet. We will cover best practices for updating multiple nodes in the next part of this blog series. There is no monitoring built into unattended-upgrades or APT. If APT ends up in a broken state, then unattended-upgrades will stop working. You should therefore either add APT tests into your monitoring or run apt update and apt upgrade manually on a periodic basis.

Required reboots and restarting services

Even if you install a new, secure version of a package onto the disk, you can still be running the old, insecure version in memory. The easiest way to ensure that you are only running the updated versions of packages is to reboot the instance, but this of course means disrupting your workloads.

You can tell whether or not any of the updates you have installed have signalled that an update is required by running pro system reboot-required (or use the API endpoint if doing so programmatically). If so, you can normally see which packages signalled the reboot by looking in /var/run/reboot-required.pkgs (it will often be packages like the Linux kernel, libc or dbus).

If a restart is inconvenient, in many cases you can restart individual services instead of the entire system. On recent versions of Ubuntu, a tool called needrestart will run after APT to tell you about these cases and help you do so.

Livepatching your kernel until the next reboot

The Linux kernel is at the heart of your Ubuntu system and it is not really feasible to restart it without restarting your whole system. It also receives frequent security updates. The traditional approach to staying secure is to apply kernel security updates as soon as they are available and immediately reboot the system. This has a significant impact on service availability, as it will often mean a reboot outside a scheduled maintenance window.

Canonical offers a service called Livepatch (as part of Ubuntu Pro) that can help. Livepatch security-patches the version of the kernel running in memory to keep you secure until you reboot in your next scheduled maintenance window. When using Livepatch you still should be applying any available security updates to the kernel packages on disk whenever they become available as normal. The benefit is that you are taking less of a security risk to continue running the old version of the kernel in memory until your next reboot.

Cloud-optimised rolling kernels

Ubuntu improves the user experience on public clouds by maintaining cloud-optimised kernels for our public cloud partners (see a blog on this here). This includes boot time, security and enablement of the latest cloud features. By default, kernels on Ubuntu are normally installed via metapackages, such as linux-generic, linux-generic-hwe, linux-azure, linux-aws, etc. These metapackages point to specific kernel images.

You can find further information on the kernel release cycle and hardware enablement (HWE) kernels here. Importantly, the default cloud-optimised kernels on most of our public cloud partners’ platforms are based on HWE kernels. This means that they will “roll” to a new kernel version approximately every six months for the first two years of the release. It will then stay on the version matching the initial kernel in the next Ubuntu LTS release for the remainder of the release’s support lifetime. As an example, Ubuntu 20.04 LTS started with 5.4, rolled through 5.9 and 5.11 and settled on 5.15.

We use HWE kernels where we have consulted with the public cloud and believe this default to be the best balance for most of their customers. Rolling the kernel, however, can cause issues for people using tools that rely on very specific kernel versions. If you do not wish to use the HWE kernels on a cloud image, you must downgrade your kernel. You can do this by installing the linux-${CLOUD}-lts-${UBUNTU_VERSION} metapackage, e.g. linux-azure-lts-22.04, and removing the more recent specific kernel packages. Your kernel will then stay on the GA version of the kernel that was released with that LTS release (e.g. 5.4 in Ubuntu 20.04 LTS). These kernels receive security maintenance for the life of the release (including ESM for Ubuntu Pro customers), but they may not have enablement for the latest cloud features or instance types. 

Conclusion

If you are running Ubuntu in production, it is crucial that you apply available security updates to your instances. This is straightforward to do manually on demand. In our regular Ubuntu server images, we set unattended-upgrades to download and install available security updates every day, which is a sensible default to ensure that the Ubuntu instances in the world stay secure. While they can never be infallible, we set out in the first part of this blog series some of the steps that we take to minimise the risk that a security update will have negative impacts on our users. 

Even if you are automatically downloading and installing updates, it is possible that you are still using the old, insecure version in memory. The foolproof way to ensure you are running updated versions is to reboot, but this impacts availability. You can use tools like the reboot-required API endpoint, needrestart and Livepatch to reduce unplanned downtime.

Finally, by default many of our public cloud images use rolling kernels. This means that the kernel rolls to a later kernel version approximately every six months for the first two years. This gives you access to the latest cloud features, but can cause issues with tools that require specific kernel versions. In those cases, you can downgrade to the GA kernel.

There is more ability to balance stability and security in your specific environment when you are administering multiple Ubuntu instances. We will cover some of those in the final part of this blog series, so stay tuned for future posts. Update: Part 3 is now online here.

Further reading:

Related posts


Aaron Whitehouse
24 November 2023

Ubuntu Explained: How to ensure security and stability in cloud instances—part 3

Cloud and server Article

Applying updates across a fleet of multiple Ubuntu instances is a balance of security and service uptime. We explore best practices to maximise stability. ...


Oliver Smith
17 May 2024

Migrating from CentOS to Ubuntu: a guide for system administrators and DevOps

Cloud and server Article

CentOS 7 is on track to reach its end-of-life (EoL) on June 30, 2024. Post this date, the CentOS Project will cease to provide updates or support, including vital security patches. Moving away from the RHEL-based ecosystem might appear daunting, but if you’re considering Ubuntu the switch can be both straightforward and economically viabl ...


Hugo Huang
13 May 2024

Ubuntu Pro 24.04 LTS Lands on Google Cloud: Power Up Your Cloud Experience

Cloud and server Ubuntu Pro

Exciting news for cloud enthusiasts and developers! Ubuntu Pro 24.04 LTS (Noble Numbat) is now available on Google Cloud, bringing a robust and secure platform for your cloud workloads. This latest Long Term Support release from Canonical offers a wealth of features and enhancements, making it the perfect choice for building and deploying ...