Skip to main content

Your submission was sent successfully! Close

Thank you for signing up for our newsletter!
In these regular emails you will find the latest updates from Canonical and upcoming events where you can meet our team.Close

Thank you for contacting us. A member of our team will be in touch shortly. Close

  1. Blog
  2. Article

Andreea Munteanu
on 19 March 2025


The world of edge AI is rapidly transforming how devices and data centers work together. Imagine healthcare tools powered by AI, or self-driving vehicles making real-time decisions. These advancements rely on bringing AI directly to edge devices. However, building a robust architecture for diverse edge environments presents significant hurdles.

This blog introduces our new reference architecture, designed to simplify edge AI deployment. It outlines a clear solution, highlights key design considerations, and explains the selection of each component, with a strong focus on our collaboration with NVIDIA.

The challenges of edge deployment

Managing edge solutions presents unique challenges. Picture vast networks of devices, scattered across diverse locations. This scale demands exceptional scalability. As the number of devices grows, deployment, updates, and monitoring must remain streamlined.

Moreover, edge software often needs to operate independently. Many deployments face limited or no remote connectivity. Therefore, devices must be capable of self-recovery when issues arise.

Security is another crucial factor. Edge devices are susceptible to both physical and software attacks. Strong protective measures and consistent security updates are essential.

Finally, resource limitations pose a significant hurdle. Edge devices typically have constrained RAM, necessitating efficient resource utilization.

Open source and NVIDIA: a powerful combination

Edge devices, equipped with NVIDIA GPUs, require a lightweight and secure operating system and appropriate GPU drivers. The model is then deployed across multiple devices, with continuous retraining and updates. Effective model deployment and maintenance in offline environments are essential due to diverse deployment locations.

This architecture emphasizes the use of open source solutions alongside NVIDIA accelerated computing for optimal performance and flexibility. The solution provides guidance for both the training and inference environments, including hardware and software components, tool versions, and integration considerations.

What makes edge AI a reality: inference and training

The inference environment is where the magic happens, where edge devices actively process data. It’s designed for flexibility, running on diverse hardware as long as minimum requirements are met. To boost performance, optimization engines like NVIDIA TensorRT are employed, especially for models built with popular frameworks. Quantization further refines models, reducing their size and power needs. Lightweight, secure operating systems like Ubuntu Core provide the foundation, with features like TPM-based encryption and seamless NVIDIA driver integration.

Meanwhile, the training environment is where models are conceived and honed. Secure software distribution through Snaps, powered by Ubuntu Core’s robust security, is key. Data from edge devices is efficiently managed by Apache Kafka, feeding into the retraining process. MLOps platforms like Kubeflow and MLflow automate these pipelines, while tools like Juju and Charms simplify the orchestration of complex components.

A comprehensive stack for edge AI success

Organizations can successfully implement AI at the edge by carefully considering hardware and software compatibility and their ability to deploy and maintain ML models. Open source solutions like Charmed Kubeflow and KServe are well-suited for these scenarios, allowing organizations to use a consistent software vendor across data centers and edge environments. Canonical provides a comprehensive stack for easy and secure edge AI deployment and management.

Get started today

Dive deeper into our Reference Architecture and discover how our solutions, powered by NVIDIA, can transform your approach to distributed intelligence. Learn more about the components and capabilities discussed and explore how they can be applied to your specific use case.Ready to unlock the potential of Edge AI for your organization?

Download it now

Related posts


Andreea Munteanu
19 March 2025

Building optimized LLM chatbots with Canonical and NVIDIA

AI Downloads

The landscape of generative AI is rapidly evolving, and building robust, scalable large language model (LLM) applications is becoming a critical need for many organizations. Canonical, in collaboration with NVIDIA, is excited to introduce a reference architecture designed to streamline and optimize the creation of powerful LLM chatbots. T ...


Andreea Munteanu
20 March 2025

Accelerating AI with open source machine learning infrastructure

AI Article

The landscape of artificial intelligence is rapidly evolving, demanding robust and scalable infrastructure. To meet these challenges, we’ve developed a comprehensive reference architecture (RA) that leverages the power of open-source tools and cutting-edge hardware. This architecture, built on Canonical’s MicroK8s and Charmed Kubeflow, ru ...


Canonical
18 March 2025

Ubuntu now officially supports NVIDIA Jetson: powering the future of AI at the edge

AI Article

Canonical announces the General Availability of Ubuntu for the NVIDIA® Jetson Orin™ for edge AI and robotics, bringing optimized performance, out-of-the-box compatibility, and an easy pathway to high-performance AI solutions for AI developers everywhere. Canonical, the publisher of Ubuntu, announced official support for the NVIDIA Jetson ...