Your submission was sent successfully! Close

Thank you for signing up for our newsletter!
In these regular emails you will find the latest updates from Canonical and upcoming events where you can meet our team.Close

Thank you for contacting our team. We will be in touch shortly.Close

  1. Blog
  2. Article

Rui Vasconcelos
on 24 June 2020

Demystifying Kubeflow pipelines: Data science workflows on Kubernetes – Part 1

Kubeflow Pipelines are a great way to build portable, scalable machine learning workflows. It is one part of a larger Kubeflow ecosystem that aims to reduce the complexity and time involved with training and deploying machine learning models at scale.

In this blog series, we demystify Kubeflow pipelines and showcase this method to produce reusable and reproducible data science. 🚀

We go over why Kubeflow brings the right standardization to data science workflows, followed by how this can be achieved through Kubeflow pipelines.

In part 2, we will get our hands dirty! We’ll make use of the Fashion MNIST dataset and the Basic classification with Tensorflow example, and take a step-by-step approach to turn the example model into a Kubeflow pipeline so that you can do the same.

Why use Kubeflow?

A machine learning workflow can involve many steps and keeping all these steps in a set of notebooks or scripts is hard to maintain, share and collaborate on, which leads to large amounts of “Hidden Technical Debt in Machine Learning Systems”.

In addition, it is typical that these steps are run on different systems. In an initial phase of experimentation, a data scientist will work at a developer workstation or an on-prem training rig, training at scale will typically happen in a cloud environment (private, hybrid, or public), while inference and distributed training often happens at the Edge.

Containers provide the right encapsulation, avoiding the need for debugging every time a developer changes the execution environment, and Kubernetes brings scheduling and orchestration of containers into the infrastructure.

However, managing ML workloads on top of Kubernetes is still a lot of specialized operations work which we don’t want to add to the data scientist’s role. Kubeflow bridges this gap between AI workloads and Kubernetes, making MLOps more manageable.

What are Kubeflow Pipelines?

Kubeflow pipelines are one of the most important features of Kubeflow and promise to make your AI experiments reproducible, composable, i.e. made of interchangeable components, scalable, and easily shareable.

A pipeline is a codified representation of a machine learning workflow, analogous to the sequence of steps described in the first image, which includes components of the workflow and their respective dependencies. More specifically, a pipeline is a directed acyclic graph (DAG) with a containerized process on each node, which runs on top of argo.

Each pipeline component, represented as a block, is a self-contained piece of code, packaged as a Docker image. It contains inputs (arguments) and outputs and performs one step in the pipeline. In the example pipeline, above, the transform_data step requires arguments that are produced as an output of the extract_data and of the generate_schema steps, and its outputs are dependencies for train_model

Your ML code is wrapped into components, where you can:

  • Specify parameters – which become available to edit in the dashboard and configurable for every run.
  • Attach persistent volumes – without adding persistent volumes, we would lose all the data if our notebook was terminated for any reason. 
  • Specify artifacts to be generated – graphs, tables, selected images, models – which end up conveniently stored on the Artifact Store, inside the Kubeflow dashboard.

Finally, when you run the pipeline, each container will now be executed throughout the cluster, according to Kubernetes scheduling, taking dependencies into consideration.

This containerized architecture makes it simple to reuse, share, or swap out components as your workflow changes, which tends to happen.

After running the pipeline, you are able to explore the results on the pipelines interfaces, debug, tweak parameters, and run experiments by executing the pipeline with different parameters or data sources.

Kubeflow dashboard – observing logs from pipeline run.

This way, experiments allow you to save and compare the results of your runs, keep your best performing models, and version control your workflow.

That’s it for this part!

In Part 2, we will create the pipeline you see on the last image using the Fashion MNIST dataset and the Basic classification with Tensorflow example, taking a step-by-step approach to turn the example model into a Kubeflow pipeline, so that you can do the same to your own models.

Further reading

To keep on learning and experimenting with Kubeflow and Kubeflow Pipelines:

  1. Watch the webinar on building Kubeflow Pipelines.
  2. Play with Kubeflow examples on GitHub.
  3. Read our Kubernetes for data science: meet Kubeflow post.
  4. Visit

How to try Kubeflow?

To deploy Kubeflow on your Windows, macOS, or Ubuntu machine follow one of these:

  1. The tutorial here
  2. The youtube video here

This blog series is part of the joint collaboration between Canonical and Manceps.
Visit our AI consulting and delivery services page to know more.

Related posts

Andreea Munteanu
22 March 2024

A deep dive into Kubeflow pipelines 

AI Article

Widely adopted by both developers and organisations, Kubeflow is an MLOps platform that runs on Kubernetes and automates machine learning (ML) workloads. It covers the entire ML lifecycle, enabling data scientists and machine learning engineers to develop and deploy ML models. Kubeflow is designed as a suite of leading open source project ...

11 March 2024

Large Language Models (LLMs) Retrieval Augmented Generation (RAG) using Charmed OpenSearch

AI Article

This article guides you on leveraging Charmed OpenSearch to maintain a relevant and up-to-date LLM application. ...

Andreea Munteanu
19 February 2024

Edge AI: what, why and how with open source

Ubuntu Article

Edge AI is transforming the way that devices interact with data centres, challenging organisations to stay up to speed with the latest innovations. From AI-powered healthcare instruments to autonomous vehicles, there are plenty of use cases that benefit from artificial intelligence on edge devices. This blog will dive into the topic, capt ...