--- title: 'Demystifying Kubeflow pipelines: Data science workflows on Kubernetes – Part 1' description: Kubeflow pipelines are core to Kubeflow! In this blog, we demystify Kubeflow pipelines and showcase how to produce reusable and reproducible data science. url: https://canonical.com/blog/data-science-workflows-on-kubernetes-with-kubeflow-pipelines-part-1?format=md --- 1. [Blog](https://canonical.com/blog) 2. Article --- [Rui Vasconcelos](https://canonical.com/blog/author/ruivasconcelos "More about Rui Vasconcelos") 24 June 2020 # Demystifying Kubeflow pipelines: Data science workflows on Kubernetes – Part 1 [AI](https://canonical.com/blog/tag/ai) [deep learning](https://canonical.com/blog/tag/deep-learning) [Kubeflow](https://canonical.com/blog/tag/kubeflow) [machine learning](https://canonical.com/blog/tag/machine-learning) [MLOps](https://canonical.com/blog/tag/mlops) [pipeline](https://canonical.com/blog/tag/pipeline) --- Share the article [Kubeflow Pipelines](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/) are a great way to build portable, scalable machine learning workflows. It is one part of a larger [Kubeflow](http://www.kubeflow.org) ecosystem that aims to reduce the complexity and time involved with training and deploying machine learning models at scale. In this blog series, we demystify Kubeflow pipelines and showcase this method to produce reusable and reproducible data science. 🚀 We go over why Kubeflow brings the right standardization to data science workflows, followed by how this can be achieved through Kubeflow pipelines. In part 2, we will get our hands dirty! We’ll make use of the Fashion MNIST dataset and the [Basic classification with Tensorflow](https://www.tensorflow.org/tutorials/keras/classification) example, and take a step-by-step approach to turn the example model into a Kubeflow pipeline so that you can do the same. ### Why use Kubeflow? A machine learning workflow can involve many steps and keeping all these steps in a set of notebooks or scripts is hard to maintain, share and collaborate on, which leads to large amounts of “[Hidden Technical Debt in Machine Learning Systems](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf)”. In addition, it is typical that these steps are run on different systems. In an initial phase of experimentation, a data scientist will work at a developer workstation or an on-prem training rig, training at scale will typically happen in a cloud environment (private, hybrid, or public), while inference and distributed training often happens at the Edge. Containers provide the right encapsulation, avoiding the need for debugging every time a developer changes the execution environment, and [Kubernetes](https://kubernetes.io/) brings scheduling and orchestration of containers into the infrastructure. However, managing ML workloads on top of Kubernetes is still a lot of specialized operations work which we don’t want to add to the data scientist’s role. Kubeflow [bridges this gap](https://ubuntu.com/blog/kubernetes-for-data-science-meet-kubeflow) between AI workloads and Kubernetes, making MLOps more manageable. ### What are Kubeflow Pipelines? [Kubeflow pipelines](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/) are one of the most important features of Kubeflow and promise to make your AI experiments reproducible, composable, i.e. made of interchangeable components, scalable, and easily shareable. A pipeline is a codified representation of a machine learning workflow, analogous to the sequence of steps described in the first image, which includes components of the workflow and their respective dependencies. More specifically, a pipeline is a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) with a containerized process on each node, which runs on top of [argo](https://argoproj.github.io/). Each pipeline component, represented as a block, is a self-contained piece of code, packaged as a [Docker image](https://docs.docker.com/get-started/). It contains inputs (arguments) and outputs and performs one step in the pipeline. In the example pipeline, above, the *transform\_data* step requires arguments that are produced as an output of the *extract\_data* and of the *generate\_schema* steps, and its outputs are dependencies for *train\_model*. Your ML code is wrapped into components, where you can: * Specify [parameters](https://www.kubeflow.org/docs/pipelines/sdk/parameters/#:~:text=You%20pipeline%20function%20should%20have,instantiate%20them%20and%20create%20tasks.) – which become available to edit in the dashboard and configurable for every run. * Attach [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) – without adding persistent volumes, we would lose all the data if our notebook was terminated for any reason. * Specify [artifacts](https://www.kubeflow.org/docs/pipelines/overview/concepts/output-artifact/) to be generated – graphs, tables, selected images, models – which end up conveniently stored on the Artifact Store, inside the Kubeflow dashboard. Finally, when you run the pipeline, each container will now be executed throughout the cluster, according to Kubernetes scheduling, taking dependencies into consideration. This containerized architecture makes it simple to reuse, share, or swap out components as your workflow changes, which tends to happen. After running the pipeline, you are able to explore the results on the [pipelines interfaces](https://www.kubeflow.org/docs/pipelines/overview/interfaces/#user-interface-ui), debug, tweak parameters, and run experiments by executing the pipeline with different parameters or data sources. Kubeflow dashboard – observing logs from pipeline run. This way, experiments allow you to save and compare the results of your runs, keep your best performing models, and version control your workflow. ### That’s it for this part! In [Part 2](https://ubuntu.com/blog/tag/ai), we will create the pipeline you see on the last image using the Fashion MNIST dataset and the [Basic classification with Tensorflow](https://www.tensorflow.org/tutorials/keras/classification) example, taking a step-by-step approach to turn the example model into a Kubeflow pipeline, so that you can do the same to your own models. ### Further reading To keep on learning and experimenting with Kubeflow and Kubeflow Pipelines: 1. Watch the [webinar](https://www.brighttalk.com/webcast/6793/424551) on building Kubeflow Pipelines. 2. Play with [Kubeflow examples](https://github.com/kubeflow/examples) on GitHub. 3. Read our [Kubernetes for data science: meet Kubeflow](https://ubuntu.com/blog/kubernetes-for-data-science-meet-kubeflow) post. 4. Visit [ubuntu.com/kubeflow](http://www.ubuntu.com/kubeflow) ### How to try Kubeflow? To deploy Kubeflow on your Windows, macOS, or Ubuntu machine follow one of these: 1. The tutorial [here](https://ubuntu.com/tutorials/deploy-kubeflow-ubuntu-windows-mac#1-overview) 2. The youtube video [here](https://www.youtube.com/watch?v=KPEGKKNB63Q) *This blog series is part of the joint collaboration between Canonical and [Manceps](https://www.manceps.com/).* *Visit our [AI consulting and delivery](https://ubuntu.com/kubeflow/consulting) services page to know more.* ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical.com/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### Arduino® VENTUNO™ Q is available for pre-order with Ubuntu pre-installed](https://canonical.com/blog/arduino-ventuno-q-is-available-for-pre-order-with-ubuntu-pre-installed) London, UK – August 25, 2026 – Following our initial collaboration announcement in March 2026, Canonical and Arduino (a subsidiary of Qualcomm Technologies, Inc.) are excited... [Canonical](https://canonical.com/blog/author/canonical) 25 August 2026 [### A look into Ubuntu Core 26: Building a local AI inference appliance in a virtual machine](https://canonical.com/blog/ubuntu-core-26-ai-box) Welcome to this blog series which explores innovative uses of Ubuntu Core. Throughout this series, Canonical’s Engineers will show what you can build with this Core 26 release,... [Gabriel Aguiar Noury](https://canonical.com/blog/author/g-aguiar-noury) 16 June 2026 [### AI at the edge: simplifying infrastructure with Cisco and Canonical](https://canonical.com/blog/ai-at-the-edge-simplifying-infrastructure-with-cisco-and-canonical) Legacy infrastructure was not designed for the requirements of the AI era. While large-scale model training remains centralized in data centers, test-time inference is rapidly... [Pedro Lazzarotto](https://canonical.com/blog/author/pedro-lazzarotto) 11 June 2026 [### What is RDMA over Converged Ethernet (RoCE)?](https://canonical.com/blog/what-is-rdma-over-converged-ethernet-roce) Previous articles walked through RDMA (Remote Direct Memory Access) as a programming model and InfiniBand as the fabric that was built around it. Both led to the same... [Benjamin Ryzman](https://canonical.com/blog/author/benjaminryzman) 9 June 2026