--- title: 'Building Kubeflow pipelines: Data science workflows on Kubernetes – Part 2' description: Kubeflow pipelines are core to Kubeflow! In this blog, we demystify Kubeflow pipelines and showcase how to convert a TensorFlow example into an ML pipeline. url: https://canonical.com/blog/data-science-workflows-on-kubernetes-with-kubeflow-pipelines-part-2?format=md --- 1. [Blog](https://canonical.com/blog) 2. Article --- [Rui Vasconcelos](https://canonical.com/blog/author/ruivasconcelos "More about Rui Vasconcelos") 2 July 2020 # Building Kubeflow pipelines: Data science workflows on Kubernetes – Part 2 [AI](https://canonical.com/blog/tag/ai) [deep learning](https://canonical.com/blog/tag/deep-learning) [Kubeflow](https://canonical.com/blog/tag/kubeflow) [machine learning](https://canonical.com/blog/tag/machine-learning) [MLOps](https://canonical.com/blog/tag/mlops) [pipeline](https://canonical.com/blog/tag/pipeline) --- Share the article *This blog series is part of the joint collaboration between Canonical and [Manceps](https://www.manceps.com/).* *Visit our [AI consulting and delivery](https://ubuntu.com/kubeflow/consulting) services page to know more.* ### **Introduction** [Kubeflow Pipelines](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/) are a great way to build portable, scalable machine learning workflows. It is a part of the [Kubeflow](http://www.kubeflow.org) project that aims to reduce the complexity and time involved with training and deploying machine learning models at scale. For more on Kubeflow, read our [Kubernetes for data science: meet Kubeflow](https://ubuntu.com/blog/kubernetes-for-data-science-meet-kubeflow) post. In this blog series, we demystify Kubeflow pipelines and showcase this method to produce reusable and reproducible data science. 🚀 In [Part 1](https://www.manceps.com/articles/tutorial/data-science-workflows-on-kubernetes-with-kubeflow-pipelines-part-1), we covered WHY Kubeflow brings the right standardization to data science workflows. Now, let’s see HOW you can accomplish that with Kubeflow Pipelines. In Part 2 of this blog series, we’ll work on building your first Kubeflow Pipeline as you gain an understanding of how it’s used to deploy reusable and reproducible ML pipelines. 🚀 Now, it is time to get our hands dirty! 👨🏻‍🔬 ### **Building your first Kubeflow pipeline** In this experiment, we will make use of the fashion MNIST dataset and the [Basic classification with Tensorflow](https://www.tensorflow.org/tutorials/keras/classification) example and turn it into a Kubeflow pipeline, so you can repeat the same process with any notebook or script you already have worked on. You can follow the process of migration into the pipeline on this [Jupyter notebook](https://github.com/lildonpancho/manceps-canonical/blob/master/KF_Fashion_MNIST.ipynb). Ready? 🚀 #### **Step 1: Deploy Kubeflow and access the dashboard** If you haven’t had the opportunity to launch Kubeflow, that is ok! You can deploy Kubeflow easily using Microk8s by following the tutorial – [Deploy Kubeflow on Ubuntu, Windows and MacOS](https://ubuntu.com/tutorials/deploy-kubeflow-ubuntu-windows-mac). We recommend deploying Kubeflow on your workstation if you have a machine with 16GB of RAM or more. Otherwise, spin up a virtual machine with these resources (e.g. t2.xlarge EC2 instance) and follow the same deployment process. You can find alternative deployment options [here](https://www.manceps.com/articles/tutorial/how-to-install-kubeflow-on-various-operating-systems). #### **Step 2: Launch notebook server** Once you have access to the Kubeflow dashboard, setting up a Jupyter notebook server is fairly straightforward. You can follow the steps [here](https://www.kubeflow.org/docs/notebooks/setup/). Launch the server, wait a few seconds, and connect to it. #### **Step 3: Git clone example notebook** Once in the Notebook server, launch a new terminal from the menu on the right (New > Terminal). In the terminal, download the notebook from GitHub: ``` $ git clone https://github.com/manceps/manceps-canonical.git ``` Now, open the “KF\_Fashion\_MNIST” notebook: Jupyter notebook for this experiment – download [here](https://github.com/manceps/manceps-canonical/blob/master/KF_Fashion_MNIST.ipynb). #### **Step 4: Initiate Kubeflow pipelines SDK** Now that we’re on the same page, we can kickstart our project together in the browser. As you see, the first section is adapted from the [Basic classification with Tensorflow](https://www.tensorflow.org/tutorials/keras/classification) example. Let’s skip that and get on with converting this model into a running pipeline. To ensure access to the packages needed through your Jupyter notebook instance, begin by installing Kubeflow Pipelines SDK ([kfp](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/)) in the current userspace: ``` !pip install -q kfp --upgrade --user ``` #### **Step 5:** **Convert Python scripts to docker containers** The Kubeflow Python SDK allows you to build lightweight components by defining python functions and converting them using *[func\_to\_container\_0p](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.func_to_container_op).* To package our python code inside containers you define a standard python function that contains a logical step in your pipeline. In this case, we have defined two functions: *train* and *predict*. The *train* component will train, evaluate, and save our model.  The *predict* component takes the model and makes a prediction on an image from the test dataset. ``` # Grab an image from the test dataset Img = test_images[image_number] ``` ``` # Predict the label of the image predictions = probability_model.predict(img) ``` The code used in these components is in the second part of the [Basic classification with Tensorflow](https://www.tensorflow.org/tutorials/keras/classification) example, in the “Build the model” section. The final step in this section is to transform these functions into container components. You can do this with the *func\_to\_container\_op* method as follows. ``` train_op = comp.func_to_container_op(train, base_image='tensorflow/tensorflow:latest-gpu-py3') predict_op = comp.func_to_container_op(predict, base_image='tensorflow/tensorflow:latest-gpu-py3') ``` #### **Step 6: Define Kubeflow pipeline** Kubeflow uses Kubernetes resources which are defined using [YAML](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#describing-a-kubernetes-object) templates. Kubeflow Pipelines SDK allows you to define how your code is run, without having to manually manipulate YAML files. At compile time, Kubeflow creates a compressed YAML file that defines your pipeline. This file can later be reused or shared, making the pipeline both scalable and reproducible. Start by initiating a Kubeflow client that contains client libraries for the Kubeflow Pipelines API, allowing you to further create experimentsand runs within those experiments from the Jupyter notebook. ``` client = kfp.Client() ``` We then define the pipeline name and description, which can be visualized on the Kubeflow dashboard. Next, define the pipeline by adding the arguments that will be fed into it. In this case, define the path for where data will be written, the file where the model is to be stored, and an integer representing the index of an image in the test dataset: #### **Step 7: Create a persistent volume** One additional concept we need to add is the concept of [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). Without adding persistent volumes, we would lose all the data if our notebook was terminated for any reason. kfp allows for the creation of persistent volumes using the *[VolumeOp](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.VolumeOp)* object. *VolumeOp* parameters include: * **name** – the name displayed for the volume creation operation in the UI * **resource\_name** – name which can be referenced by other resources. * **size** – size of the volume claim * **modes** – access mode for the volume (See [Kubernetes docs](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) for more details on access mode). #### **Step 8: Define pipeline components** It is now time to define your pipeline components and dependencies. We do this with *[ContainerOp](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.ContainerOp),* an object that defines a pipeline component from a container. The *train\_op* and *predict\_op* components take arguments which were declared in the original python function. At the end of the function we attach our *VolumeOp* with a dictionary of paths and associated *Persistent Volumes* to be mounted to the container before execution. Notice that while *train\_op* is using the *vop.volume* value in the *pvolumes* dictionary, the *.pvolume* argument used by the other components ensures that the volume from the previous *ContainerOp* is used, rather than creating a new one. This inherently tells Kubeflow about our intended order of operations. Consequently, Kubeflow will only mount that volume once the previous ** has completed execution. The final *print\_prediction* component is defined somewhat differently. Here we define a container to be used and add arguments to be executed at runtime. This is done by directly using the *[ContainerOp](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.dsl.html#kfp.dsl.ContainerOp)* object*.* *ContainerOp* parameters include: * **name** – the name displayed for the component execution during runtime. * **image** – image tag for the Docker container to be used. * **pvolumes** – dictionary of paths and associated *Persistent Volumes* to be mounted to the container before execution. * **arguments** – command to be run by the container at runtime. #### **Step 9: Compile and run** Finally, this notebook compiles your pipeline code and runs it within an experiment. The name of the run and of the experiment (a group of runs) is specified in the notebook and then presented in the Kubeflow dashboard. You can now view your pipeline running in the Kubeflow Pipelines UI by clicking on the notebook link run. ### **Results** Now that the pipeline has been created and set to run, it is time to check out the results. Navigate to the Kubeflow Pipelines dashboard by clicking on the notebook link run or *Pipelines → Experiments → fasion\_mnist\_kubeflow*. The components defined in the pipeline will be displayed. As they complete the path of the data pipeline will be updated. To see the details for a component, we can click directly on the component and dig into a few tabs. Click on the logs tab to see the logs generated while running the component. Once the *echo\_result* component finishes executing, you can check the result by observing the logs for that component. It will display the class of the image being predicted, the confidence of the model on its prediction, and the actual label for the image. ### **Final thoughts** Kubeflow and Kubeflow Pipelines promise to revolutionize the way data science and operations teams handle machine learning operations (MLOps) and pipelines workflows. However, this fast-evolving technology can be challenging to keep up with. In this blog series we went through a conceptual overview, in [part 1](https://www.manceps.com/articles/tutorial/data-science-workflows-on-kubernetes-with-kubeflow-pipelines-part-1), and a hands-on demonstration in part 2. We hope this will get you started on your road to faster development, easier experimentation, and convenient sharing between data science and DevOps teams. ### **Further reading** To keep on learning and experimenting with Kubeflow and Kubeflow Pipelines: 1. Watch the [Webinar](https://www.brighttalk.com/webcast/6793/424551) we created with the exact same example. 2. Play with [Kubeflow examples](https://github.com/kubeflow/examples) on GitHub. 3. Read our [Kubernetes for data science: meet Kubeflow](https://ubuntu.com/blog/kubernetes-for-data-science-meet-kubeflow) post. 4. Visit [ubuntu.com/kubeflow](http://www.ubuntu.com/kubeflow) ### How to try Kubeflow? To try Kubeflow on your Windows, macOS, or Ubuntu machine follow one of these: 1. Tutorial [here](https://ubuntu.com/tutorials/deploy-kubeflow-ubuntu-windows-mac#1-overview) 2. Video: *Would you like us to deploy and maintain your Kubeflow deployments? Find out more on Canonical’s [Kubeflow consulting](https://ubuntu.com/kubeflow/consulting) page.* ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical.com/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### Arduino® VENTUNO™ Q is available for pre-order with Ubuntu pre-installed](https://canonical.com/blog/arduino-ventuno-q-is-available-for-pre-order-with-ubuntu-pre-installed) London, UK – August 25, 2026 – Following our initial collaboration announcement in March 2026, Canonical and Arduino (a subsidiary of Qualcomm Technologies, Inc.) are excited... [Canonical](https://canonical.com/blog/author/canonical) 25 August 2026 [### A look into Ubuntu Core 26: Building a local AI inference appliance in a virtual machine](https://canonical.com/blog/ubuntu-core-26-ai-box) Welcome to this blog series which explores innovative uses of Ubuntu Core. Throughout this series, Canonical’s Engineers will show what you can build with this Core 26 release,... [Gabriel Aguiar Noury](https://canonical.com/blog/author/g-aguiar-noury) 16 June 2026 [### AI at the edge: simplifying infrastructure with Cisco and Canonical](https://canonical.com/blog/ai-at-the-edge-simplifying-infrastructure-with-cisco-and-canonical) Legacy infrastructure was not designed for the requirements of the AI era. While large-scale model training remains centralized in data centers, test-time inference is rapidly... [Pedro Lazzarotto](https://canonical.com/blog/author/pedro-lazzarotto) 11 June 2026 [### What is RDMA over Converged Ethernet (RoCE)?](https://canonical.com/blog/what-is-rdma-over-converged-ethernet-roce) Previous articles walked through RDMA (Remote Direct Memory Access) as a programming model and InfiniBand as the fabric that was built around it. Both led to the same... [Benjamin Ryzman](https://canonical.com/blog/author/benjaminryzman) 9 June 2026