--- title: Why we built a Spark solution for Kubernetes description: We’re super excited to announce that we have shipped the first release of our solution for big data - Charmed Spark. Fully supported Apache Spark for K8s. url: https://canonical.com/blog/why-we-built-a-spark-solution-for-kubernetes?format=md --- 1. [Blog](https://canonical.com/blog) 2. Article --- [Rob Gibbon](https://canonical.com/blog/author/robgibbon "More about Rob Gibbon") 17 October 2023 # Why we built a Spark solution for Kubernetes [apache spark](https://canonical.com/blog/tag/apache-spark) [Big Data](https://canonical.com/blog/tag/big-data) [data fabric](https://canonical.com/blog/tag/data-fabric) [spark](https://canonical.com/blog/tag/spark) --- Share the article We’re super excited to announce that [we have shipped](https://canonical.com/blog/canonical-releases-charmed-spark/) the first release of our solution for big data – [Charmed Spark](https://canonical.com/data/spark). Charmed Spark packages a supported distribution of Apache Spark and optimises it for deployment to Kubernetes, which is where most of the industry is moving these days. ## Reimagining how to work with big data Having the opportunity to rethink how big data is processed meant that we could challenge the status-quo based on the more traditional Hadoop YARN stack. And with our [Charmed Kubernetes](https://ubuntu.com/kubernetes/charmed-k8s) and [MicroK8s](https://microk8s.io/) systems, we have a greatly simplified, yet powerful family of cluster managers to enable full stack deployment of big data clusters and a consistent user-experience across local, on-premises and cloud environments. Of course, you’re not limited to our Kubernetes distributions – you can use Charmed Spark on other conformant Kubernetes – for example AWS EKS. ## And reimagining big data storage too For storage we chose S3 API-compliant [Ceph](https://ubuntu.com/ceph/docs) instead of Hadoop HDFS storage system, although the solution is designed to work with most S3 compatible scale-out storage solutions. HDFS has many problems such as its NameNode with [the entire inode map of the big data filesystem held in Java heap](https://issues.apache.org/jira/browse/HDFS-5711) or the [NameNode’s active/passive failover architecture](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Architecture). We opted to sidestep these and adopt more contemporary object storage solutions as the preferred backing tier for our solution. With modern, high capacity networking (for example > 100GbE), bits can typically be shifted to and from the Spark cluster faster than they can be processed by the Spark cluster, so the HDFS design paradigm of bringing the compute problem to the data makes less sense nowadays. Of course, users can still connect Charmed Spark to HDFS if they so wish. ## Simplifying operations In terms of operations, we wanted to keep the user experience as true to upstream Apache Spark as possible, so that users can drop in our runtime as a replacement for the upstream Spark Kubernetes container image with minimal fuss. CLI commands like `spark-submit`, `pyspark` and `spark-shell` work exactly as you would expect. We provide an [Ubuntu snap package](https://snapcraft.io/spark-client) with client tools to help you get started quickly and easily and this can be installed on the edge nodes of your big data cluster. The snap package also includes our [spark8t](https://github.com/canonical/spark-k8s-toolkit-py) Python library and CLI for managing service accounts and profiles for jobs on your Kubernetes cluster. Our aim with this tool is to make the lives of cluster admins, data engineers and data scientists a little bit easier by allowing them to preconfigure Spark job settings for different types of workloads and for the different Kubernetes service accounts that the jobs will run under. We also offer a [Juju Charm](https://juju.is/docs/juju) for [Spark History Server](https://charmhub.io/spark-history-server-k8s). A Juju Charm is like a copilot for an application, and it contains codified knowledge about how to operate it. This one helps you to deploy and operate the Spark History Server on Kubernetes in a straightforward way. [Read the docs to get started](https://canonical.com/data/docs/spark/k8s/h-deploy-spark-history). Juju is a powerful system for day-2 operational management of complex distributed systems on clouds and on Kubernetes. We’ll be adding more Juju Charms to our Spark solution that cover more functionality over time. ## Get working fast We’ve integrated JupyterLab into the Charmed Spark solution, so that you can conveniently spin up a Jupyter environment on an edge node using Docker and have it start a spark session on your MicroK8s cluster, to make it even easier to work with Spark on K8s. [Learn how to use JupyterLab with Charmed Spark](https://github.com/canonical/charmed-spark-rock#rocks-usage). The full documentation suite for Charmed Spark is available at [canonical.com/data/docs/spark/k8s](https://canonical.com/data/docs) and we also have a reference architecture guide that you can [download](https://ubuntu.com/engage/spark-reference-architecture-for-big-data). Good to know – we offer enterprise grade paid support on the entire solution through our Ubuntu Pro + Support subscription which covers up to **10 years** of break/fix support and security maintenance per major release, in line with our wider commitment to long term support. If you’re interested in learning more, [contact our sales team](https://canonical.com/data/spark) via the form [or call us](https://canonical.com/contact-us). We can also offer help with solution deployment through our fixed-fee deployment services – learn more [here](https://canonical.com/data). Community support is available via our [chat server](https://chat.charmhub.io/charmhub/channels/data-platform) and our [community forum](https://discourse.charmhub.io/). ## More Data Fabric solutions to come Charmed Spark is actually the first in a series of system solutions for data management that we’ll be releasing over the coming months. You can [sign up for the beta program](https://ubuntu.com/engage/canonical-data-fabric-beta-signup) and try out tomorrow’s awesome tech today. If you’ll be at Gitex Dubai or KubeCon North America this Autumn, you’re welcome to come to the Ubuntu booth (that’s Booth B31 DevSlam for Gitex and Booth A2 for KubeCon) and meet me in person; I’ll be pleased to discuss how we can help you to accelerate innovation. [Get in touch Interested in running Ubuntu in your organization?](https://ubuntu.com/about/contact-us/form) ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical.com/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### Can it play Doom? Running an AI LAN party on a Spark cluster with ViZDoom](https://canonical.com/blog/can-it-play-doom-running-an-ai-lan-party-on-a-spark-cluster-with-vizdoom) It’s all about AI these days, so I decided to try and answer the important question: can you make a Spark cluster run AI agents that play a game of Doom, in a multiplayer LAN... [Rob Gibbon](https://canonical.com/blog/author/robgibbon) 23 May 2024 [### Implement an enterprise-ready data lakehouse architecture with Spark and Kyuubi](https://canonical.com/blog/implement-an-enterprise-ready-data-lakehouse-architecture-with-spark-and-kyuubi) Here at Canonical we are excited to announce that we have shipped the first release of our solution for enterprise-ready data lakehouses, built on the combination of Apache... [Giulia Lanzafame](https://canonical.com/blog/author/giulialanzafame) 4 September 2025 [### Accelerating data science with Apache Spark and GPUs](https://canonical.com/blog/accelerating-data-science-with-apache-spark-and-gpus) Apache Spark has always been very well known for distributing computation among multiple nodes using the assistance of partitions, and CPU cores have always performed... [Giulia Lanzafame](https://canonical.com/blog/author/giulialanzafame) 26 June 2025 [### Apache Spark security: start with a solid foundation](https://canonical.com/blog/apache-spark-security-start-with-a-solid-foundation) Everyone agrees security matters – yet when it comes to big data analytics with Apache Spark, it’s not just another checkbox. Spark’s open source Java architecture introduces... [Giulia Lanzafame](https://canonical.com/blog/author/giulialanzafame) 10 June 2025