Your submission was sent successfully! Close

Thank you for signing up for our newsletter!
In these regular emails you will find the latest updates from Canonical and upcoming events where you can meet our team.Close

Thank you for contacting our team. We will be in touch shortly.Close

  1. Blog
  2. Article

on 27 August 2021

DataOps: keeping the data flowing with Model-driven Operations

DataOps and model-driven operations

DataOps: Prologue – why is it so hard?

If you’ve ever lived DataOps, you’ll know that it’s a challenge at the best of times. A day in the life of a typical data engineering team involves securing, releasing, debugging and stabilising complex and oftentimes fragile data pipelines. These pipelines can involve many source applications and intermediaries, and troubleshooting them under management pressure when it’s all going wrong is stressful.

There are so many complex layers to modern data platforms that need to be cared for that the team can often feel overwhelmed. Sometimes it feels like it isn’t unusual for 80% of the sprint to be focused on low-level plumbing and fixing technical debt.

Nobody likes technical debt and with less than 20% of the team’s work time available for actual product development, the DataOps team is often on the receiving end of management frustration. We’ve all been in That review call—you know the one I mean: why are things taking so long? Why is there so little to show? Why is your cadence so poor?

I want you to step out of that difficult and uncomfortable conference call, just for a moment. Stop trying to defend and justify yourselves; instead, reflect.

Exegesis: doing less with more

If even 20% of the team’s distraction is removed, the team’s productivity, in theory at least, doubles. And so I introduce you to Model-driven Operations. Model-driven Operations models the relationships between components, their interactions and interdependencies. This paradigm has slowly been gathering momentum in the Kubernetes community. Some of you may already be using, or thinking of using, Kubernetes operators to help automate deployment of trade tools like Apache Spark and Flink.

Model-driven Operations, however, is a compute paradigm that goes much further than point deployment automation of an individual software component. With model-driven operations we can model the complete solution. 

This means going beyond deployment. It also means going beyond individual components. We can also model the day-0 to day-N tasks associated with the solution. These tasks go beyond deployment and integration, to daily operational management tasks such as certificate renewal, password rotation, post-mortem snapshotting, performance reporting, backup and restore, high-availability failover, disaster recovery testing and DRP execution.

Too busy going places to focus on improvement? Image courtesy of | credit Holapaco77

This all sounds time consuming! Well, perhaps at first. But investing time in building an automation model of your team’s operational tasks can and will yield a return on investment, improving quality, repeatability, predictability and most importantly, productivity.

Introducing Juju
Recognising the complexity and productivity challenges contemporary operations teams face, and seeking to help teams to encode their wisdom into automation models, Canonical responded with Juju – the reference framework for Model-driven Operations.

Juju is a free, open source model-driven operations solution. Juju supports teams by encoding operations knowledge and wisdom into composable solution-building blocks.

Connecting two building blocks—or Charms, as we call them—is as easy as
`juju add-relation kafka:monitoring prometheus`

Synthesis: Model-driven Operations for good

Let’s reimagine that sprint review call. Management is seldom happy—we all know that! But after investing some story points on codifying a few of the team’s operational tasks into an automation model, the team’s productivity has measurably increased. Perhaps productivity has doubled – or more! And management has voiced far less unhappiness lately.

As the DataOps team has more new projects completed and in production, management has more success stories to share. It’s starting to feel like the Sword of Damocles no longer hangs above the team. For the first time ever, the team has had the same scrum master for more than two quarters. Things are looking up!

Envision the situation: with less time spent on mundanity and extinguishing fires, the team can all spend more time in the flow state doing what they love to do best – productive, net-new engineering.

Epilogue: make the happy ending real

Let’s make it real.

  • Want to make this happen? Learn more about Model-driven Operations, Juju and charmed operators.
  • Contact us to find out about our managed application services and how they can help your data engineering teams to double down on productivity.
  • Further reading: Data Lab Architectures

Related posts

21 May 2024

Canonical supports Microsoft Azure Cobalt 100 VMs

Canonical announcements Article

Canonical announces that Ubuntu and Ubuntu Pro now support Microsoft Azure Cobalt 100 VMs, ideal for diverse workloads and leveraging Arm-based architectures for enhanced performance and efficiency. ...

Tytus Kurek
20 May 2024

Cloudify your data centre – A guide to VMware infrastructure transformation

Cloud and server Cloud

You know what’s going on. You’ve been monitoring the situation around VMware for at least a year now. There is no need to convince you that whatever comes next, you have to prepare for a big change. You and your team are already well prepared. You have a budget, timeline and necessary resources. However, one ...

Oliver Smith
17 May 2024

Migrating from CentOS to Ubuntu: a guide for system administrators and DevOps

Cloud and server Article

CentOS 7 is on track to reach its end-of-life (EoL) on June 30, 2024. Post this date, the CentOS Project will cease to provide updates or support, including vital security patches. Moving away from the RHEL-based ecosystem might appear daunting, but if you’re considering Ubuntu the switch can be both straightforward and economically viabl ...