Your submission was sent successfully! Close

Jump to main content
  1. Blog
  2. Article

on 27 August 2021

DataOps: keeping the data flowing with Model-driven Operations

DataOps and model-driven operations

DataOps: Prologue – why is it so hard?

If you’ve ever lived DataOps, you’ll know that it’s a challenge at the best of times. A day in the life of a typical data engineering team involves securing, releasing, debugging and stabilising complex and oftentimes fragile data pipelines. These pipelines can involve many source applications and intermediaries, and troubleshooting them under management pressure when it’s all going wrong is stressful.

There are so many complex layers to modern data platforms that need to be cared for that the team can often feel overwhelmed. Sometimes it feels like it isn’t unusual for 80% of the sprint to be focused on low-level plumbing and fixing technical debt.

Nobody likes technical debt and with less than 20% of the team’s work time available for actual product development, the DataOps team is often on the receiving end of management frustration. We’ve all been in That review call—you know the one I mean: why are things taking so long? Why is there so little to show? Why is your cadence so poor?

I want you to step out of that difficult and uncomfortable conference call, just for a moment. Stop trying to defend and justify yourselves; instead, reflect.

Exegesis: doing less with more

If even 20% of the team’s distraction is removed, the team’s productivity, in theory at least, doubles. And so I introduce you to Model-driven Operations. Model-driven Operations models the relationships between components, their interactions and interdependencies. This paradigm has slowly been gathering momentum in the Kubernetes community. Some of you may already be using, or thinking of using, Kubernetes operators to help automate deployment of trade tools like Apache Spark and Flink.

Model-driven Operations, however, is a compute paradigm that goes much further than point deployment automation of an individual software component. With model-driven operations we can model the complete solution. 

This means going beyond deployment. It also means going beyond individual components. We can also model the day-0 to day-N tasks associated with the solution. These tasks go beyond deployment and integration, to daily operational management tasks such as certificate renewal, password rotation, post-mortem snapshotting, performance reporting, backup and restore, high-availability failover, disaster recovery testing and DRP execution.

Too busy going places to focus on improvement? Image courtesy of | credit Holapaco77

This all sounds time consuming! Well, perhaps at first. But investing time in building an automation model of your team’s operational tasks can and will yield a return on investment, improving quality, repeatability, predictability and most importantly, productivity.

Introducing Juju
Recognising the complexity and productivity challenges contemporary operations teams face, and seeking to help teams to encode their wisdom into automation models, Canonical responded with Juju – the reference framework for Model-driven Operations.

Juju is a free, open source model-driven operations solution. Juju supports teams by encoding operations knowledge and wisdom into composable solution-building blocks.

Connecting two building blocks—or Charms, as we call them—is as easy as
`juju add-relation kafka:monitoring prometheus`

Synthesis: Model-driven Operations for good

Let’s reimagine that sprint review call. Management is seldom happy—we all know that! But after investing some story points on codifying a few of the team’s operational tasks into an automation model, the team’s productivity has measurably increased. Perhaps productivity has doubled – or more! And management has voiced far less unhappiness lately.

As the DataOps team has more new projects completed and in production, management has more success stories to share. It’s starting to feel like the Sword of Damocles no longer hangs above the team. For the first time ever, the team has had the same scrum master for more than two quarters. Things are looking up!

Envision the situation: with less time spent on mundanity and extinguishing fires, the team can all spend more time in the flow state doing what they love to do best – productive, net-new engineering.

Epilogue: make the happy ending real

Let’s make it real.

  • Want to make this happen? Learn more about Model-driven Operations, Juju and charmed operators.
  • Contact us to find out about our managed application services and how they can help your data engineering teams to double down on productivity.
  • Further reading: Data Lab Architectures

Related posts

26 September 2023

Canonical releases Charmed MLFlow

AI Article

Canonical announced today that Charmed MLFlow, Canonical’s distribution of the popular machine learning platform, is now generally available. Charmed MLFlow is part of Canonical’s growing MLOps portfolio. ...

26 September 2023

CVE 우선순위 지정을 통한 오픈 소스 보안

Security Security

최근 연구에 따르면 엔터프라이즈 시장의 애플리케이션 중 96%가 오픈 소스 소프트웨어를 사용합니다. 오픈 소스 환경이 점점 더 세분화됨에 따라 조직에 대한 잠재적인 보안 취약점의 영향을 평가하는 작업이 엄청날 수 있습니다. 우분투는 가장 안전한 운영 체제 중 하나로 알려져 있습니다. 하지만 그 이유는 무엇일까요? 우분투 보안팀은 매일 알려진 취약점에 대해 업데이트된 소프트웨어 패키지를 수정하고 릴리스하기 때문에 ...

Serdar Vural
22 September 2023

Meet Canonical at India Mobile Congress 2023

Canonical announcements Article

India Mobile Congress (IMC) is the largest telecom, media, and technology forum in Asia, jointly organised by India’s Department of Telecommunications and the country’s Cellular Operators Association. It is also the biggest networking event in India, establishing itself as a showcase of innovation, technology and digital transformation. C ...