Introduction - what is it?
In the grand halls of corporate ambition and government mission statements, a popular phrase echoes with almost religious fervour: “We aim to be data driven.” It has a certain ring to it, doesn’t it? A rallying cry that suggests precision, clarity, and a future where every decision is as sharp as a Savile Row suit. Scratch beneath the surface, though, and you’ll find that “data driven” often means nothing more than, “We touched some data somewhere in the process and called it a day.”
The truth? For many organisations, “data driven” doesn’t mean making smarter decisions based on insights. It means someone ran a report once, waved it in the air, and called it strategy.
To put it simply, being data driven means using data to inform decisions at every level of an organisation, ensuring they are grounded in reality rather than conjecture. It’s about making choices based on evidence, patterns, and insights rather than gut feelings or outdated habits. The problem is, achieving this requires a level of organisational maturity, technical competence, and financial commitment that many are unwilling to entertain. It’s a bit like wanting a championship garden but balking at the cost of seeds and a good trowel.
In the sections that follow, we’ll explore the technical and organisational challenges of becoming data driven and why it’s an investment worth making.
The Two-Headed Data Dog
To be data driven, one must first embrace the simple truth: decisions informed by data require, well, data. Not anecdotes. Not gut feelings. Actual, structured, usable data. That means collecting it, analysing it, and drawing conclusions to guide decision-making. It’s not rocket science (though, for some organisations, it might as well be).
Picture a two-headed dog, one head representing the technical side and the other the organisational. They’re supposed to work in harmony but often end up looking like Cerberus on a bad day—pulling in opposite directions, barking at nothing, and causing chaos wherever they go.
The Technical Leg
On the technical side, the journey begins with data ingestion, transformation, and storage. It’s about building the pipelines that funnel raw data into usable insights. Sounds straightforward, right? Here’s where reality intrudes: there is no magical endpoint API that can deliver all your data needs in a neat, unchanging package. Your business evolves, your data changes, and so must your systems.
But let’s be honest—how many times have we seen organisations cling to brittle pipelines held together by duct tape and prayers, hoping they’ll last one more quarter? Spoiler alert: they won’t.
And let’s clear up a common misconception: files are not a suitable vehicle for modern data transfer. Yes, I’m looking at you, the team still emailing Excel sheets named “final-final-REALLYFINAL_v2”. Stop it. Think events, APIs, and specialised data distribution mechanisms. The goal isn’t to drown in a sea of CSV files but to streamline and modernise how data flows.
For large-scale needs, consider data lakes or purpose-built databases tailored to specific requirements. The Azure Cloud Adoption Framework (CAF) provides a robust guide for implementing a scalable, secure data platform. It encourages leveraging patterns like the medallion architecture, where raw data is ingested into a bronze layer, cleaned and enriched in a silver layer, and finally optimised for analytics in a gold layer. Think of it as Marie Kondo-ing your data—everything has its place, and it sparks joy (or at least accurate insights).
This structured approach ensures data quality and usability throughout its lifecycle. And yes, your systems will need constant evolution, so embrace it now or prepare for the chaos later.
The Organisational Leg: A Tug-of-War Over Data
The organisational side is no less important—and no less maddening. It requires a cultural shift to view data as a critical enabler of business outcomes rather than just a technical by-product. But let’s face it: shifting culture is hard. Most organisations are more likely to have a team-building retreat than actually build a culture of data-driven collaboration. Spoiler: the latter is much more effective.
Success in this area often hinges on fostering collaboration across teams, ensuring that data governance is embedded in daily operations, and aligning stakeholders around common goals. Sounds lovely in theory, right? In practice, it’s a bit like herding cats—each department fiercely guarding its little kingdom of data while blaming “the process” for inefficiencies.
Enter the Data Product Owner, the unsung hero of this mess. Acting as stewards, they bridge technical teams and business stakeholders, ensuring data products are reliable, secure, and accessible. Think of them as the peacekeepers, walking a fine line between appeasing stakeholders and delivering something that actually works.
This approach aligns perfectly with Azure’s CAF, where the data platform is the foundation for delivering these products. Leveraging tools, governance models, and the medallion architecture within CAF helps to standardise and streamline data product delivery, ensuring scalability and adaptability. Translation: it turns the chaos into something resembling order—most of the time.
Too often, though, organisations fall into power struggles and silos. Instead of recognising data as a shared resource with immense value, it becomes a territorial battle over budgets, responsibilities, and who gets to write the next memo about “data-driven innovation.” Embracing the data product paradigm shifts the focus to collaboration and shared goals, breaking down silos and unlocking the true potential of data. Because, let’s be real: the only thing worse than bad data is good data trapped in a turf war.
Collecting Data: Easier Said Than Done
Everyone loves the idea of “harvesting all available data”—it sounds so futuristic, like plugging into a Matrix of infinite wisdom. But here’s the catch: it’s no good collecting every scrap of information if it’s as reliable as a Magic 8-Ball. Quality is paramount. Garbage data just leads to garbage decisions, only now those decisions come with a fancy dashboard.
From there, data must be transformed, correlated, and exposed in ways that meet diverse business needs. Sounds straightforward, right? Until you realize the needs change weekly, the data sources are a mess, and half the organisation thinks APIs are just the things that call their apps.
A well-designed data platform—such as one built using Azure’s CAF—provides the tools to achieve this. Using event-driven architectures, APIs for dynamic data delivery, and scalable data lakes ensures your platform can handle both large-scale raw data and refined subsets crafted for specific use cases. Employing the medallion paradigm (bronze for raw data, silver for semi-processed, gold for business-ready) helps maintain data fidelity and structure. Think of it as Marie Kondo-ing your data: everything in its place, everything sparking joy.
Flexibility is key, but so is rigour. Rigour is the boring but necessary sibling of flexibility—it makes sure your brilliant, scalable data platform doesn’t collapse the moment someone changes a field name in Salesforce. And yes, your data systems will need constant evolution, just like your business. If that sounds exhausting, here’s the hard truth: if you’re not prepared for this, you might as well pack up your data dreams and focus on collecting something easier—like Pokémon cards.
Breaking Silos: Addressing Organisational Challenges
Data ownership is like a high-stakes game of Monopoly. Should it be centralised? Distributed? Owned by specific departments? These aren’t just technical questions—they’re deeply political ones, and the stakes are higher than Park Place.
Too often, organisations focus on cost rather than value. Instead of asking, “What can we achieve with this data?” the conversation becomes, “Who’s going to pay for it?” The result? Departments squabble over budgets like seagulls fighting over a chip, and silos spring up where collaboration should thrive.
To break this cycle, organisations must embrace the data product paradigm. Think of data as a shared asset, like a public park—one that actually gets maintained, not neglected. Clear ownership and accountability through Data Product Owners are key. These stewards ensure data is managed as a valuable resource, delivering specific data products tailored to meet organisational needs. The goal? Actionable insights and high-quality information that drive business outcomes, not just another pretty spreadsheet gathering dust.
Building the right organisational structures is equally crucial. Following principles like those in Azure’s CAF, businesses can create a cohesive data platform that supports these data products. This means robust governance, clear roles, and a shift in mindset: data isn’t a burden—it’s an enabler of innovation.
When done right, the organisational challenges of becoming data driven don’t have to feel like a Game of Thrones-style power struggle. Instead, they can become opportunities for growth, collaboration, and finally putting an end to the eternal question of, “Whose budget is this coming out of?”
Conclusion: Everybody Wants It, Nobody Wants to Pay for It
In the end, becoming data driven is both an art and a science—and a masterclass in patience. Everyone loves the idea of making smarter decisions, driving efficiency, and gaining a competitive edge, but few are willing to confront the messy reality: it requires investment, effort, and a willingness to change deeply ingrained habits.
For many organisations, “data driven” doesn’t actually mean what it should. Instead, it becomes a vague aspiration, with the hope that touching some data somewhere along the line magically makes it so. It’s not about structured insights or actionable metrics—it’s a checkbox on a strategic roadmap. And frankly, that’s frustrating for those of us trying to make meaningful progress.
It’s a story we’ve seen too often: teams eager to collect data and learn from users, but the initiatives are dismissed or deprioritised because “we already know what’s best” or because the overhead of data collection seems insurmountable or is not prioritized by incompetent “product owners”. Observing users, gathering feedback, or even creating a basic loop for insights is treated as a luxury instead of a necessity. It’s as though the very idea of pausing to listen to the people consuming our services feels like a detour, rather than the foundation of good decision-making.
And that’s the crux of it—many organisations crave the label “data driven” but balk at the groundwork it takes to get there. They want the prestige without the process, the insights without the effort. But the truth is, being data driven isn’t about dabbling with a dashboard or sprinkling some metrics into a presentation. It’s about building systems, fostering a culture of inquiry, and maintaining a relentless focus on learning.
Becoming data driven is an investment—time, money, and energy. It’s not just a project; it’s a mindset shift that requires everyone, from leadership to teams in the trenches, to commit. That means being open to experimentation, allowing failures, and embracing feedback cycles that don’t always confirm your assumptions. Most importantly, it means recognising that real transformation doesn’t happen in isolation or in theory. It happens through hard work, collaboration, and the courage to admit when the data proves you wrong.
So, roll up your sleeves, pour yourself a strong cup of coffee, and get ready to sweat for it. The results, as they say, will be worth it—but only if you’re willing to truly earn them. Because touching data isn’t enough. You have to engage with it, learn from it, and act on it to make meaningful change.