Building data capability at scale: unlocking AI’s potential in the UK public sector

As AI becomes a cornerstone of digital transformation in the public sector, the spotlight has turned back to a familiar yet unresolved challenge - data. While proof-of-concept pilots in generative AI and machine learning are flourishing across government departments, few progress to scaled, operational services. The reason? The data isn’t ready.

Leaders across departments are now grappling with a central question: how can we build the data capability required to turn experiments into lasting transformation? To unlock AI’s transformative potential, the UK public sector must build and sustain data capability at scale - through robust infrastructure, clear standards, cross-functional teams, and strong governance.

We spoke to senior leaders from the Office for National Statistics (ONS), the Scottish Government and Equal Experts to gauge where different public sector departments currently are in their data capability journeys, as they grapple with the fundamentals of data management, engineering and leadership.

Breaking down silos: From data ownership to data platforms

A foundational issue for many departments is the siloed nature of data. Whether through legacy systems, fragmented governance or organisational inertia, data often remains trapped in isolated pockets.

Simon Case, Global Head of Data and AI at Equal Experts, believes that while silos are a persistent feature, modern platforms can help erode them.

“Silos exist for technical, organisational, and cultural reasons - there’s always a mix. If we’re talking about practical steps, on the technical side, modern data platform technologies - hyperscale platforms like Snowflake, Databricks, Starburst - are good at enabling information sharing. They support the creation of data products that can be shared across the enterprise, with governance capabilities like access control and visibility baked in.

Charles Baird, Chief Data Architect at the ONS, takes a systems-level view. “You can’t manage what you can’t see,” he says. “We need universal data discovery tools – data catalogues that dynamically track and describe our assets. Static spreadsheets of data assets are obsolete the moment you save them.”

At the Scottish Government, Head of the Scottish Digital Academy, Lee Dunn, sees this as a leadership and alignment challenge.

“Tackling siloed data ownership is fundamentally a leadership challenge,” he says. “You need a common data strategy, senior sponsorship, and clarity across the ecosystem on who’s responsible for what - design, maintenance, engineering, analysis. Everyone needs to understand their role and how they interact with others.”

Also in the Scottish Government, Eilidh McLaughlin, Deputy Director of Digital Ethics, Inclusion and Assurance, highlights a similar challenge.

“Many enterprise systems unintentionally create barriers to sharing,” she notes. “AI could help summarise and surface insights from these systems, but only if there’s integration, interoperability and reliable HR metadata to understand team structures.”

Prioritising engineering over experiments

A common issue in the public sector is over-investment in data science at the expense of foundational work in data engineering and infrastructure. This imbalance can leave organisations with multiple impressive pilots – but few operational services.

“Many data leaders find they spend lots on data science, which yields promising proofs of concept but are not integrated into the business," says Case, pointing to a widespread structural issue. “Without engineers to productionise those models, you don’t get value. You need someone to build the production data pipelines, monitor performance, and handle edge cases.”

Baird agrees: “AI can’t supply context. Well-structured, well-described data - supported by solid metadata and standards - is what makes AI and analytics effective.”

Dunn points to the importance of foundations. “It’s easy to get caught up in the shiny new things in AI - the dashboards, visualisations, demos - and all of that is exciting,” he says. “But if you don’t get the foundations right, it’s never going to stand up. Engineers and architects are vital - they might not be as visible as data scientists, but they’re just as essential.”

Moving beyond the AI pilot plateau

While experimentation is important, many leaders are now focused on converting pilots into operational services that deliver long-term value. And this requires a different mindset, funding model and operational readiness.

“Pilots are just the start,” says Case. “You need to have a pathway to scale from day one – think about long-term operations, product management, and how the system will evolve over time.”

Baird highlights how the ONS has instituted a formal governance model to sift signal from noise. “We’ve created the AI Leadership Group to assess proposed AI projects. We’re open to new ideas, but they must demonstrate real value. We ask: does it move the dial? Does it scale?”

But there is still an important place for pilot projects in the public sector digital toolbox. McLaughlin emphasises that pilots also offer a way to evaluate risk and bias before deployment. “Testing AI in a secure, transparent environment is essential. We need to understand where it helps, where it harms, and how to control for discrimination.” 

Designing cross-functional teams that deliver

Building effective cross-functional teams, combining engineers, analysts, product owners and users, remains a cultural and structural challenge for government.

Dunn advocates for deeper integration of teams and clearer cultural alignment. “We need to distinguish line management hierarchies from agile, interdisciplinary teams,” he explains. “With interdisciplinary working, you get alignment, autonomy, and psychological safety. That’s where you can reward collaboration and tackle real complexity.”

He urges departments to organise teams around the work, not the hierarchy. “Leaders need to free people from those constraints so teams can form dynamically around real problems.”

Case is a strong advocate for embedding users directly into delivery teams. “It’s not just about gathering requirements, it’s about co-design. Having a frontline worker or policy user in the team really helps us create something that’s easily used and will generate business value.”

Baird, meanwhile, is building capability from the ground up. “We’ve had huge success with apprenticeships. People join us with no prior experience and go on to become outstanding data engineers. That long-term investment pays off.”

Governance: Trust as the enabler

Data governance in government isn’t just a matter of compliance – it’s a precondition for public trust. But it must be approached as an enabler, not a blocker.

“Security is always top of mind,” says Case. “But your InfoSec team should be your partner, not your obstacle. Start small, be specific, and build trust incrementally.”

Dunn sees governance as a proactive design principle. “Governance needs to be built in from the start, not bolted on later,” he says. “We need to design for governance - access controls, transparency, shared component libraries - and balance usability with cyber resilience. If systems are too locked down, they’re unusable. But they still need to be robust.”

McLaughlin adds that transparency is the most powerful tool for building public confidence. “We try to publish everything - data standards, code, workflows - and involve civil society and the ICO in shaping our approach. Ethics isn’t an afterthought, it’s embedded.”

Baird also highlights the need for architectural discipline. “If you want to future-proof your data, you need standards, change control, and good metadata. That’s what makes interoperability and reuse possible across departments.”

Leadership and the shift to product thinking

The transition to data-driven government requires strong leadership, capable of articulating vision, setting priorities, and fostering a product-centric culture.

“Leaders must constantly reinforce the ‘why’,” says Baird. “Why are we doing this? What does success look like? What’s your role in achieving it? That clarity unlocks momentum.”

Dunn stresses the importance of shared leadership. “It’s not just about hiring or buying,” he says. “It’s about a long-term vision. Yes, senior leaders set direction, but leadership exists at all levels. Everyone needs to share that vision, call out risks, and avoid reinventing wheels when others have already done the work.”

McLaughlin cites the Scottish Digital Academy as a key enabler. “We offer leadership courses in data and product management to help senior staff understand how to make data work for them and their users.”

Future-proofing for AI: back to basics

Looking ahead, the leaders are united in their advice: focus on the fundamentals. “Get the basics right,” says McLaughlin. “Data standards, quality collection, fair design, and clear architecture will set you up for success.”

Baird reinforces that point: “A well-designed data product, with predictable structure and clear semantics, is usable now and in six months’ time. That’s what future-proofing looks like.”

Case points to the tools that will power the next wave. “Vector databases, retrieval-augmented generation, better search layers - these are technical must-haves if you want to use GenAI meaningfully.”

Dunn echoes the call for foundational strength. “We need clean, accessible, transparent data. We need to identify our single sources of truth and build a strategy around them. And we need modular platforms - so we can meet today’s needs, but also adapt to future ones. Ultimately, future-proofing is as much about leadership and management as it is about tech. If we don’t get that right, even the best AI tools won’t succeed.”

Conclusion: Capability before complexity

AI offers incredible opportunities for the public sector - but real transformation won’t come from the technology alone. It requires a sustained, strategic investment in data capability: platforms that scale, teams that collaborate, governance that enables, and leadership that leads.

This is not about chasing hype. It’s about building enduring foundations. If government can shift from pilots to wide-scale platforms, from data siloes to business-as-usual services, the rewards - for both citizens and the Civil Service - could be huge. But as the sector’s data leaders have reminded us, it all starts with getting the basics right.