Data sharing can improve government services and save lives. So why aren’t we doing it?

I have spent over two decades working across government, most of that time as a senior Civil Servant, on projects ranging from welfare modelling to education, and one lesson has stood out above the rest: data, used responsibly, not only shapes better policy, it protects the vulnerable and it saves lives. 

And yet, time and time again, I have seen progress blocked by misunderstanding, fear, and outdated attitudes towards data sharing. Even after all these years and amid an ongoing technological revolution, Britain is stuck in Groundhog Day when it comes to cross Government data sharing. 

One of the major things that keeps us there is data protection. While Data protection is very important, it is one of those phrases - like ‘health and safety’ - that is overused to the extent that it has become a cliché. Yet data protection laws do not prevent safe, intelligent, responsible sharing of data. 

Britain’s Information Commissioner John Edwards and Professor Chris Whitty are both on the record saying this explicitly when it comes to sharing data to safeguard children. In 2023, an Institute of Government report said that the UK had “virtually no data to analyse” during the COVID-19 pandemic, singling out technical challenges in linking local council data with the NHS. 

There have been myriad other examples: In 2024, MPs spoke out over government plans to scrap the UK Census and replace it with ‘other sources of data’, warning that the ‘other sources of data’ could not be relied on because of long-standing issues with cross-departmental data sharing. “Data withers in silos across countless government bodies,” MPs warned. 

It is a fact that good data sharing drives smarter policy. Back in 2004, my team linked data, in a sustainable repeatable manner, across HMRC and the Department for Work and Pensions to track individuals from welfare into work. It became a foundation for understanding not just what was happening in the system, but what was working. We could model the effects of new employment programmes, understand what created the revolving door of low-paid work and benefits, and improve outcomes using completely anonymised data. 

Challenges are even more interconnected today. Education data alone won’t tell you which segments of society might be most at risk from crime, but link that data with health, social services, and justice datasets, and patterns emerge. Early warning signs become visible, and interventions become possible.

When we analysed school absence data - including small, regular absences rather than just severe and persistent absence - we discovered that the first two weeks of term were predictive of attendance for the entire year. That insight now shapes national policy. But we could only uncover it because we had individual-level, linked, de-identified data across multiple systems.

The technology exists. The data is there. So what is the barrier? 

It is culture. Every department has a data protection officer (DPO) but too often they come from a school of thought that says: ‘Computer says no.’ I have long argued that a culture change needs to come from the top: We need a single Government chief data officer heading the profession across government, establishing a method and a philosophy that can trickle down across departments. 

Let’s stop trying to prove why we should share data and instead ask, why shouldn’t we? And how can we do it safely?

Make no mistake, inaction is a choice, and the longer we hold off building a network of data sharing across government, the longer we will find ourselves the victims of fraud, the longer we will miss individuals who are in crisis. Critics may point to areas of data misuse, but these are not errors of data, they are failures on the part of us, humans, to govern and analyse that data.

Consider the UK exam grading fiasco, under which an algorithm was used to determine A-Level and GCSE results after exams were cancelled during the COVID-19 pandemic. It was the interpretation of the data that was the problem, not the data sharing itself. 

We recently published a white paper where executives from several government departments shared their views on the current state of affairs, and the insights were telling. We heard from departments that struggled to build cross Government data sets that would have had genuinely positive financial outcomes for citizens, or would have enabled services to be provided where they were not. 

I have spent a good deal of time in Estonia over the years and, like many others here in Britain, have looked enviously at the Baltic nation and thought: ‘Why couldn’t we do that?’. Indeed, their system is impressive, built around a platform, which enables real-time, citizen-controlled data sharing transparently. But Estonia started from scratch after independence, building on a system of trusted national registers, when it made digital government a top national priority.

We are not Estonia, and we can never have a ‘Big Bang’ approach here. When it comes to data, Britain has to fix the plane in mid air - incrementally, safely, but urgently. The good news is we are not starting from scratch, and the National Data Library is a step in the right direction - if, indeed, government departments back its vision and the public is kept on board. 

Of course, we must separate operational data use - like that used to calculate Universal Credit - from policy-driven, aggregated research. Both require a careful linking of individual-level data, but within secure, controlled environments like the Office for National Statistics’ Secure Research Service. That’s where real insight happens and it is where lives can be improved and not just measured.

The British public deserves a government that works better, not one paralysed by fear of getting it wrong. We must move from a mindset of risk aversion to one of responsible enablement, which means building the systems, the governance, and most importantly, the culture needed to unlock the value of data.

Data sharing isn’t a threat, it is the foundation of effective, compassionate government. And the sooner we realise that, the better off we’ll all be.