Deputy National Statistician on the future of linked data in government

The new Integrated Data Service (IDS) sets a clear direction for the future of linked data in government, says Alison Pritchard, Deputy National Statistician and Director General for Data Capability at the Office for National Statistics (ONS). 

A cross-government project led by the ONS, the IDS focuses on linking data to provide otherwise untapped insight and to allow real-time analysis on a growing range of integrated data assets. “The potential we have here is immense,” Pritchard tells Government Transformation Magazine. “We finally have a service that removes friction from the way data is provided and accessed across government, enabling a simple flow of analysis into policy evaluation and development – putting us on the front foot to solve pressing challenges.”

ONS has been developing the service over the last 18 months using the Google Cloud Platform. It represents a significant step in achieving the aims of the government’s National Data Strategy and the roadmap for digital and data from 2022 to 2025. 

Earlier this year, the IDS received Digital Economy Act accreditation; it's first major milestone and a formal recognition that it’s a safe and secure platform, allowing the service to operate as a Trusted Research Environment. Since then, ONS has made de-identified 2021 Census data available for faster analysis  – “opening the door for other departments to bring together critical datasets where, over time, users will be able to join data with other sources within the IDS data catalogue", Pritchard explains.

“The question isn't what's most important to you as a department, but what data you think is most valuable to be shared. It’s turning up some very interesting examples of the power of the data that is being held in silos.”

In terms of size and scale, “we're up to about 69 data sets that are inside the IDS at the moment,” Pritchard notes. “By early next year, it will be up to about 113 datasets.” 

The intention, she adds, is to have every badged government analyst - roughly 14,000 members – using the IDS as a regular tool. 

A nuanced picture 

Julian McCrae, Deputy Director of the Integrated Data Service Strategy, highlights the “massive potential” for the IDS to provide analysts with “more meaningful insights across different domains.” Such insights, and how they can inform policy and improve public services, are already coming to fruition.

The first report based on IDS analysis has recently been published, looking at the differences between estimates of Welsh language ability reported in Census 2021 and the Labour Force Survey. It is a joint project between the Welsh Government and ONS to improve the understanding of Welsh language statistics. 

ONS has also been analysing data produced during the pandemic to understand the relationship between people’s health and the labour market. “We’re starting to tease out what's happening here on a more nuanced level than ever before," McCrae says. "By getting to the underlying causes of declining health and how this impacts job retention we can target policies much more effectively and prevent people from going through incredibly traumatic experiences."

The IDS is also intended for use by local governments. McCrae says there’s a lot of interest here from local authorities that may have less analytical capability in-house but will benefit from a more granular picture of the local economy to inform local strategies, for example. 

“We’re particularly interested in local authorities and how we can help them,” Prichard adds. “There's always improvements to be made on the relationship between local and central government and we’re looking to broaden those relationships in due course.” 

‘Designed for tomorrow, built for today’

From the very start, Pritchard set out with clear intentions for the IDS to be “designed for tomorrow, built for today.” A common mistake made by government when working with data, she argues, is its short-termism and failure to future proof. 

Instead, the technical architecture of IDS looks at bringing people from what they do in the present to how to operate in the future. It’s legacy free, allowing users to access it from standard appropriate computers through a secure browser environment, therefore “balancing the need to push ahead with transformation and simultaneously bring users along on the journey,” Pritchard says. “This was a challenge we were acutely aware of throughout.”

The service is built to be entirely cloud native with opportunities for distributed data operations. Rather than bringing data into it and operating like a big data lake, it can reach out and access data however departments are storing it. “It’s designed to make data sharing agreements more sophisticated, broad and effective – allowing government to tackle big thematic challenges like climate change, economic growth and healthcare,” Pritchard says.

Underpinning the IDS is the Reference Data Management Framework, an indexing mechanism that allows data to be linked according to the need of the analysis on a consistent and repeatable basis. 

All of these foundational elements underpin the ultimate purpose of the IDS: frequent and more effective policymaking that “puts data sitting at the very heart of government machinery to best possible use,” Pritchard says.  “When a new issue comes up, we want government and academic researchers to start answering those questions much more rapidly than they have done in the past - in a matter of hours as opposed to weeks.”

Future growth 

ONS are focusing efforts on improving user experience and maturing the service. “We’re focused on getting the research community transitioned from the Secure Research Service into the IDs,” Pritchard says. “More broadly, we’re looking at opening it up to other users - why should the public not have a means of accessing and undertaking analysis in a totally appropriate and controlled way?” she asks. 

ONS has just agreed for the provision of mobility data from a telecoms company. For Pritchard, this signals an opportunity to realise the brokerage of data procurement; the acquisition of data from private sources for broader analytical use and combining that with government data sources. “I think this is going to grow significantly in the years ahead and I want IDS to be at the forefront of it.”

Looking even further into the future, she jokingly adds: “I'm looking forward to quantum IDS. I've got no idea what's going to be in it, but I know it will be operating safely, securely and ethically with vast amounts of data.”

Pritchard recognises that there are still challenges to overcome in effective data-sharing across government, particularly around the limitations in what people are willing to share. She believes the narrative around data-sharing is often misplaced: “We put a lot of focus on ensuring that data doesn't go astray, but we tend not to count the risk of data not being used around quality of life. If you bring that into a risk model then you have a slightly different approach to making sure the data is being used rather than just being protected. That's quite an important change that we need to make ahead.”

A key lesson to emerge in recent years, Pritchard says, is the importance of government's preparedness of its data in responding to challenges. “One thing I've been really clear about with the IDS is it provides the means to respond to unknown challenges because we have the data, the linkage and the service available, to respond to whatever the future throws at us.