Making federated data more discoverable

DEFRA

The Department for Environment, Food and Rural Affairs (DEFRA) is a UK government department responsible for safeguarding our natural environment, supporting our food and farming industry and sustaining a thriving economy. They have around 3,500 staff, including policy makers, scientists and research specialists, based in London, Bristol, York and other offices throughout England.

The challenge

Defra is a ministerial department supported by 33 agencies and public bodies. Managing data between and across these various departments is challenging and Defra had a strategic issue with data visibility: there was no single system for handling data across constituent agencies, and searching for and being able to effectively use different data sets between departments was incredibly difficult.

The central Data and Information team wanted to help staff find data more easily from across the entire Defra estate. As a secondary goal, they were also keen to highlight problems with data (such as quality issues) to shape future work.

Register Dynamics was asked to help find and build a solution these challenges.

How we helped

Alongside a multidisciplinary team, drawing on the expertise of a product manager, delivery manager, lead developer, business analyst, user researcher and UX designer, we designed and built an internal data catalogue.

This mixed team of contractors and civil servants ran using an agile sprint-based model and a Kanban board to track progress. Between development sprints we held planning and retrospective sessions, which included several Defra stakeholders. These show-and-tell sessions encouraged the sharing of knowledge and progress, and enabled interested staff to be involved throughout.

With this insight, we developed a prototype for a metadata catalogue which all departments would have access to. This new data catalogue offers domain-specific search experiences and various dataset visualisation features. It was developed using CKAN and GDS design patterns. Our use of well-supported open software delivered rapid value without needing expensive suppliers.

It was further refined with the help of iterative user testing. We extended the design to show the most helpful information in search results and to harvest more locations, including Sharepoint, to provide better search coverage. We also designed integration with an Azure single sign-on service to make authentication easy.

We also established that to make finding data easier we would need to improve data publishing across the organisation. So, we helped to develop and promote consistent data standards for metadata and schemas. We designed new interfaces and tools to enable publishers to create better metadata and our publishing colleagues worked with us to share best practice.

We also ran a Discovery that researched the needs of authoritative data publishers and consumers. Here, we explored both the data finding needs of policy teams and the approach into linked data, as well as the issues and challenges currently being experienced and identified areas for potential improvement. From this insight, we developed a roadmap and model for increasing the maturity of Defra’s linked reference data use.

The result

The centralised metadata catalogue continues to be highly regarded by the business and policy teams. We were praised across the business for focusing on real users and for the easy search experience that the data catalogue offers.

Our support and technical recommendations have also been well received and have helped to enhance data visibility and the overall data management strategy across the organisation. Our research laid the groundwork for further improvement on data governance and management.