Evidence Gathering: The Field Guide

Evidence?  Of what?

We have great lives as Evidence Gatherers – really, we do. Swanning around to meetings, reading interesting blogposts from MRD02 projects, being nosy about what the programme’s projects are doing and writing about stuff that engages us.  But there is a more serious side to the role.  The clue’s in the name, really: we’re here primarily to Gather Evidence.  But evidence of what, and why?

Well, like everyone else in the research sector, JISC is under considerable obligation to provide clear and compelling evidence of the value of its activities.  Everyone on an MRD02 project knows that what the programme’s projects are doing is going to really change things for the better in research data management – whether that’s at our institution, our discipline or more broadly across the sector – but how do we prove this?  We all know there’s less money going around to fund the sort of research we want to do in RDM, so how do we make the case in clear and irrefutable terms that our work brings benefits?  Real, measurable, trackable benefits?  Hence the decision to undertake a structured approach to gathering and presenting the evidence of the benefits of MRD projects.

Anyone from an MRD01 infrastructure project will remember the requirement for a benefits case study near the end of project activity.  (These were brought together in a handy summary document.)  But we couldn’t help thinking, ‘If only we’d been able to plan for writing this case study earlier.  Then we could’ve put some pre-activity benchmarks in place to show how great we were.’  So this time around, projects were introduced to the benefits work at the programme’s kick-off meeting, and then wrote a blog post early in the project to outline the benefits they were expecting to realise.

The Field Guide to our approach

One of the main things we’ve noticed when reading these blog posts is that there is often a bit of confusion around what constitutes an output; a benefit; and a piece of evidence.  For our purposes, here is the Field Guide to the MRD Evidence Gathering Approach:

  • An output is something that the project is going to make, produce, put in place or that it otherwise aims to deliver.  These will be specified in your project plan.
  • Benefits can be identified by asking, ‘What does this help us (the institution / researchers) to do better?’
  • Evidence consists of specific, clear metrics (quantitative measures) and specific, clear qualitative evidence such as narratives and short case studies, all of which support or prove the benefit.

So for the Evidence Gathering work, we need to establish a list of benefits for each project, and each benefit in turn needs to be supported by evidence.

An example:

  • Output: Production and approval of a data policy is an output (and a great one!  Go you!)
  • Benefits: How does this output help the institution / researchers to do RDM and/or research better?  Well, having this policy to refer to can contribute to i) easier compliance with funder policy, ii) improved availability of RDM infrastructure, and iii) improved ability of the institution to plan for future requirements.  These are three benefits.
  • Evidence: So appropriate evidence can be the tracking of quantitative measures, e.g. an increased number of references to the data policy within research proposals,  against an existing benchmark.  A case study with a researcher showing how the policy helped them with success in bidding would be inspiring and could show compliance with funder policy to good effect.  Evidence of increasing reference to the data policy, over time, along with an increased number of datasets held securely and in a context that makes them available for re-use would be compelling.  Interviews with key staff from planning office or research office (as appropriate) about use of RDM roadmap/policy, or a narrative detailing how the policy is being used to improve the institution’s RDM infrastructure could also be used.

Tailored solutions

At programme events, project staff will probably have noticed how diverse the programme is.  The types and sizes of institutions, the aims of projects and the approaches to RDM in all these circumstances make for interesting meetings and energetic debate.  However, it also means that we don’t propose a one-size-fits-all approach to the Evidence Gathering work, so much of our time is currently spent crafting a tailor-made list of sensible and appropriate pieces of evidence for each project. These are to be delivered in an Evidence Report along with Final Reports, but projects should also find the material very helpful when putting together their sustainability business cases.

The goal is to have clear, understandable and compelling evidence for each project which contributes to an evidence base for the programme as a whole.  This will show the difference made for the better – how we as a programme have improved matters, changed the game and moved RDM onwards in the UK HE sector.