Monthly Archives: August 2012

Evidence Gathering: The Field Guide

Evidence? Of what?

We have great lives as Evidence Gatherers – really, we do. Swanning around to meetings, reading interesting blogposts from MRD02 projects, being nosy about what the programme’s projects are doing and writing about stuff that engages us. But there is a more serious side to the role. The clue’s in the name, really: we’re here primarily to Gather Evidence. But evidence of what, and why?

Well, like everyone else in the research sector, JISC is under considerable obligation to provide clear and compelling evidence of the value of its activities. Everyone on an MRD02 project knows that what the programme’s projects are doing is going to really change things for the better in research data management – whether that’s at our institution, our discipline or more broadly across the sector – but how do we prove this? We all know there’s less money going around to fund the sort of research we want to do in RDM, so how do we make the case in clear and irrefutable terms that our work brings benefits? Real, measurable, trackable benefits? Hence the decision to undertake a structured approach to gathering and presenting the evidence of the benefits of MRD projects.

Anyone from an MRD01 infrastructure project will remember the requirement for a benefits case study near the end of project activity. (These were brought together in a handy summary document.) But we couldn’t help thinking, ‘If only we’d been able to plan for writing this case study earlier. Then we could’ve put some pre-activity benchmarks in place to show how great we were.’ So this time around, projects were introduced to the benefits work at the programme’s kick-off meeting, and then wrote a blog post early in the project to outline the benefits they were expecting to realise.

The Field Guide to our approach

One of the main things we’ve noticed when reading these blog posts is that there is often a bit of confusion around what constitutes an output; a benefit; and a piece of evidence. For our purposes, here is the Field Guide to the MRD Evidence Gathering Approach:

An output is something that the project is going to make, produce, put in place or that it otherwise aims to deliver. These will be specified in your project plan.
Benefits can be identified by asking, ‘What does this help us (the institution / researchers) to do better?’
Evidence consists of specific, clear metrics (quantitative measures) and specific, clear qualitative evidence such as narratives and short case studies, all of which support or prove the benefit.

So for the Evidence Gathering work, we need to establish a list of benefits for each project, and each benefit in turn needs to be supported by evidence.

An example:

Output: Production and approval of a data policy is an output (and a great one! Go you!)
Benefits: How does this output help the institution / researchers to do RDM and/or research better? Well, having this policy to refer to can contribute to i) easier compliance with funder policy, ii) improved availability of RDM infrastructure, and iii) improved ability of the institution to plan for future requirements. These are three benefits.
Evidence: So appropriate evidence can be the tracking of quantitative measures, e.g. an increased number of references to the data policy within research proposals, against an existing benchmark. A case study with a researcher showing how the policy helped them with success in bidding would be inspiring and could show compliance with funder policy to good effect. Evidence of increasing reference to the data policy, over time, along with an increased number of datasets held securely and in a context that makes them available for re-use would be compelling. Interviews with key staff from planning office or research office (as appropriate) about use of RDM roadmap/policy, or a narrative detailing how the policy is being used to improve the institution’s RDM infrastructure could also be used.

Tailored solutions

At programme events, project staff will probably have noticed how diverse the programme is. The types and sizes of institutions, the aims of projects and the approaches to RDM in all these circumstances make for interesting meetings and energetic debate. However, it also means that we don’t propose a one-size-fits-all approach to the Evidence Gathering work, so much of our time is currently spent crafting a tailor-made list of sensible and appropriate pieces of evidence for each project. These are to be delivered in an Evidence Report along with Final Reports, but projects should also find the material very helpful when putting together their sustainability business cases.

The goal is to have clear, understandable and compelling evidence for each project which contributes to an evidence base for the programme as a whole. This will show the difference made for the better – how we as a programme have improved matters, changed the game and moved RDM onwards in the UK HE sector.

Ways of approaching an RDM Service

A few weeks back an interesting discussion took place on the JISCMRD mailinglist, picked up on this blog under the main question of ‘ What are the requirements an RDMI has to cover?’.

One of the conclusions was that requirements gathering, scope and application obviously depend on a number of factors, one major one being the remit of the project: implementing a system for (pilot) audiences is different from getting an institution-wide service into place and running. The MaDAM project between 2009 and 2011 implemented a pilot RDMI for users from the biomedical domain – and was followed up by MiSS (MaDAM into Sustainable Service) in autumn last year. Whilst having all the crucial experience, findings, contacts and committed stakeholders through MaDAM to build upon, it was clear that a set of new challenges would arise in implementing a RDMI as a service for the whole (and benefit) of the University.

1) Especially having the main stakeholders on board for such an endeavour is of utmost importance – but soon we realised that this process basically never stops: aims, benefits and plans have to be mulled over, renegotiated and agreed upon. There is always one office and internal project more to be included.

2) The University of Manchester also released their RDM Policy in May this year – and also other internal endeavours change structures (research offices), infrastructure (storage, eDMP) and processes. As a research project implementing a service we have to closely liaise with all those stakeholders to be on the same page and coordinate our efforts (and integrate with existing systems and the IT landscape) – despite a small core project team.

3) Outreach, awareness raising and training become more and more important. To this end (also including the points made before) we had to draw up a communication plan, collaborate and plan for an interim service and training for staff. We always strived to hold as many presentations as possible for various audiences at the University about our projects – now we have to start organising dedicated events e.g. at the moment on DMPs. Also: tangible benefits have to be communicated to all audiences!

4) One major challenge for the devleopment of the service and how it will look like lies in balancing the generic with the specific needs. Trying to gather and negotiate as many views, feedback and concerns is a huge task and some evolvement over time is needed – especially as only a system (service!) in use shows all the specifics required for every user, group and discipline. The service coming into life in about a year’s time will be a ‘thin layer’ across the whole RDM lifecycle, but the development is a continuous one and the RDM Service will get more tailored, sophisticated and mature while running.

Meik Poschen <meik.poschen@manchester.ac.uk>
Twitter: @MeikPoschen

Research data + research records management = research management?

Interesting meeting at Leicester this week with our Information Assurance Services. Andrew Burnham and myself have developed a roadmap to implement research data management policy as required for EPSRC and other funders and are currently leading the development of guidance and support for researchers in time for the new academic year. We are doing this in collaboration with colleagues in IT Services, the Academic Practice Unit, Library and Research Support Office on behalf of the Research Computing Management Group, chaired by our PVC Research & Enterprise Kevin Schurer, which feeds recommendations up to the University Research Committee. We are feeding in the many relevant external information and guidance resources as produced through the JISCMRD programme, UKDA, DCC and related.

I’ve lost some of you haven’t I?! To add to the confusion, there is also the university code of practice for researchers.

However, the question has arisen as to when and how we distinguish the university records management policy with this research data management policy? We are of course referring here to differences in language between different parts of central services in a university – never mind the disciplinary differences across the university academic community.

Information Assurance Services (IAS) currently have a records management policy tabled for university approval. It might be described as corporate based and is not immediately identified with research. And yet if sensitive data were to be lost in a researcher’s lab notes the matter would probably reach IAS before any other body in the university (they also handle FoI requests in the first instance). IAS identify the lab notes as “research records” not “research data” so is a “research records management policy” then required?

From a research data viewpoint we might think of research records as being about the process including funder specific and personal information about the researchers rather than the research itself. Indeed we made exactly this distinction when following up on an external research data audit to all research staff: “Do you use, reuse or generate sensitive (including commercial in confidence) research data?” Researchers tend to assume that departmental and university administrators will be looking after the “research records”.

So let’s clarify how IAS see it. They incorporate information compliance and security (including FoI and environmental concerns), risk management and business continuity. The question then arises: is there clear legal ownership of research data? Many researchers are somewhat surprised to find out their hard work is actually “owned” by their institution for these purposes. This becomes particularly relevant when researchers move institution, as of course they often do.

So I found myself wondering aloud: do we need a “research managment policy” which then refers to the “records management policy” guidance and the rather more prescriptive “research data management policy”?

What are the requirements an RDMI has to cover?

The idea for this post came through a discussion Tom Parsons (ADMIRe project, University of Nottingham) started on the JISCMRD mailinglist about the ‘Requirements for an RDM system’. As we all know there are a lot of challenges involved in figuring out what requirements have to be covered by a Research Data Management Infrastructure. What is it an RDMI has to deliver in the end, what functional specifications have to be defined and on what scale?

Requirements have to be gathered on different levels from various researchers, research groups and other stakeholders. Approaches, case studies and outcomes are depending not only on projects’ remits (e.g. pilot projects vs. 2nd phase projects implementing an institutional wide service), but also on resources, buy-in from stakeholders and varying needs of the target audiences. The wider the field to play on, the harder it gets to balance out disciplinary specific and generic needs and to decide what parts of the research lifecycle can/have to be covered.

At the same time the institutional landscape has to be taken into account as well, especially the IT landscape and systems already existing – Tom pointed to the danger of “straying into the territory of a research administration system when we are thinking about an overall RDM system”. Trying to inter-connect with existing systems across the institution is something e.g. the MiSS project is doing at the moment, integrating existing infrastructure while adhering to the existing IT framework at the University of Manchester; this means the project initiation and accounting systems will be connected to the MiSS RDM core system (as will eScholar as the dissemination platform) and e.g. DMPs are automatically transferred into the active data stage. The temporary downside of this approach – creating an RDMI across the whole lifecylce, but with a thin layer for a service to start with – is that certain specific needs can only be addressed over time. Then again, such a new infrastructure needs to be able to evolve in use anyway.

To round up this post, here are some thoughts mentioned in the discussion:

Chris Morris pointed to a “concept of operations for research data management for a specific domain, molecular biology: http://pims.structuralbiology.eu/docs/beforePiMS.html” and also “got one general suggestion, which is not to start with use cases about deposition, but with retrieval. There is no scholarly value in a write-only archive.”

Some examples for use cases provided by Joss Winn: “https://github.com/lncd/Orbital-Core/wiki/Case-Studies. He remarks that those are very high level looking at them now and provides another link with “more detail from our project tracker, although it may not be as easy to follow: https://www.pivotaltracker.com/projects/366731. Basically, other than the usual deposit, describe, retrieve functionality, our two pilot research groups are looking to use the RDM platform for data analysis, such that is is a working tool from the start of the research project, rather than a system for depositing data at the end of the project.”

Steve Welburn addresses another important question, namely what technical framework to choose and how they came to choose DSpace: http://rdm.c4dm.eecs.qmul.ac.uk/platform_choice

Thanks to everyone mentioned for their links and thoughts!