Bendiscioli, Sandra; Firpo, Teo; Bravo-Biosca, Albert; Czibor, Eszter; Garfinkel, Michele; Stafford, Tom; et al. (2022): The experimental research funder’s handbook (Revised edition, June 2022, ISBN 978-1-7397102-0-0). Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.19459328.v2
Across the research funding community, there is a growing appetite for more sophisticated approaches to evidence gathering and experimentation, to inform decision-making. Research funders are at different stages of this journey, and don’t always benefit as much as they might from the experiences of others, or from the latest academic work in this field. This Handbook aims to provide a practical resource for research funders looking to move further or faster down an experimental path.
The Handbook synthesises insights from funders within the Research on Research Institute consortium that have conducted trials with new approaches to review, allocation and evaluation. From these accounts, we have assembled practical descriptions of how to design and implement experiments with funding processes.Thanks to direct contributions from the funders involved, we hope to shine fresh light on the challenges they have experienced and the lessons they have learned. In line with RoRI’s mission to support more dynamic, diverse and inclusive research systems through evidence and experimentation, our hope is that this Handbook will help funders in designing effective interventions that can be properly evaluated, and yield robust findings and insights of relevance to others.
An initial version of this Handbook was published as a working paper in December 2021, and formed the basis of a two-day workshop, attended by around twenty-five research funders - a mix of public funding agencies and private foundations. Their reflections informed this final version, published in June 2022, alongside the launch of RoRI's second phase. It will also influence the design of a next wave of funder experiments that RoRI aims to undertake with its partners over the next two to five years.
TABLE OF CONTENTS
Part 1 - The case for experimental research funding
2. Why experiment?
3. Tools for experimenting with research funding
3.1 Tools to diagnose
3.2 Tools to design
3.3 Tools to evaluate
4. Becoming more experimental
4.1 Attracting applicants
4.2 Selecting reviewers
4.3 Assessing proposals
4.4 Making funding decisions
Part 2 - Funder experiments with partial randomisation
2. An introduction to partial randomisation
2.1 What is partial randomisation?
2.2 Why partial randomisation?
2.3 Arguments for and against partial randomisation
3. Case studies: Funder experiments with partial randomisation
3.1 Volkswagen Foundation
3.2 Swiss National Science Foundation (SNSF)
3.3 Austrian Science Fund (FWF)
3.4 Health Research Council of New Zealand
4. A comparison of experiments with partial randomisation
Part 3 - Checklist
1. A checklist for funder experiments with partial randomisation