Review of diversity and inclusion literature and an evaluation of methodologies and metrics relating to health research. Duncan Chambers, Louise Preston, Anna Topakas, Stevienna de Saille, Sarah Salway, Andrew Booth, Jeremy Dawson & James Wilsdon. Working paper - May 2017.
University of Sheffield in partnership with the Wellcome Trust.
Over the past decade, the need for greater diversity and inclusion across research systems and institutions has received greater emphasis from policymakers, funders, universities, learned societies and wider stakeholders. In this context, the impetus being placed on diversity and inclusion in Wellcome Trust’s latest strategy is timely and important.
In support of this strategy, the primary aim of this project is to undertake a systematic and critical review of the evidence base for a positive relationship between a diverse and inclusive health research community, and the qualities and impacts of the research they undertake. The review draws on evidence from across the research system, with a primary focus on health and biomedical research. It also draws on related literatures on diversity, inclusion, equality and coproduction across health systems and services, and organisational diversity and inclusion.
A second aim of the project is to evaluate the efficacy of the metrics used to measure diversity, inclusion, quality and impact in health research, and the relationship between these metrics and wider agendas for diversity and inclusion.
A mapping review was selected as the most appropriate approach for a wide-ranging consideration of diversity and inclusion across the health research system. Coding for the mapping review was undertaken using the PROGRESS-Plus framework, developed by the Cochrane Equity Group for analysis of equality and diversity issues in health. From an initial set of 1466 studies, the mapping exercise generated 246 papers for detailed analysis. This evidence was supplemented by three qualitative institutional case studies; a stakeholder workshop; and a targeted look at evidence for the relationship between research metrics, diversity and inclusion.
From the review process, we have identified ten broad conclusions about the state of
the evidence base, and gaps that persist:
There is a strong dominance of US-based research in the literature, which raises questions about the transferability of findings, given the cultural specificity of some aspects of diversity and inclusion.
There is a far more extensive literature relating to gender and race/ethnicity (although the latter also related predominantly to the US), and comparatively little on other PROGRESS-Plus axes of difference. The literature highlights persistent patterns of disadvantage, but also variability by field and subfield – particularly with regard to gender.
The majority of the studies we examined focused on clinical or biomedical research. Other areas of health-related research did not feature so heavily. Given that the relevance of the PROGRESS-Plus variables differs depending on the type of health research, and on the sub-cultures and degree of diversity within health research disciplines, the transferability of evidence across disciplines is debatable.
The predominant level of analysis is that of individuals (in terms of metrics, interventions etc.). Multiple (dis)advantages and inter-locking aspects of people’s experience can reinforce one another. The degree of isolation and exclusion felt by women and minorities can be underestimated. There is a relative lack of attention paid in the literature to measures of diversity or inclusion at the aggregate or organisational level.
There is a focus on individual parts of the health system, and only a few examples of more holistic, systems-based and/or longitudinal approaches that try to examine how elements interplay and (re)create disadvantage.
The literature predominantly takes a national, rather than international or comparative focus, despite the fact that dimensions of diversity and inclusion look very different from a more international or global perspective.
The studies we examined reflect a limited amount of theoretical framing, and often rely on implicit assumptions about mechanisms of action and causality, rather than more explicit development and testing of models and mechanisms.
There are persistent areas of controversy and complexity, such as how to conceptualise and operationalise race/ethnicity. These demand careful and explicit consideration.
Trickle-down or trickle-out effects to other parts of health research systems is far from automatic: this takes time, and requires actions to promote diversity and inclusion across all elements of the system. Gender-related initiatives can be seen as benefiting women only. Informal processes can reinforce the advantages of dominant groups. Mentoring schemes that pay attention to culture and tacit knowledge, rather than simply skills, seem more promising
Finally, we can conclude that there is limited available evidence that directly addresses the guiding research question of this project; and a relatively weak evidence base for processes and explanations of patterns of inequality, exclusion or lack of diversity that are visible in the health research system.
Based on this review, we identify the following recommendations for future research and related activities that Wellcome Trust could support (on its own, or in partnership with others) in order to strengthen the evidence base in these areas:
First, there needs to be greater investment in comprehensive studies that examine interactions across the health research system, and longitudinal studies that look at changes over time at individual, collective and institutional levels.
Second, more work is required to improve comparability across studies, to define and standardise indicators and metrics; and to collect data in consistent ways.
Third, there needs to be greater experimentation and research investment in neglected aspects of diversity and inclusion, including: aggregate measures of inclusion; axes of difference and disadvantage beyond gender and race/ethnicity; enablers and obstacles; and diversity and inclusion across health research systems.
Fourth, to achieve this more systemic perspective, there need to be closer links between future research on diversity and inclusion in health research, and issues relating to research cultures, career pipelines, reward and recognition structures, responsible metrics and research integrity – increasingly addressed under the broad umbrella of the “science of science” (Ioannidis et al., 2015; Wilsdon et al., 2015). Given the existing portfolio of Wellcome Trust activities, there is scope for Wellcome Trust to pioneer creative and ambitious funding, policy and advocacy strategies that draw links between these (at times) disparate and siloed agendas, to advance a more holistic understanding of links between diversity, inclusion, integrity, responsibility and public engagement.
Finally, to help inform Wellcome Trust’s future efforts across these linked agendas, we offer an illustrative synthetic model that seeks to highlight the importance of taking a more holistic, less compartmentalised approach than we found evident in much of the literature. We hope this is helpful in suggesting future priorities and opportunities for research.