A toolkit that can be used to collect feedback on the effects of involvement and engagement initiatives in research projects. The toolkit consists of a set of different practical tools that can be used at different times along the research project to plan and evaluate the project's societal impact through both quantitative and qualitative methods. The toolkit includes detailed guidelines, templates for questionnaires and interviews and more. Together, they provide a comprehensive picture whether and in how far the impact elements of the "OIS Impact Model" were achieved in an open, collaborative and participative project.
The OIS Impact Reflection Instruments do not claim to be complete, but provide a comprehensive foundation that can be easily and flexibly adapted to the project’s needs, complement existing instruments or provide a basis for individual approaches.
OVERVIEW OF INSTRUMENTS
The following instruments are part of the OIS Impact Reflection Instruments toolkit:
1. Participation check: Are all involved Experts of Practice satisfied with the participation process? Are the quality criteria of involvement fulfilled? The toolkit includes a template for a short, written questionnaire at the end of meetings, workshops or other activities and events along the participation process: the Participation check. By evaluating participation through this questionnaire, the project team can identify lack of satisfaction or other grievances in the participation process at an early stage.
2. Desk Research: A guideline with suggestions for data to be collected in desk research is included in the toolkit. Desk Research includes every analysis that can be made "sitting at a desk", including analysis of social media, other websites, existing documents, participants lists, etc. The collection of data can be carried out continuously and/or collected at specific time points, while the analysis should take place in regular time intervals (e.g. once per year). Desk research methods have the benefit of being flexible and rather low-cost, but should ideally be combined with other methods to increase their informational value.
3. Qualitative Interviews with Experts of Practice: In the toolkit, you can find a template for qualitative interviews with Experts of Practice meant to be carried out in the first half of the project. Qualitative interviews will give valuable insights in the experience of the involvement process. Besides using the interviews for impact analysis, the results can be used for adapting/improving the project. You can also use the results of the qualitative interviews to identify additional impact areas (to be measured in the survey at the end of the project). Qualitative interviews need to be carried out by a neutral person (no member of the research team or research organization).
4. Qualitative Interviews with Researchers: A template for the interview guideline can be found in the toolkit. Same as the qualitative interviews with Experts of Practice, these interviews are meant to be carried out in the first half of the project. Results can be used in the same way.
5. Guided Team Reflection: The toolkit includes a guideline for discussion items for a Guided Team Reflection, which is an internal reflection meeting of the researchers' team responsible for designing the research process. The meeting should be carried out in a later stage or at the end of the project and should be around 3 hours long. Ideally, the reflection meeting is facilitated by a neutral person. The results of the reflection meeting are meant to be documented. You can also use the results to identify implications for further research projects (e.g. improve future involvement processes).
6. Experts of Practice Survey: In the toolkit, you can find a template for a questionnaire to be filled out by Experts of Practice at the end of the project (preferably online). The survey focuses on quantitative results. However, we suggest the use of additional "comment options" (open fields) to allow deeper insights. The survey should take around 10 minutes. Whenever possible, the Experts of Practice Survey and the Researchers Survey (see below) use similar questions and items. This way, perspectives of scientific and non-scientific participants can be compared.
7. Researchers' Survey: A template for the questionnaire can be found in the toolkit. Same as the qualitative interviews with Experts of Practice, the survey is meant to be carried out at the end of the project. Results can be used in the same way.
8. Focus Group: The toolkit includes a guideline and question template for a focus group of Experts of Practice that is carried out around a year after the project has ended. The aim of the focus group is to identify long-term effects of the participation in the research project. The duration of the reflection meeting should be around 1,5 hours and should be carried out by a neutral person (no member of the research team) to reduce bias.
You can find all templates and guidelines for each instrument in the toolkit.