Instructions for Evaluators

Evaluators are crucial to the success of these policies. Overall, evaluators need more training in SG&DA. Trainings are found in Section 4 of these webpages.

The European Commission and German Research Foundation (DFG) score SG&DA under the ‘excellence’ or ‘intellectual merit’ criteria for research design. The European Commission states under 'Excellence', (p. 4): The following aspects will be taken into account, to the extent that the proposed work corresponds to the description in the work programme:

• • Clarity and pertinence of the project’s objectives, and the extent to which the proposed work is ambitious, and goes beyond the state of the art.
• • Soundness of the proposed methodology, including the underlying concepts, models, assumptions, inter-disciplinary   approaches, appropriate consideration of the gender dimension in research and innovation content, and the quality of open   science practices, including sharing and management of research outputs and engagement of citizens, civil society and end   users where appropriate.

To be successful, agencies must instruct evaluators to consider sex, gender and/or diversity analysis across all stages of the research process. The Canadian Institutes of Health Research (CIHR) found that ‘targeting applicants alone to adopt new sciences policies without concomitant pressure by evaluators…may not be effective.’ Since 2018, CIHR’s assessment forms instruct reviewers to rate applicants’ integration of sex and gender as a strength or weakness of the proposal, and to provide recommendations for improvement: Key considerations for the appropriate integration of sex and gender in research. This information is also available as a 4.5 minute video: Assessing Sex and Gender Integration in Peer Review. Each application is evaluated by three independent evaluators; applications that receive the top score from at least two evaluators is considered high quality (Haverfield & Tannenbaum, 2021).

The Irish Research Council provides a guide for assessors (see pages 9-10; 22-24).

Funders, such as the NRF, Republic of Korea, provide applicants and evaluators similar forms and instructions for consistency across the research process. Funders with multi-step grant proposal forms can create aligning multi-step evaluation forms.

Some agencies, such as the EC and the DFG, are limited in the overall instructions they can provide on this particular requirement given the number of topics that need to be covered. These agencies may provide ‘good research guides’ that reference assessing SG&DA alongside other elements of peer review, such as ethics and reproducibility (German Research Council, 2019).

The agency or sector heads must monitor the evaluation process to confirm that SG&DA is addressed in reviewer comments and that those comments are high quality. CIHR, for example, samples 5% of reviewer rationale for their SG&DA ratings (Haverfield & Tannenbaum, 2021).

Works Cited

German Research Council, Guidelines for Safeguarding Good Research Practice: Code of Conduct. download the pdf (2019).

Haverfield, J., & Tannenbaum, C. (2021). A 10-year longitudinal evaluation of science policy interventions to promote sex and gender in health research. Health research policy and systems, 19(1), 1-12.

Previous page | Next page