Evaluation of Policy Implementation

Very few agencies in our study have performed policy implementation evaluations. We recommend that agencies implement evaluation plans as they develop policies.

Evaluations found, for example:

  • • Confusion of sex and gender analysis in research (knowledge issues) with gender equality on teams (participation issues)
  • • Conflation of the terms sex and gender (Haverfield & Tannenbaum, 2021)
  • • Failure to evaluate for intersectionality of sex and gender with other diversity factors.

We recommend a multi-part evaluation, reporting:

1. The number and proportion of grants that include SG&DA. CIHR found that from 2011 to 2019, the proportion increased from 22% to 83% for sex analysis and from 12% to 33% for gender analysis. The level of integration differed across sectors with the lowest in biomedical and the highest in clinical research (Haverfield & Tannenbaum, 2021). An independent study of the NIH found that applicants who addressed SABV in their experimental design, analysis, and reporting rose from 51% in 2016 to 66% in 2017 (Woitowich & Woodruff, 2019).

In a thorough evaluation of their postdoctoral funding program, the Irish Research Council(IRC) found that the likelihood of including sex and gender analysis in a research proposal depended on the field, with those in arts, humanities, and social sciences more like to do so, those in biological and medical sciences next most likely, and those in physical sciences and engineering least likely. This also differed by gender with women more likely than men less likely to include the gender dimension (Irish Research Council, 2021). To overcome these disparities, the IRC recommend developing more training with specific examples for the fields that are lagging.

2. The quality of SG&DA in proposals. The EC conducted a mid-term evaluation of Horizon 2020 in 2017, including the quality of the gender dimension. They considered methods, impacts, dissemination, and whether the project had moved sex or gender methodology ahead in that field. They concluded that the quality of the gender dimension in project proposals was not high and that more training was needed.

For agencies where SG&DA is a separate question, agency policy evaluators may monitor the quality of reviewers’ work by checking correlations between reviewers’ scores and the quality of applicants’ proposals. This may also be used to measure evaluation quality across different funding streams (Science Foundation Ireland, 2020).

CIHR built agency assessment of the quality of SG&DA in proposals into the review process. As noted above, CIHR requires evaluators to rate the sex and gender aspects of proposals and to provide a rationale for that rating (Haverfield & Tannenbaum, 2021).

3. The quality of evaluators’ scoring and comments. CIHR manually sampled 5% of evaluators’ comments to check the quality of responses (Haverfield & Tannenbaum, 2021).

The EC reviewed the effectiveness of review panels and found that only 36% considered the gender dimension and of those 70% included a gender expert, suggesting that review panels require guidance from experts (European Commission, 2017). The EC experimented with computer-assisted textual analysis given the volume of applications per year (European Commission, 2017). These methods are in their infancy and might be developed further.

An external review of US NIH found that, in 2017, 88% of reviewers felt confident that they understood the SABV policy, but only 68% thought that SABV was important for all US NIH funded research (Woitowich & Woodruff, 2019). This study did not evaluate the quality of reviewers’ evaluations.

4. The number of applicants, evaluators, and staff who engaged in trainings and what type of training. If possible, the correlation between trainings attended and the success of proposals submitted post-training should be assessed. Some funders, such as the Spanish Carlos III Health Institute, reported that they monitor the number of participants in SG&DA training and are setting targets to improve this over time. The IRC evaluated applicants' responses to training and found that most wanted examples specific to their discipline (Irish Research Council, 2022).

5. The number and proportion of peer-reviewed publications or other recognized modes of dissemination that result from grants that incorporated SG&DA. To monitor this, funders will need to track papers and research outputs using grant numbers. Science Foundation Ireland (SFI) reported in our study that they collect researcher-reported publication data to check that proposals that included sex and/or gender analysis reported that dimension in publications. This will allow SFI scientific program managers to raise any concern at the mid-term award review.

Through their review process, CIHR found other correlations of note that we did not evaluate. They found that applicants who scored well on the sex/gender question scored well overall, i.e. sex/gender analysis improved the overall quality of the proposal, and applicants were more likely to get funded (Haverfield & Tannenbaum, 2021). Further, consistent with other literature (Nielsen et al., 2017), they found that women applicants are more likely to integrate sex and gender analysis into their proposals.

Works Cited:


Arnegard, M. E., Whitten, L. A., Hunter, C., & Clayton, J. A. (2020). Sex as a biological variable: a 5-year progress report and call to action. Journal of Women's Health, 29(6), 858-864.

European Commission. (2017). Interim evaluation: Gender equality as a crosscutting issue in 2020. Publications Office of the European Union, Annexe 1.

Hunt, L. & Schiebinger, L. (2022). Global review of sex, gender, and/or diversity analysis in research policies of major public granting agencies.

Haverfield, J., & Tannenbaum, C. (2021). A 10-year longitudinal evaluation of science policy interventions to promote sex and gender in health research. Health research policy and systems, 19(1), 1-12.

Irish Research Council, Review of the IRC Gender Strategy and Action Plan, 2022.

Nielsen, M. W., Andersen, J. P., Schiebinger, L., & Schneider, J. W. (2017). One and a half million medical papers reveal a link between author gender and attention to gender and sex analysis. Nature human behaviour, 1(11), 791-796.

Science Foundation Ireland, Call for Submission of Proposals. https://www.sfi.ie/funding/funding-cal ls/frontiers-for-the-future/SFI-Frontiers-for-the-Future-Programme-2020-Call-Document.pdf (2020).

Tannenbaum, C., Ellis, R. P., Eyssel, F., Zou, J., & Schiebinger, L. (2019). Sex and gender analysis improves science and engineering. Nature, 575(7781), 137-146.

Tannenbaum, C., & Van Hoof, K. (2018). Effectiveness of online learning on health researcher capacity to appropriately integrate sex, gender, or both in grant proposals. Biology of sex differences, 9(1), 1-8.

Woitowich, N. C., & Woodruff, T. K. (2019). Implementation of the NIH sex-inclusion policy: attitudes and opinions of study section members. Journal of Women's Health, 28(1), 9-16.

 

 

Previous page | Back to policy page

 

 

 

double logo double logo double logo

 

TermsSite Map