Frequently Asked Questions
The San Francisco Declaration on Research Assessment (DORA)

Guidance

• 1. Why do we have new guidance?

  CIHR's approach to research assessment within the Project Grant competition already reflected many of the DORA principles, such as encouraging peer reviewers to consider a range of research outputs broader than published journal articles. However, CIHR heard from stakeholders that additional guidance was needed to support applicants and peer reviewers in the inclusion of a broad range of research contributions in their applications and assessments, respectively. That's why CIHR updated its program guidance to provide examples of more inclusive and expansive contributions to help in the crafting and assessment of applications. These updates encourage the assessment of the content and output/outcomes of research rather than the prestige of the journals in which researchers publish, including directing reviewers to not use journal-based metrics as surrogate measures of the quality of individual research publications.

• 2. What are the changes to applicant guidance or materials? Where can applicants highlight these research contributions and impacts?

  The application instructions for the Project Grant and Fellowship competitions have been updated to include additional DORA-related guidance to identify where applicants can highlight contributions and inform them of how DORA principles will be considered in the assessment of their applications. The funding opportunities has also been updated to include a notice that DORA principles should be considered in the crafting and peer review of applications.

  Depending on the funding opportunity, applicants can highlight a range of research contributions and impacts in their CV, Summary of Progress and/or in their Most Significant Contributions sections of their applications. This could include contributions such as: research publications, reports, books, guidelines, datasets, code, tools, training and mentorship, volunteerism, community engagement, standards, software, and commercialized products—and impacts such as how their work has influenced policy and practice, health outcomes, societal outcomes, and whether they have engaged in distinctions-based, meaningful, and culturally safe health research.

  Applicants can refer to the resource, How to highlight your research contributions and impacts, for tips related to these updates. These DORA-related updates will be communicated to the research community through various mechanisms, including applicant webinars, the Access newsletter, and social media.

• 3. What are the changes to peer reviewer guidance or materials?

  The Project Peer Review and Banting Postdoctoral Fellowships review guides have been updated to reflect the new DORA-related guidance. This includes directing peer reviewers to consider a range of contributions (e.g., publications, knowledge mobilization outputs, community engagement, guidelines, standards, software, commercialized products, books) and impacts (e.g., influence on policy and practices, health, and societal outcomes) in their assessment of applications. Reviewers are also now asked to consider the context of applicants (career stage, leave, pandemic impacts) and how this may have affected their productivity. There is also new guidance on the use of metrics in peer review.

  Reviewers can refer to the resource, Broadening your assessment of research contributions and impacts.

• 4. To what programs do the changes apply and when do they take effect?

  CIHR is aligning all of its funding programs with DORA's recommendations. The enhanced DORA-related guidance launched in July 2023 with CIHR's Fall 2023 Project Grant competition and CIHR's 2023-24 fellowships competition. CIHR will begin aligning its other funding programs (e.g., priority-driven, and other training/salary awards programs) throughout 2023–2024. For each competition, the inclusion of guidance aligned with DORA's recommendations will be clearly communicated within the funding opportunity, application, and peer review materials at the time of launch.

How to assess contributions

• 5. How will the assessment of contributions change?

  CIHR's approach to research assessment within the Project Peer Review Manual (2021) already reflected many of the DORA principles, such as encouraging the inclusion of outputs broader than published journal articles. The guidance updates further encourage the assessment of research quality and impact, rather than prestige, with instructions that journal-based metrics not be used in assessment of quality and impact. The updated materials provide examples of more inclusive and expansive contributions and impacts to help in the crafting and assessment of applications.

• 6. What is the concern with using journal-based metrics such as journal impact factor (JIF)?

  An overreliance on metrics-based assessment and the inappropriate use of journal-based metrics can introduce bias into the peer review system. The use of journal-based metrics in assessment has several limitations, for example:

  • The impact factor of a journal is derived from citations of all articles in a journal, thus this number cannot indicate the quality of the work of a specific author.^{Footnote 1}
  • The h-index favours researchers with longer careers and researchers in publication- and citation-dense fields and rewards bad publishing and referencing behaviour.^{Footnote 2}
  • Citation rates vary widely between research disciplines and the JIF does not consider the context of the research and findings. More importantly, the JIF does not take into account how well read and discussed the journal is outside of the scientific community, or the outputs that it informs (e.g., policy changes).

  Considering diverse contributions to research and avoiding the inappropriate use of journal-based metrics allows research to be assessed on its own merits, improves the quality of research and addresses inequalities in academia.

  Aligned with the growing body of literature that points to the negative effects of exclusively using metrics such as JIF, it is also important for reviewers to consider the context and limitations of quantitative metrics used and to balance them with the qualitative components of the proposal.

• 7. Are certain research contributions worth more?

  CIHR funds individuals in many health-related areas, and the forms of research contributions (including publications) can vary greatly among disciplines. CIHR acknowledges that there is not a one-size-fits-all approach to research and that different disciplines and environments have different opportunities and expectations for research contributions and other activities.

  Research contributions should be considered equally (regardless of the output) when assessing quality and impact. Reviewers must be aware of unintended biases (e.g., seniority or discipline) that may result in consideration of certain outputs, such as publication in high-impact journals being considered the "gold standard," while other outputs, such as knowledge mobilization activities, are considered "second class" or "grey literature."

• 8. Should ratings be lower if applicants include only journal articles, papers, citations and associated metrics?

  All contributions including publications are valued when contextualized by the applicant using indicators of quality and demonstration of impact on the applicant's research.

  CIHR acknowledges that there is not a one-size-fits-all approach to research assessment, and that different disciplines and environments have different opportunities and expectations for research contributions and other activities. As such, applicants should not be penalized for focusing on historically valued contributions (e.g., publications, reports, books, etc.) if peer reviewers deem them appropriate. However, applicants are encouraged to think more broadly about their research contributions and impacts. For example, while some researchers have a strong and impactful training/mentorship program, others are deeply engaged with their community, and others are focused on the creation of guidelines, standards, and tools for public use. CIHR's approach is to increase awareness and acceptance of these broader contributions as equally valuable and worthwhile.

• 9. How do we take an applicant's context into account?

  CIHR acknowledges that there is not a one-size-fits-all approach to research and that researchers have different contexts and have had different opportunities. In addition,^{Footnote 3} systemic barriers within academia and the research ecosystem may influence applicant ability to fit into the mold of the "ideal researcher." When peer reviewers are considering an applicant's achievements, they should consider achievements relative to the applicant's context. Peer reviewers should consider that there are legitimate causes for effects on and delays to research programs, for example:

  • Career interruptions such as childrearing, caregiving, or illness, can influence opportunity for knowledge production, publications, and other research-related activities;
  • Different disciplines and environments offer different opportunities for research contributions, publication, and other research-related activities;
  • Institutions in Canada have different budgets that can influence the infrastructure to support grant and award applications and publications;
  • Co-developed, distinctions-based, patient-oriented, meaningful, and culturally safe health research takes considerable time, effort and care, which can impact a researcher's opportunity for publications and other research-related outputs;
  • Researchers who take a non-traditional career path may have research track records that are different from the "norm";
  • Equity-deserving scholars may participate in more committees or in extensive community-based research, both activities that should be considered equal to research contributions; and
  • Researchers from underrepresented, rights-holding and/or equity-deserving groups face systemic barriers to entry that compound over time to affect scientific output and opportunities for development.

  These factors, among others, must be considered when assessing the productivity, progress, and impact of an applicant's research program.

• 10. How do we assess productivity, progress and impact?

  In alignment with DORA, CIHR directs reviewers to assess productivity and progress broadly, by considering a range of contributions and impacts. This acknowledges that productivity, progress, and impact can be demonstrated in many ways.

  Peer reviewers must assess the appropriateness of the complement of expertise, experience, and resources among the applicants (Nominated Principal Applicant, Principal Applicant(s) and Co-Applicant(s)), and their institutions/organizations, as it relates to the ability to collectively deliver on the objectives of the project.

• 11. How does the Tri-Agency CV support excellence in research funding and responsible research assessment?

  CIHR is committed to excellence in research funding and responsible research assessment practices. An important part of that is recognizing a wide range of research outputs in the review process and ensuring that applicants’ work is assessed on its own merits.
  

  In alignment with the San Francisco Declaration on Research Assessment, CIHR wants to ensure that peer reviewers look beyond the traditional indicators of productivity when assessing contributions and impacts. Peer reviewers are asked to avoid using metrics (e.g., number of publications and citations; size/number of research grants) in isolation and to avoid using journal-based metrics (e.g., Journal Impact Factors) as surrogate measures of quality and impact.

  The Tri-Agency CV allows applicants to capture, by providing indicators of quality and impact, and provide context to their relevant contributions in a narrative manner to support peer reviewers in assessing their track record.

Date modified:

2024-02-26