CIHR Peer Review Manual for the Knowledge Translation Prize and the Betty Havens Prize

Updated: January 2014

Table of Contents

  1. Purpose of the Manual
  2. Peer Review at CIHR
  3. Principles of Peer Review
  4. Policies Impacting Peer Review
  5. Peer Review Committee Members
  6. Application Review Process
  7. Rating of the Applications

1. Purpose of the Manual

On behalf of CIHR, we would like to thank you for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like yourself who generously give of their time and expertise, and your efforts are greatly appreciated by CIHR and the scientific community.

The peer review process is described in detail in this manual and on CIHR's website. It is essential that committee members read and be familiar with this Manual and the Funding Opportunity for which you peer review. Concise information on the role of each committee member and their responsibilities is also available on the CIHR website: please refer to the Peer Review Committee Members Role page.

The purpose of this manual is to provide information on CIHR's objectives, governance and policies; to outline the roles and responsibilities of peer review committee members evaluating the Knowledge Translation Prize and the Institute of Aging Betty Havens Prize for Knowledge Translation in Aging applications; and to define the policies and procedures for peer review of the Knowledge Translation Prize and the Institute of Aging Betty Havens Prize for Knowledge Translation in Aging applications.

This manual is addressed primarily to committee members, but is also of use to applicants in explaining the peer review process from submission to final judgment.

2. Peer Review at CIHR

The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:

"To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system."

The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also ensures accountability, not only to the Government of Canada and the Canadian taxpayer - the source of CIHR funding - but to the research community at large. Peer review is carried out by committees of experts that encompass all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health).

Peer review is overseen by CIHR's Science Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with CIHR's 5-year Strategic Plan. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC's responsibilities.

For more information on the different types of Peer Review and meeting formats, please refer to Types of Review at CIHR.

3. Principles of Peer Review

3.1 Confidentiality

Confidentiality is information about a person that shall not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. Thus, information contained in applications, reviewer reports, names of reviewers and committee discussions are all strictly confidential. The use of this information for any other purpose than what is outlined here is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.

Committee members must not discuss with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the chances of success or failure. Applicants must not contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank within committee, etc.). All requests for information on an application or a reviewer report should be referred to the Deputy Director at CIHR responsible for the committee in question.

By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision and CIHR will not edit the reviews provided. The identity of the reviewers will not be revealed to the applicants under any circumstances. However, a list of peer review committee members will be published on the CIHR website 60 days after the Scientific Council approves funding for a competition cycle.

3.2 Conflict of Interest

CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a Conflict of Interest means a conflict between a Participant's duties and responsibilities with regard to the Review Process, and a Participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the Participant:

  1. would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
  2. has a professional or personal relationship with an Applicant or the Applicant’s institution;
  3. has a direct or indirect financial interest in a funding opportunity or application being reviewed; or
  4. is currently under investigation for an alleged breach of Funding Organization policies.

A conflict of interest may be deemed to exist or perceived as such when review committee members, external reviewers or observers:

  • are a relative or close friend, or have a personal relationship with the applicants;
  • are in a position to gain or lose financially/materially from the funding of the application;
  • have had long-standing scientific or personal differences with the applicants;
  • are currently affiliated with the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
  • are closely professionally affiliated with the applicants, as a result of having in the last six years:
    • frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
    • been a supervisor or a trainee of the applicants;
    • collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
    • been employed by the institution, when an institution is the applicant; and/or
  • feel for any reason unable to provide an impartial review of the application.

All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff and the Chair are responsible for resolving areas of uncertainty during the committee meeting.

All committee members must read and agree to abide by the COIC policy prior to viewing any application information.

3.3 Fairness

Success of the peer review system is critically dependent upon the willingness and ability of all committee members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application. In programs where written reviews are required, these reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that your review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, which includes helping the applicant by pointing out deficiencies that could be repaired in a resubmission, will help to convince a disappointed applicant that you provided a fair assessment of the proposal.

4. Policies Impacting Peer Review

4.1 International Collaborations

As stated in the CIHR Act, one of the ways CIHR fulfills its mandate is by "pursuing opportunities and providing support for the participation of Canadian scientists in international collaborations and partnerships in health research." As a result, CIHR accepts applications for research to be carried out in, or in collaboration with applicants based in, other countries. The international nature of the research should not be a factor in the scientific assessment of the proposal, beyond how it relates to the feasibility of the proposed research and the quality of the research question. Reviewers should also not be influenced by the funding obtained or requested for the international components when recommending a budget for the Canadian component(s). For detailed information on applying for funding with an international partnership component, please refer to the subsections titled Global Health Research and International Collaborations in the Grants and Awards Guide.

For detailed information on how CIHR supports international collaborations and global health research, please view Internationalization of CIHR funding policy and program tools.

4.2 Knowledge Translation

Knowledge translation is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants.

For end of grant KT, many means of dissemination exist and the onus is on the researcher to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified. Please consult About Knowledge Translation for more information.

4.3 Open Access

Applicants and peer reviewers are reminded that the CIHR Open Access Policy applies to all new and renewed funding awarded after January 1, 2008. Researchers must ensure that all published peer-reviewed articles are freely available online within twelve months. Authors can adhere with the policy by archiving peer-reviewed manuscripts in an open access repository (e.g. institutional repository) or by publishing in an open access journal. Please consult the policy web site for more detail.

4.4 Gender, Sex and Health Research

Applicants are encouraged to demonstrate the use of gender and sex-based analysis in applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands understanding of health determinants in both sexes and results in improvements in health and health care. For more information on how peer reviewers can assess whether gender and/or sex are appropriately integrated into CIHR applicants' proposed research designs, please refer to Integrating Gender and Sex in Health Research: A Tool for CIHR Peer Reviewers.

4.5 Official Languages

Federal agencies are required to take positive measures to ensure the support and recognition of minority language communities in Canada. For CIHR, this means an obligation to promote health research in and for these communities. For further information, please refer to the CIHR Policy Statement on Official Languages. Research proposals in these areas should still be subject to the same rigorous peer review process as any other application.

4.6 Publications and Productivity

An important evaluation criterion in all grant programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing the quality of publications, peer review committees should focus on the quality of a publication's content and not simply the number of publications nor the quality or impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.

CIHR funds researchers in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."

When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer review committees must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.

5. Peer Review Committee Members

A typical CIHR peer review committee consists of a Chair, Scientific Officer, peer reviewers, CIHR staff and other specialized roles depending on the funding opportunity. Individual committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment. For more details, please refer to the Peer Review Membership Guidelines. Committees as a whole should also satisfy the need to cover the range of research areas for which the committee is responsible, to appropriately represent the Canadian health research community, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members. For more details, please refer to the Procedure for Selection of Peer Review Committee Members.

CIHR competitions can be held on a recurring or an ad hoc basis. For recurring competitions, standing peer review committees are formed, and committee members are recruited for a term of service (typically three years) in order to ensure consistency and continuity in the review process. Standing committee membership may be supplemented by additional members as required for a specific competition, based on the applications received and expertise needed for their review. For ad hoc competitions, committees are formed to review applications for that particular competition and then disbanded.

For further information on peer reviewers at CIHR, please consult the Peer Review Committee Members Role page.

Complete instructions for peer review committee members are available at the Instructions for Committee Members page.

6. Application Review Process

6.1 Relevance Review

The Relevance Review Process is used by strategic leads and/or partners to assess the alignment of an application with a specific research theme described in the funding opportunity (FO). As the name implies, the process is used when it is important for applications to be relevant to (or in alignment with) targeted research components of the FO.

Applications submitted to the Betty Havens Award for KT in Aging will be reviewed for relevance prior to peer-review and all applications deemed not relevant will be withdrawn from the Betty Havens Award for KT in Aging competition. All applications submitted to the Knowledge Translation Award competition, which have agreed to the use and disclosure of full application and nominative information for relevance review and funding decisions will be considered for any additional partner funding available through the Knowledge Translation Award competition.

6.2 Peer Review Overview

All eligible applications received by the appropriate deadline date (posted in the competition announcement) are entered into the competition. Applications must be complete at the time of submission; otherwise they are withdrawn from the competition. Specific exceptions to this rule can be found in the funding opportunity descriptions.

The selection process for applications is comprised of two steps:

  1. Pre-ranking and review of assigned applications
  2. Peer Review of selected applications

6.3 Pre-Ranking Stage (Stage 1)

6.3.1 Pre-Ranking and Review of Assigned Applications

After the list of applications is compiled, committee members are given access to the application summaries on ResearchNet to declare any conflicts of interest and indicate their level of expertise. Each application will be assigned to a minimum of two (2) reviewers. Upon receipt of the ratings, CIHR will identify the most competitive applications which will be considered for the next step of the selection process. This will allow the committee to focus on the applications that have the highest probability of receiving the award.

6.3.2 Reviewing Applications During the Pre-Ranking Stage

It is the responsibility of all peer reviewers to familiarize themselves in advance of the meeting with all applications to be assessed by their committee, as well as with any available external reviewer reports. This will assist peer reviewers to prepare their internal reviewer's reports. Details on what to include in the report can be found in Section 7.3.

As part of the pre-ranking process, internal reviewers are required to complete the following tasks on ResearchNet in regards to their initial assignments:

  1. upload reviews;
  2. provide an initial rating for each application reviewed (note that the initial ratings submitted will be used to establish a subset of applications to review during Stage 2 of the selection process)

The deadline for uploading reviews to ResearchNet, along with assessments of overall quality and initial ratings, is two weeks before the meeting date. Reviews can be saved as drafts, by selecting "save draft copy" on ResearchNet prior to submission. In order to access the other reviews for the applications that you were assigned, you must select "submit final review" on ResearchNet. Afterwards, you will no longer be able to modify it prior to the meeting. If you wish to revise your reviews after the committee meeting, you will have one week to directly upload your changes to ResearchNet.

Please contact your committee coordinator if you encounter any technical issues.

6.4 Peer Review of Selected Applications (Stage 2)

Peer Reviewers who are not in conflict with any of the selected applications will be asked to attend the peer review committee meeting. These applications will be discussed in depth and compared against each other and ultimately the recipient of each funding opportunity will be chosen by the committee. The same respective evaluation criteria, listed below, will be used when discussing these applications.

An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This discussion provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.

6.5 Flagging of Applications

Any concerns in the following areas should be discussed and, if necessary, flagged for CIHR staff to address. These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please see the Grants and Awards Guide.

  • Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria to receive CIHR funding.
  • Relevance: While anonymized versions of all Betty Havens Award applications have been reviewed by the Institute of Aging and deemed relevant for the award, reviewers should raise any concerns with respect to relevancy to CIHR as soon as possible.

6.6 After the Meeting

As soon as possible after the peer review committee meeting, CIHR staff generates a funding proposal based on committee recommendations, to be reviewed by the Chief Scientific Officer and the Chief Financial Officer. The CSO and CFO consider the funding recommendation in light of criteria established by SC and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications page as soon as it is available.

Nominators and nominees are informed of the results of the competition once CIHR's Scientific Council has given its approval of the results of the competition.

7. Rating of the Applications

7.1 Types of Applications

Applications are being reviewed for two funding opportunities:

  • Knowledge Translation Prize (which may include additional secondary partner funding)
  • Institute of Aging Betty Havens Prize for Knowledge Translation in Aging

In some cases, the same peer review committee may review applications for more than one funding opportunity. At the conclusion of the committee meeting, these applications will be separated into their own overall ranking lists and funding decisions will be made based on the funding envelopes provided through their respective programs. Therefore, the presence of these applications will have no impact on the funding of other applications being reviewed by this committee.

7.2 Evaluation Criteria

The evaluation criteria for the Knowledge Translation Prize and the Institute of Aging Betty Havens Prize for Knowledge Translation in Aging are provided below:

Knowledge Translation Prize

  • To what extend does the application demonstrate evidence of influence on the synthesis, dissemination, exchange and ethically sound application of knowledge in Canada and/or abroad?
  • To what extent does the application demonstrate significant improvement of the health of individuals in Canada and/or abroad, more effective health services and products, and/or strengthening of the health system?
  • How strong are the letters of support for the nominee(s)?

Institute of Aging Betty Havens Prize for Knowledge Translation in Aging

  • To what extend does the application demonstrate evidence of influence on the synthesis, dissemination, exchange and ethically sound application of knowledge at the local or regional level?
  • To what extent does the application demonstrate significant improvement of the health of individuals in Canada, more effective health services and products, and/or strengthening of the health system?
  • How strong are the letters of support for the nominee(s)?

7.3 Internal Reviewer Report - Format

  1. A brief synopsis of the proposal
  2. An assessment of the proposal, based on the evaluation criteria as presented in the funding opportunity details:
    • consider all factors and the strengths or weaknesses of the applications in relation to each criteria;
    • emphasis may be placed on specific criteria in the funding opportunity details, in order to meet funding program objectives; consider this when formulating your rating;
    • focus your comments on the factors most relevant to your rating.
  3. If necessary, comments on issues the reviewer feels should be flagged Section 6.5. These concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the application.

The review should be clear and concise, using objective and non-inflammatory language, and include justification. Constructive advice to the applicant will allow him/her to improve the quality and efficiency of the proposed research. The applicant will receive the review as it is submitted by the reviewer. For this reason, please do not identify yourself in order to ensure the confidentiality of the review process.

7.4 The Rating

Criteria to assess the application are as described above in Section 7.2. The relative weighting of these criteria depends on the program objectives as described in the funding opportunity description; if in doubt, please contact the Deputy Director responsible for the committee.

To ensure consistency, all reviewers must adhere to a common scale. It is particularly important that the full scale be used and the same conventions applied to assigned ratings. To facilitate this, the following scale and descriptors should be used:

Descriptor1 Range2 Outcome
  1. Only applications rated 3.5 or higher are eligible for CIHR funding (including partner funded programs).
  2. In the committee meetings, reviewers assign scores to one decimal place, but the final average rating is calculated to two decimal places.
Outstanding 4.5 - 4.9 May Be Funded - Will be Discussed by the Committee
Excellent 4.0 - 4.4
Very good 3.5 - 3.9
Acceptable, but low priority 3.0 - 3.4 Not Fundable - May or May Not be Discussed by the Committee
Needs revision 2.5 - 2.9
Needs major revision 2.0 - 2.4
Seriously flawed 1.0 - 1.9
Rejected 0.0 - 0.9

Appendix I - Sequence of Steps for the Peer Review Process

Pre-ranking of assigned applications 1. Chair, SO and CIHR staff assign the applications to peer reviewers.
2. Peer reviewers rate and provide written reviews for their assigned applications.
3. CIHR staff identifies the most competitive applications.
Peer Review of selected applications 4. Reviewers not in conflict with any of the selected applications are identified.
5. Peer review meeting which consist of an open discussion of all the selected applications amongst committee members.
6. Winner of each application process is chosen by the committee.

Appendix II - Knowledge Translation Prize Reviewer Worksheet

Applicant:

Reviewer:

The following table is meant to guide you in the evaluation of the application.

Review Criteria Outstanding Excellent Very good Acceptable, but low priority Needs revision Needs major revision Seriously flawed Rejected
To what extend does the application demonstrate evidence of influence on the synthesis, dissemination, exchange and ethically sound application of knowledge in Canada and/or abroad?
To what extent does the application demonstrate significant improvement of the health of individuals in Canada and/or abroad, more effective health services and products, and/or strengthening of the health system?
How strong are the letters of support for the nominee(s)?

Appendix III - Institute of Aging Betty Havens Prize for Knowledge Translation in Aging Reviewer Worksheet

Applicant:

Reviewer:

The following table is meant to guide you in the evaluation of the application.

Review Criteria Outstanding Excellent Very good Acceptable, but low priority Needs revision Needs major revision Seriously flawed Rejected
To what extend does the application demonstrate evidence of influence on the synthesis, dissemination, exchange and ethically sound application of knowledge at the local or regional level?
To what extent does the application demonstrate significant improvement of the health of individuals in Canada, more effective health services and products, and/or strengthening of the health system?
How strong are the letters of support for the nominee(s)?