CIHR Peer Review Guide for Training and Salary Awards

Table of Contents

Introduction

On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide information on CIHR’s objectives, governance and policies; to outline of the roles and responsibilities of peer review committee members; and to define the policies, principles and procedures for peer review of training and salary awards applications.

It is important to note that some policies may not apply to all training or salary award programs. Reviewers should refer to the program-specific Reviewers’ Guide or contact the committee coordinator for more information.

In addition to the present guide, it is essential that reviewers become familiar with the pertinent documentation associated with the program for which they are reviewing before starting the review process. This includes:

Peer Review at CIHR

CIHR’s mandate is to “excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.”

The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also ensures accountability, not only to the Government of Canada and the Canadian taxpayer – the source of CIHR funding – but to the research community at large. Peer review is carried out by committees of experts that encompass all four pillars of health research: biomedical, clinical, health systems & services, and social, cultural and environmental factors that affect the health of populations.

Peer review is overseen by CIHR’s Scientific Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with the CIHR Act and the overarching strategic directions set out by Governing Council. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC’s responsibilities.

For further information on the management of peer review, please consult CIHR’s website.

Principles of Peer Review

The integrity of the peer review process relies on well-established principles and policies that ensure fair and effective evaluation and support CIHR objectives and strategic funding targets.

The principles guiding peer review are:

  • Absence of conflict of interest;
  • Confidentiality;
  • Fairness.

Conflict of Interest and Confidentiality (COIC)

CIHR must meet the highest ethical and integrity standards in all that it does in order to continue to merit the trust and confidence of the research community, the government and the public. CIHR review committee members, external reviewers and observers must meet the highest standards of ethical behaviour to maintain and enhance public confidence in CIHR’s ability to act in the public’s best interest and for the long-term public good. Where a conflict arises between private and public interests, review committee members, external reviewers, and observers will be expected to take the necessary measures to ensure that the public interest is protected.Footnote 1

The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. Thus, information contained in applications, reviewer reports, names of reviewers and committee discussions are all strictly confidential. The use of this information for any other purpose than what is outlined here is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.

All committee members must read and agree to abide by the COIC policy prior to viewing any application information.

For the most up-to-date definition of conflict of interest and confidentiality, please consult the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Fairness

The success of the peer review system is critically dependent upon the willingness and ability of all committee members to:

  • be fair and reasonable;
  • to exercise rigorous scientific judgment; and
  • to understand, and take into account in a balanced way, the particular context of each application.

In programs where written reviews are required, these reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that the review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, that helps the applicant by pointing out deficiencies that could be repaired in a resubmission, will help convince a disappointed applicant that a fair assessment of the proposal was provided.

Policies Impacting Peer Review

The following are the policies guiding peer review:

  • International Collaborations;
  • Knowledge Translation;
  • Tri-Agency Open Access Policy on Publications;
  • Gender, Sex and Health Research;
  • Official Languages.

For the full updated description of each policy, please visit CIHR’s website.

Peer Review Logistics

Review Types and Meeting Formats

Peer Review at CIHR is typically conducted using one of two fundamental types of review: structured review or unstructured review - although, in some instances, other methods of adjudication may be used where appropriate. The meeting format can be a face-to-face meeting, a teleconference and/or a virtual meeting (at-home) using the ResearchNet web platform.

Notwithstanding the type of review, the objective remains the same - to fund excellence. The following key points determine the type of review and meeting format used for a Funding Opportunity:

  1. Program objectives;
  2. Anticipated number of applications;
  3. Length of applications;
  4. Nature of the assessment required (i.e., weighted criteria, reaching consensus, ranking).

Refer to Types of Review at CIHR for complete details on the review types and meeting formats.

Reviewers are invited to consult the appropriate program’s Reviewers’ Guide to find out what type of review will be used for their committee and the detailed step-by-step instructions of the process.

Committee Members

The committee membership will vary depending on the type of peer review. All committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment. Committees as whole should also satisfy the need to cover the range of research areas for which the committee is responsible, to appropriately represent the Canadian health research community, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members.

For further details on each of the topic listed below, please consult the appropriate link:

Review Process

All applications received by the appropriate deadline date are reviewed by CIHR staff for eligibility and compliance. Those deemed not eligible are withdrawn from the competition.

Step 1: Attending the Orientation Session

The objective of the orientation session is to familiarize the reviewers with the program, explain the peer review process and provide details on how to complete the review.

Step 2: Declaring Conflicts and Assignment of Applications

Reviewers cannot assess applications if they have a conflict of interest with said application.

Depending on the type of review being completed, the reviewers will be given access to the applications to declare conflicts of interest and, if applicable, indicate their level of expertise either prior or before the assignments. Chairs, Scientific Officers and/or CIHR staff assign the applications to committee members. External members can also be solicited if required. However, the final authority for the assignments rests with CIHR.

Step 3: Evaluating Applications

All applications submitted to a funding opportunity are treated equally for evaluation; the same criteria and funding cut-offs are applied to all. The applications are evaluated in reference to adjudication criteria that are specific to each program. These criteria are listed in the funding opportunity details and in the program’s Reviewers’ Guide.

The prime responsibilities of a peer review committee are 1) to evaluate applications submitted for a particular competition and 2) to rate/rank them in order of excellence using the appropriate scale. It is important that committees follow defined procedures in order to function in a consistent manner.

Depending on the program (thus the review type being used), the evaluation of applications can be completed in multiple stages.

Instructions on how to review applications for a specific Funding Opportunity will be detailed in the program’s Reviewers’ Guide.

Unconscious bias

As reviewers read the applications, they should be alert to unconscious bias related to gender, discipline or geographic location and remember that:

  • Career interruptions for child bearing and raising can influence opportunity for knowledge production, publications and related variables;
  • Different disciplines and environments offer different opportunities for research contributions, publication and other research related activities; and,
  • The reputation of institutions should not affect the reviewer’s view of applicants or their research training environment.

Publications and Productivity

An important evaluation criterion in most training and salary awards funding programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing publications, peer review committees should focus on the quality of a publication's content and NOT simply on the number of publications nor the impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.

CIHR funds individuals in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."

When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer review committees must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.

Step 4: Streamlining

For many programs, less than one-third of the applications are ultimately funded; thus, it is important that committees focus their time and efforts on the most competitive applications to ensure that an accurate rank-order list is generated. To help support this goal, a streamlining process is used to eliminate non-competitive applications, allowing peer reviewers more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding.

Step 5: Special Considerations

Budget and Term

The budget and term for training and salary award programs are predetermined and are stated in the Funding Opportunity Details. Nevertheless, given that all programs have different objectives, some committees could require reviewers to comment on those aspects of the applications. Questions about the requested budget should not influence the rating of the application, unless they bear directly on the scientific merit. For further details, please refer to the budget and term details in the funding opportunity, at the beginning of the peer review process.

Special Attention Issues

Any concerns in the following areas should be flagged for CIHR staff to address. Note that some issues listed below may not apply to all training or salary award programs. Please contact the committee coordinator for more information.

These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please refer to the Grants and Awards Guide.

  1. Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria specified in the Funding Opportunity to receive CIHR funding.
  2. Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may comment on specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  3. Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”, but it is essential that this be verified by committee members.
  4. Budget justification: Issues related to the budget should be brought to the attention of CIHR staff if the peer review committee cannot properly assess the budget request because of an unclear justification by the applicant. CIHR staff will follow up before funds are released, if the application is funded.
  5. Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded.

For more information, please refer to the Pending Grants and Awards page in the Grants and Awards Guide.

Feedback

An important component of the peer review process is the review of the committee’s effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR’s ongoing efforts to maintain an effective and high quality peer review system.

For face to face or teleconference meetings, this occurs at the end of deliberations; however, for individual review, feedback should be communicated to the committee coordinator by email.

Funding Decisions

Following peer review, CIHR staff generates a rank list based on the committee recommendations, to be reviewed by CIHR’s Chief Scientific Officer (CSO) and Chief Financial Officer (CFO). Applications will be funded from the top down in order of ranking as far as the budget will allow. The CSO and CFO consider the funding recommendation in light of criteria established by Scientific Council (SC) and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications page.

Applicants are informed of the results of the competition via ResearchNet once the SC has approved the list of applicants to be funded. All applicants are sent a Notice of Decision, indicating whether or not their application was approved. They will also receive a copy of all reviews, the Scientific Officers notes (if applicable) and, for the successful candidates, an Offer of Award that details the budget, term and condition of funding.

Applications that have been flagged for special attention and followed up by CIHR staff are withheld as “pending”. The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members if necessary prior to a final decision regarding funding.

Program-Specific Reviewers’ Guides

For the specific instructions on how to conduct reviews for each program, please consult the appropriate CIHR Peer Review Guide: