Peer Review Manual for Salary and Training Award Applications
Table of Contents
- Purpose of the Manual
- Peer Review at CIHR
- Principles of Peer Review
- Policies Impacting Peer Review
- Peer Review Committee Members
- Peer Review Types & Formats
- Peer Review Process
- 7.1 Initializing the Review Process
- 7.2 Evaluating the Applications
- 7.3 Finalizing the Evaluations
- 7.4 Special Considerations
- Funding Decision
Section I – Summary of Changes
Detailed Instructions for the streamlining procedure have been added. Furthermore, one aspect of the Streamlining procedure for Merit Review has been revised: the average of the internal reviewers’ initial ratings for both Potential Impact and Scientific Merit is <3.5. Refer to section 126.96.36.199 for more details.
- Gender, Sex and Health Research - A tool for peer reviewers
A new tool was developed to aid peer reviewers to properly assess whether gender and/or sex are appropriately integrated into CIHR applicants’ proposed research designs. For more information, please refer to section 4.3.
Section II – Policy & Guiding Principles
1. Purpose of the Manual
On behalf of CIHR, we would like to thank you for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like yourself who generously give of their time and expertise. Your efforts are greatly appreciated by CIHR and the health research community.
The peer review process is described in detail in this manual and on CIHR’s website. It is essential that committee members read and be familiar with this Manual and the Funding Opportunity for which you peer review. Concise information on the role of each committee member and their responsibilities is also available on the CIHR website: please refer to the Peer Review Committee Members Role page.
The purpose of this manual is to provide information on CIHR’s objectives, governance and policies; to outline the roles and responsibilities of peer review committee members; and to define the policies and procedures for peer review of applications.
Policies and Procedures regarding the assessment of Grant applications can be found in the Peer Review Manual for Grants. For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.
2. Peer Review at CIHR
The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:
“To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system.”
The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also ensures accountability, not only to the Government of Canada and the Canadian taxpayer – the source of CIHR funding – but to the research community at large. Peer review is carried out by committees of experts that encompass all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health).
Peer review is overseen by CIHR’s Scientific Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with CIHR’s 5-year Strategic Plan. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC’s responsibilities.
For more information on the different types of Peer Review and meeting formats, please refer to Types of Review at CIHR.
3. Principles of Peer Review
Confidentiality is information about a person that shall not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. Thus, information contained in applications, reviewer reports, names of reviewers and committee discussions are all strictly confidential. The use of this information for any other purpose than what is outlined here is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.
Committee members must not discuss with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the chances of success or failure. Applicants must not contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank within committee, etc.). All requests for information on an application or a reviewer report should be referred to the Deputy Director at CIHR responsible for the committee in question.
By law, applicants have access to their own application files. Therefore, written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision and CIHR will not edit the reviews provided. The identity of the reviewers will not be revealed to the applicants under any circumstances. However, a list of peer review committee members will be published on the CIHR website 60 days after the Scientific Council approves funding for a competition cycle.
3.2 Conflict of Interest
CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a Conflict of Interest means a conflict between a Participant's duties and responsibilities with regard to the Review Process, and a Participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the Participant:
- would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
- has a professional or personal relationship with an Applicant or the Applicant’s institution; or
- has a direct or indirect financial interest in a funding opportunity or application being reviewed.
A conflict of interest may be deemed to exist or perceived as such when review committee members, external reviewers or observers:
- are a relative or close friend, or have a personal relationship with the applicants;
- are in a position to gain or lose financially/materially from the funding of the application;
- have had long-standing scientific or personal differences with the applicants;
- are currently affiliated with the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
- are closely professionally affiliated with the applicants, as a result of having in the last six years:
- frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
- been a supervisor or a trainee of the applicants;
- collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
- been employed by the institution, when an institution is the applicant; and/or
- feel for any reason unable to provide an impartial review of the application.
All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff and the Chair are responsible for resolving areas of uncertainty during the committee meeting.
All committee members must read and agree to abide by the COIC policy prior to viewing any application information.
Success of the peer review system is critically dependent upon the willingness and ability of all committee members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application. In programs where written reviews are required, these reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that your review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, which includes helping the applicant by pointing out deficiencies that could be repaired in a resubmission, will help to convince a disappointed applicant that you provided a fair assessment of the proposal.
4. Policies Impacting Peer Review
Some policies may not apply to all salary or training award programs. Please refer to the guide for reviewers for your funding opportunity or contact your committee coordinator for more information.
4.1 International Collaboration & Global Health Research
As stated in the CIHR Act, one of the ways CIHR fulfills its mandate is by “pursuing opportunities and providing support for the participation of Canadian scientists in international collaborations and partnerships in health research.” As a result, CIHR accepts applications for research to be carried out in, or in collaboration with applicants based in, other countries. The international nature of the research should not be a factor in the scientific assessment of the proposal, beyond how it relates to the feasibility of the proposed research and the quality of the research question. Reviewers should also not be influenced by the funding obtained or requested for the international components when recommending a budget for the Canadian component(s). For detailed information on applying for funding with an international partnership component, please refer to the subsections titled Global Health Research and International Collaborations in the Grants and Awards Guide.
For detailed information on how CIHR supports international collaborations and global health research, please view Internationalization of CIHR funding policy and program tools.
4.2 Knowledge Translation
Knowledge translation is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants and research allowances.
For end of grant KT, many means of dissemination exist and the onus is on the researcher/trainee to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified. Please consult About Knowledge Translation for more information.
4.3 Gender, Sex and Health Research
Applicants are encouraged to demonstrate the use of gender and sex-based analysis in applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands understanding of health determinants in both sexes and results in improvements in health and health care. For more information on how peer reviewers can assess whether gender and/or sex are appropriately integrated into CIHR applicants’ proposed research designs, please refer to Integrating Gender and Sex in Health Research: A Tool for CIHR Peer Reviewers.
4.4 Official Language Minority Communities
Federal agencies are required to take positive measures to ensure the support and recognition of minority language communities in Canada. For CIHR, this means an obligation to promote health research in these communities. For further information, please refer to the initiative on Official Language Minority Communities. Research proposals in these areas should still be subject to the same rigorous peer review process as any other application. However, the justification for promoting health research in minority language communities should not be a factor in the assessment.
4.5 Publications and Productivity
An important evaluation criterion in all funding programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing the quality of publications, peer review committees should focus on the quality of a publication's content and NOT simply the number of publications nor the quality or impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.
CIHR funds researchers in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature".
When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer review committees must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.
Section III – Peer Review
5. Peer Review Committee Members
Peer Review committee membership will vary depending on the type of peer review. Individual committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment. For more details, please refer to the Peer Review Membership Guidelines. Committees as a whole should also satisfy the need to cover the range of research areas for which the committee is responsible, to appropriately represent the Canadian health research community, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members. For more details, please refer to the Procedure for Selection of Peer Review Committee Members.
For further information on peer reviewers at CIHR, please consult the Peer Review Committee Members Role page.
Complete instructions for peer review committee members are available at the Instructions for Committee Members page.
6. Peer Review Types & Formats
Peer Review at CIHR is typically conducted using one of two fundamental types of review: Structured review and Unstructured review. Furthermore, the peer review delivery is typically done in a face to face meeting, via teleconference and/or at-home using the technology of ResearchNet in a virtual setting. Notwithstanding the type of review, the objective remains the same - to fund excellence. The following key points determine the type of review and meeting format used for a Funding Opportunity:
- Program objectives
- Anticipated number of applications
- Length of applications
- Nature of the assessment required (i.e. weighted criteria, reaching consensus, ranking)
For more information on the different types of Peer Review and meeting formats, please refer to the Types of Review at CIHR.
The peer review type and meeting format can vary from one funding opportunity to another. Please refer to the Guide for Reviewers for more details on the peer review type and format that will be used for your peer review committee.
7. Peer Review Process
7.1 Initializing the Review Process
7.1.1 Relevance Review
The Relevance Review Process is used by strategic leads and/or partners to assess the alignment of an application with a specific research theme described in the funding opportunity (FO). As the name implies, the process is used when it is important for applications to be relevant to (or in alignment with) targeted research components of the FO. This review approach is generally reserved for strategic FOs and Priority Announcements (PAs).
7.1.2 Assigning Applications
All eligible applications received by the appropriate deadline date are entered into the competition. Applications must be complete at the time of submission; otherwise they are withdrawn from the competition.
Depending on the type of review being completed, committee members will be given access to the application to declare conflicts of interest and indicate their level of expertise (if applicable) either prior or after the assignments. Chairs, Scientific Officers and/or CIHR staff assign the applications to committee members (typically two internal reviewers and occasionally a reader). External reviews can also be solicited if required. The final authority for the assignment of applications rests with CIHR.
7.2 Evaluating the Applications
All applications submitted to a funding opportunity are treated equally for evaluation; the same criteria and funding cut-offs are applied to all. Reviewers evaluate applications in reference to the evaluation criteria listed in the funding opportunity details, which vary by program and by funding opportunity. In most cases, the evaluation of applications is completed in two steps: an at-home review followed by final review process to finalize the evaluations.
7.2.1 At-home Review
Reviewers are to provide a written evaluation that may include some of the following:
- A brief synopsis of the application
- An assessment of the application (strengths and weaknesses) in relation to the evaluation criteria
- Comments on issues that should be flagged
They must also provide a numerical rating of the application. To ensure consistency, all reviewers must adhere to a common rating scale. Reviewers rate applications between 0.0 and 4.9; applications with a rating of 3.5 or higher are considered for funding. For additional information, please see Ranking and Rating Scale Meaning and Use.
Finally, reviewers are required to complete the following tasks on ResearchNet:
- upload reviews;
- provide a rating for each application reviewed (note that for unstructured review, reviewers are not bound by this initial rating and can change it at the peer review meeting);
- Conduct an assessment of overall quality (if applicable).
7.3 Finalizing the Evaluations
The prime responsibilities of a peer review committee are to evaluate applications submitted for a particular competition and to rank them in order of excellence using CIHR’s rating scale. When applicable, the committee will also recommend a budget sufficient to support the proposed research. It is important that committees follow defined procedures in order to function in a consistent manner.
Any committee member who has a conflict of interest with an application (as defined in Section 3.2, above) must not take part in the evaluation of that application. For face-to-face or teleconference meetings, committee members in conflict must not be present when the application is discussed. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.
7.3.1 Unstructured Review
The evaluation of applications is finalized, in the context of an unstructured review, by a committee meeting to discuss and rate the applications, from which CIHR generates a rank-order priority list to make funding decisions.
For many programs, less than one-third of the applications are ultimately funded; thus, it is important that committees focus their discussions on the most competitive applications to ensure that an accurate rank-order list is generated. To help support this goal, a streamlining process is used to eliminate non-competitive applications from the discussion process, allowing peer reviewers more time to judge and discriminate between potentially successful applications. This also helps to ensure that the most deserving applications receive funding. Applicants whose proposals are streamlined still benefit from the review process as they receive the written at-home reviews from the assigned reviewers.
For detailed information on streamlining, please refer to Streamlining of Applications at Peer Review Committee Meetings.
188.8.131.52 Rating of Applications
If an application is not streamlined, the committee meeting discussion proceeds as follows:
- The primary reviewer presents his/her assessment, describing strengths and weaknesses of the proposal;
- The secondary reviewer follows, concentrating on points of agreement or disagreement with the first, and elaborating points not addressed by the first reviewer;
- If external reviews have been provided, a committee member reads the reviews;
- The reader may comment on issues that have been raised, or raise additional issues, as appropriate;
- The Chair leads the discussion of the proposal by all committee members;
- The Scientific Officer reads back the Scientific Officer notes, capturing the key elements of the discussion to be considered when rating the application;
- The Chair seeks a “consensus rating” from the two internal reviewers. The internal reviewers may revise their initial ratings as they see fit. If a consensus cannot be reached, the Chair declares the consensus score, usually by using the mean value of the revised respective scores after discussion (round up, if necessary, to obtain a single decimal point);
- All committee members in the room (including members on teleconference), including the two internal reviewers but excluding the Chair and Scientific Officer, then cast individual confidential votes within ±0.5 of the consensus rating. The internal reviewers are not bound to the consensus rating. The rating assigned to the proposal is the average of these confidential votes. A vote is taken even if the consensus rating is <3.5 (i.e., not in the fundable range), but the budget discussion following the vote should be brief.
184.108.40.206 End of Meeting Review
Once all applications have been reviewed, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, a consensus rating is determined by the two internal reviewers and voting proceeds as before. The committee does not review the overall rankings of all applications at the end of the meeting as individuals with conflicts of interest would inevitably be present.
7.3.2 Structured Review
The evaluation of applications is finalized by CIHR staff as generally peer reviewers for a structured review do not meet as a group and do not discuss the merits of each application. In certain instances when there is a significant discrepancy between the reviewers assessment for an application, discussion may be warranted.
220.127.116.11 How are Discrepancies Identified?
When all at-home review scores are received, CIHR will calculate an average for each applicant based on both reviewers weighted scores. CIHR will then identify applications which are at risk of an unfair decision because of a wide spread between the reviewers' ratings (the "acceptable" range is determined based on the level of variation of all scores received). In cases where narrowing the gap between the two reviewers' ratings may affect the funding decision, CIHR will ask reviewers to reconsider their initial assessment and resubmit scores. To do so, reviewers may be asked to get in contact to discuss the application. If the discrepancy persist, then an additional review will be necessary.
7.4 Special Considerations
7.4.1 Budget and Term
The budget and term for salary and training award programs is predetermined and are stated in the Funding Opportunity. Nevertheless, given that all programs have different objectives some committees could be required to comment on those aspects of the applications. Questions about the requested budget should not influence the rating of the application, unless they bear directly on the scientific merit. Please refer to the budget and term details in the funding opportunity at the beginning of the peer review process, for further details.
7.4.2 Flagging of Special Attention Issues
Any concerns in the following areas should be flagged for CIHR staff to address. Note that some issues listed below may not apply to all salary or training award programs. Please contact your committee coordinator for more information.
These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please refer to the Grants and Awards Guide.
- Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria specified in the Funding Opportunity to receive CIHR funding.
- Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may comment on specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
- Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”, but it is essential that this be verified by committee members.
- Budget justification: Issues related to the budget should be brought to the attention of CIHR staff if the peer review committee cannot properly assess the budget request because of an unclear justification by the applicant. CIHR staff will follow up before funds are released, if the application is funded.
- Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded.
For more information please refer to the Pending Grants and Awards page in the Grants and Awards Guide.
7.4.3 Discussion on Program, Process and Peer Review
An important component of any peer review committee is the final review of the effectiveness of the evaluation process and functioning, and a discussion of policy issues. This discussion provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the peer review process as part of CIHR’s ongoing efforts to maintain an effective and high quality peer review system.
For face to face or teleconference meetings, this occurs at the end of deliberations; however, for at-home review, feedback should be communicated to the committee coordinator by email.
8. Funding Decision
Following peer review, CIHR staff generates a rank list based on the committee recommendations, to be reviewed by CIHR’s Chief Scientific Officer (CSO) and Chief Financial Officer (CFO). Applications will be funded from the top down in order of ranking as far as the budget will allow. The CSO and CFO consider the funding recommendation in light of criteria established by Scientific Council (SC) and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications page as soon as it is available.
Applicants are informed of the results of the competition via ResearchNet once the SC has approved the list of applicants to be funded. All applicants are sent a Notice of Decision, indicating whether or not their proposal was approved. They will also receive a copy of all reviews, the Scientific Officers notes (if applicable) and an Offer of Award that details the budget, length and condition of funding.
Applications that have been flagged for special attention and followed up by CIHR staff (see Section 7.4.2) are withheld as “pending”. The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members if necessary prior to a final decision regarding funding.