CIHR Peer Review Manual for Grant Applications

Table of Contents


Section I – Summary of Changes

A. New Policy on Stipends Paid from Grants

As of September 15, 2011, the federal granting agencies' existing regulations on minimum and maximum stipend levels paid from grants will no longer apply. The agencies will also no longer restrict researchers from using some of their grant money to provide supplements to scholarship holders. For more information please refer to section 8.2.

B. New Merit Review Rating Scale

An updated Merit Review rating scale now applies to all funding opportunities that use merit review. To view the new scale and its descriptors, please refer to section 7.4.

C. Streamlining

Detailed Instructions for the streamlining procedure have been added. Furthermore, one aspect of the Streamlining procedure for Merit Review has been revised: the average of the internal reviewers' initial ratings for both Potential Impact and Scientific Merit is < 3.5. Refer to section 6.2.1 for more details.

D. Optical Mark Recognition Rating Sheets

To automate the recording of ratings at peer review committee meetings, CIHR is implementing the Optical Mark Recognition (OMR) to replace the current yellow rating sheets. This new process will ensure accuracy and reduce the administrative process. The phased implementation has begun and your committee coordinator will provide specific instructions at the committee meeting regarding the rating system used.

E. Equipment

Depending on the funding opportunity, equipment costs may be requested for identified items required to do the research. As of January 2012, funding for equipment does not have to be requested in the first year. It can be requested at any time during the grant term. See section 8.4 for more details.

Section II – Policy & Guiding Principles

1 Purpose of the Manual

On behalf of CIHR, we would like to thank you for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like yourself who generously give of their time and expertise, and your efforts are greatly appreciated by CIHR and the scientific community.

The peer review process is described in detail in this manual and on CIHR's website. It is essential that committee members read and be familiar with this Manual and the Funding Opportunity for which you peer review. Concise information on the role of each committee member and their responsibilities is also available on the CIHR website: please refer to the Peer Review Committee Members Role page.

The purpose of this manual is to provide information on CIHR's objectives, governance and policies; to outline the roles and responsibilities of peer review committee members; and to define the policies and procedures for peer review of applications.

Policies and Procedures regarding the assessment of Awards applications can be found in the Peer Review Manual for Salary & Training Awards. For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.

2 Peer Review at CIHR

The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:

"To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system."

The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also ensures accountability, not only to the Government of Canada and the Canadian taxpayer – the source of CIHR funding – but to the research community at large. Peer review is carried out by committees of experts that encompass all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health).

Peer review is overseen by CIHR's Science Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with CIHR's 5-year Strategic Plan. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC's responsibilities.

For more information on the different types of Peer Review and meeting formats, please refer to Types of Review at CIHR.

3 Principles of Peer Review

3.1 Confidentiality

Confidentiality is information about a person that shall not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. Thus, information contained in applications, reviewer reports, names of reviewers and committee discussions are all strictly confidential. The use of this information for any other purpose than what is outlined here is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.

Committee members must not discuss with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the chances of success or failure. Applicants must not contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank within committee, etc.). All requests for information on an application or a reviewer report should be referred to the Deputy Director at CIHR responsible for the committee in question.

By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision and CIHR will not edit the reviews provided. The identity of the reviewers will not be revealed to the applicants under any circumstances. However, a list of peer review committee members will be published on the CIHR website 60 days after the Scientific Council approves funding for a competition cycle.

3.2 Conflict of Interest

CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a Conflict of Interest means a conflict between a Participant's duties and responsibilities with regard to the Review Process, and a Participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the Participant:

  1. would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
  2. has a professional or personal relationship with an Applicant or the Applicant’s institution; or
  3. has a direct or indirect financial interest in a funding opportunity or application being reviewed.

A conflict of interest may be deemed to exist or perceived as such when review committee members, external reviewers or observers:

  • are a relative or close friend, or have a personal relationship with the applicants;
  • are in a position to gain or lose financially/materially from the funding of the application;
  • have had long-standing scientific or personal differences with the applicants;
  • are currently affiliated with the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
  • are closely professionally affiliated with the applicants, as a result of having in the last six years:
    • frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
    • been a supervisor or a trainee of the applicants;
    • collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
    • been employed by the institution, when an institution is the applicant; and/or
  • feel for any reason unable to provide an impartial review of the application.

All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff and the Chair are responsible for resolving areas of uncertainty during the committee meeting.

All committee members must read and agree to abide by the COIC policy prior to viewing any application information.

3.3 Fairness

Success of the peer review system is critically dependent upon the willingness and ability of all committee members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application. In programs where written reviews are required, these reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that your review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, which includes helping the applicant by pointing out deficiencies that could be repaired in a resubmission, will help to convince a disappointed applicant that you provided a fair assessment of the proposal.

4 Policies Impacting Peer Review

4.1 International Collaboration & Global Health Research

As stated in the CIHR Act, one of the ways CIHR fulfills its mandate is by "pursuing opportunities and providing support for the participation of Canadian scientists in international collaborations and partnerships in health research." As a result, CIHR accepts applications for research to be carried out in, or in collaboration with applicants based in, other countries. The international nature of the research should not be a factor in the scientific assessment of the proposal, beyond how it relates to the feasibility of the proposed research and the quality of the research question. Reviewers should also not be influenced by the funding obtained or requested for the international components when recommending a budget for the Canadian component(s). For detailed information on applying for funding with an international partnership component, please refer to the subsections titled Global Health Research and International Collaborations in the Grants and Awards Guide.

For detailed information on how CIHR supports international collaborations and global health research, please view Internationalization of CIHR funding policy and program tools.

4.2 Knowledge Translation

Knowledge translation is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants.

For end of grant KT, many means of dissemination exist and the onus is on the researcher to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified. Please consult About Knowledge Translation for more information.

4.3 Open Access

Applicants and peer reviewers are reminded that the CIHR Open Access Policy applies to all new and renewed funding awarded after January 1, 2008. Researchers must ensure that all published peer-reviewed articles are freely available online within twelve months. Authors can adhere with the policy by archiving peer-reviewed manuscripts in an open access repository (e.g. institutional repository) or by publishing in an open access journal. Please consult the policy web site for more detail.

4.4 Gender, Sex and Health Research

Applicants are encouraged to demonstrate the use of gender and sex-based analysis in applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands understanding of health determinants in both sexes and results in improvements in health and health care. For more information on how peer reviewers can assess whether gender and/or sex are appropriately integrated into CIHR applicants' proposed research designs, please refer to Integrating Gender and Sex in Health Research: A Tool for CIHR Peer Reviewers.

4.5 Official Language Minority Communities

Federal agencies are required to take positive measures to ensure the support and recognition of minority language communities in Canada. For CIHR, this means an obligation to promote health research in these communities. For further information, please refer to the initiative on Official Language Minority Communities. Research proposals in these areas should still be subject to the same rigorous peer review process as any other application. However, the justification for promoting health research in minority language communities should not be a factor in the assessment.

4.6 Publications and Productivity

An important evaluation criterion in all grant programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing the quality of publications, peer review committees should focus on the quality of a publication's content and NOT simply the number of publications nor the quality or impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.

CIHR funds researchers in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."

When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer review committees must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.

Section III - Peer Review

5 Grants Peer Review Committee Members

A typical CIHR grants peer review committee consists of a Chair, Scientific Officer, peer reviewers, CIHR staff and other specialized roles depending on the funding opportunity. Individual committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment. For more details, please refer to the Peer Review Membership Guidelines. Committees as a whole should also satisfy the need to cover the range of research areas for which the committee is responsible, to appropriately represent the Canadian health research community, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members. For more details, please refer to the Procedure for Selection of Peer Review Committee Members.

CIHR grant competitions can be held on a recurring or an ad hoc basis. For recurring competitions, standing peer review committees are formed, and committee members are recruited for a term of service (typically three years) in order to ensure consistency and continuity in the review process. Standing committee membership may be supplemented by additional members as required for a specific competition, based on the applications received and expertise needed for their review. For ad hoc competitions, committees are formed to review applications for that particular competition and then disbanded.

For further information on peer reviewers at CIHR, please consult the Peer Review Committee Members Role page.

Complete instructions for peer review committee members are available at the Instructions for Committee Members page.

6 Grant Application Review Process

6.1 Initializing the Review Process

6.1.1 Relevance Review

The Relevance Review Process is used by strategic leads and/or partners to assess the alignment of an application with a specific research theme described in the funding opportunity (FO). As the name implies, the process is used when it is important for applications to be relevant to (or in alignment with) targeted research components of the FO. This review approach is generally reserved for strategic FOs and Priority Announcements (PAs).

6.1.2 Assigning Applications

All eligible applications received by the appropriate deadline date (posted in the competition announcement) are entered into the competition. Applications must be complete at the time of submission; otherwise they are withdrawn from the competition. Specific exceptions to this rule can be found in the funding opportunity descriptions.

Within two weeks after the deadline, the Chairs and Scientific Officers, together with CIHR staff, review the applications assigned to their committee. Together, they are responsible for ensuring their committee is equipped with the appropriate expertise and, upon accepting an application for review by their committee, accept responsibility for ensuring that the committee performs a fair review. In some cases, applications may not meet a particular committee's mandate and may need to be reassigned (if the competition has more than one peer review committee). A maximal load for a committee should be not more than 60 applications for a 3-day meeting. The final authority for the assignment of applications to a peer review committee rests with CIHR.

After the list of applications is compiled, committee members are given access to the application summaries to declare any conflicts of interest and indicate their level of expertise.

Chairs, Scientific Officers and CIHR staff then assign the applications to committee members (typically two internal reviewers and a reader). External reviews can also be solicited if required, as noted above. All committee members, excluding external reviewers, are then given access to the full applications assigned to their committee four to six weeks before the peer review committee meeting. External reviewers see only those applications assigned to them. Committee members are able to view the external reviewer reports before the peer review committee meeting as they become available.

6.1.3 Reviewing Applications

It is the responsibility of all peer reviewers to familiarize themselves in advance of the meeting with all applications to be assessed by their committee, as well as with any available external reviewer reports. This will assist peer reviewers to prepare their internal reviewer's reports. Details on what to include in the report can be found in section 7.

In advance of the meeting, internal reviewers are required to complete the following tasks on ResearchNet:

  1. upload reviews;
  2. provide an initial rating for each application reviewed (note that reviewers are not bound by this initial rating and can change it at the peer review committee meeting);
  3. If applicable, divide the applications reviewed into a top and bottom group, based on their overall quality. The top group should include the applications considered to be highly competitive and most deserving of being funded. This assessment is to be based on the reviewers' total experience with applications over the past five years and will be used during the streamlining phase of review (Section 6.2.1). The proportion of applications in each group may vary depending on the overall quality of the applications reviewed.

The deadline for uploading reviews to ResearchNet, along with assessments of overall quality and initial ratings, is one week before the meeting date. Reviews can be saved as drafts, by selecting "save draft copy" on ResearchNet prior to submission. In order to access the other reviews for the applications that you were assigned, you must select "submit final review" on ResearchNet. Afterwards, you will no longer be able to modify it prior to the meeting. If you wish to revise your reviews after the committee meeting, you will have one week to directly upload your changes to ResearchNet.

Please contact your committee coordinator if you encounter any technical issues.

6.2 During the Meeting

The prime responsibilities of a peer review committee are to evaluate applications submitted for a particular competition and to rank them in order of excellence using CIHR's rating scale. When applicable, the committee will also recommend a budget sufficient to support the proposed research. It is important that committees follow defined procedures in order to function in a consistent manner. For a summary of the review procedures for various peer review meeting formats, please see the attached appendices.

Any committee member who has a conflict of interest with an application (as defined in Section 3.2, above) must not take part in the discussion of that application. For face-to-face meetings, committee members in conflict must leave the room before the application is discussed. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.

6.2.1 Streamlining
Background

The evaluation of applications for funding occurs in two phases: (1) an in-depth "at-home" review by at least two peer reviewers to produce written evaluations, and (2) a committee meeting to discuss and rate the applications, from which CIHR generates a rank-order priority list to make funding decisions. For many programs, less than one-third of the applications are ultimately funded; thus, it is important that committees focus their discussions on the most competitive applications to ensure that an accurate rank-order list is generated. To help support this goal, a streamlining process is used to eliminate non-competitive applications from the discussion process, allowing peer reviewers more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding. Applicants whose proposals are streamlined still benefit from the review process as they receive the written reviews from the assigned reviewers.

For detailed information on streamlining, please refer to Streamlining of Applications at Peer Review Committee Meetings.

Procedure

The assessment of each application at peer review committee meetings begins with internal reviewers announcing their initial ratings to one decimal place. An application is then streamlined if it meets the following conditions:

For Peer Review

  1. if applicable, both reviewers placed the application in their bottom group;
  2. the average of the internal reviewers' initial ratings is <3.50;
  3. there is no objection from the other committee members that the application not be discussed.

For Merit Review

  1. the average of the internal reviewers' initial ratings for both Potential Impact and Scientific Merit is <3.5;
  2. there is no objection from the other committee members that the application not be discussed.
Example for Merit Review
Reviewer's Initial Rating Mean Candidate for Streamlining
Scientific Merit Potential Impact Scientific Merit Potential Impact
  1. Normally, given the substantial discrepancy in initial ratings for SM, the reviewers or Chair will ask to discuss the application.
2.5 3.5 3.7 3.9 3.0 3.8 NO
3.2 3.6 3.8 3.2 3.4 3.5 NO
3.0 3.6 3.6 3.1 3.3 3.4 YES
2.0 4.0 3.4 3.4 3.0 3.4 YES1

Notes

  1. Other funding programs may use different standards to vary the amount of discussion overall, according to their strategic objectives.
  2. If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports (and external reviews, if any) and the Scientific Officer notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.
6.2.2 Rating of Applications

If an application is not streamlined, the committee meeting discussion proceeds as follows:

  • The primary reviewer presents his/her assessment, describing strengths and weaknesses of the proposal (see Section 7.2 for evaluation criteria);
  • The secondary reviewer follows, concentrating on points of agreement or disagreement with the first, and elaborating points not addressed by the first reviewer;
  • If external reviews have been provided, a committee member reads the reviews;
  • The reader may comment on issues that have been raised, or raise additional issues, as appropriate;
  • The Chair leads the discussion of the proposal by all committee members;
  • The Scientific Officer reads back the Scientific Officer notes, capturing the key elements of the discussion to be considered when rating the application;
  • The Chair seeks a "consensus rating" from the two internal reviewers. The internal reviewers may revise their initial ratings as they see fit. If a consensus cannot be reached, the Chair declares the consensus score, usually by using the mean value of the revised respective scores after discussion (round up, if necessary, to obtain a single decimal point);
  • All committee members in the room (including members on teleconference), including the two internal reviewers but excluding the Chair and Scientific Officer, then cast individual confidential votes within ±0.5 of the consensus rating. The internal reviewers are not bound to the consensus rating. The rating assigned to the proposal is the average of these confidential votes. A vote is taken even if the consensus rating is <3.5 (i.e., not in the fundable range), but the budget discussion following the vote should be brief.
6.2.3 Review of Lay Abstracts

After the rating of the application, the Community Reviewer (if present) provides comments on the lay abstract, if chosen as a highlight for the committee. The Community Reviewer addresses the extent to which the intent and importance of the proposed research is well explained and in a language clear to members of the general public.

6.2.4 Budget and Term

The appropriateness of the budget and the term of support are discussed, and recommendations are made. Please see Section 8 for guidelines. Note that questions about the budget should not influence the rating of the application, unless they bear directly on the scientific merit.

6.2.5 Flagging of Grants

Any concerns in the following areas should be discussed and, if necessary, flagged at the meeting for CIHR staff to address.

These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please refer to the Grants and Awards Guide.

  1. Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria to receive CIHR funding.
  2. Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may comment on specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  3. Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled "Certification Requirements", but it is essential that this be verified by committee members.
  4. Budget justification: If the peer review committee cannot properly assess the budget request because of an unclear justification by the applicant, please bring the issues to the attention of CIHR staff who will follow up before funds are released, if the application is funded.
  5. Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded. For more information please refer to the Pending Grants and Awards page in the Grants and Awards Guide.
6.2.6 End of Meeting Review

Once all applications have been reviewed, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, a consensus rating is determined by the two internal reviewers and voting proceeds as before. The committee does not review the overall rankings of all applications at the end of the meeting as individuals with conflicts of interest would inevitably be present.

An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This discussion provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.

6.2.7 Newsworthy and Highly Rated Applications

During the peer review committee meeting, the committee is asked to identify those applications that are most newsworthy, highly rated, and/or likely to create public interest. As part of CIHR's mandate to engage the Canadian public and report to Parliament on its research initiatives, the Communications and Public Outreach department will use these suggestions to develop news stories and backgrounders for parliamentarians, key Ministers and decision-makers. In addition, the information gathered can be used for corporate publications and other activities highlighting CIHR's research efforts with key stakeholders.

6.3 After the Meeting

As soon as possible after the peer review committee meeting, CIHR staff generates a funding proposal based on committee recommendations, to be reviewed by the Chief Scientific Officer and the Chief Financial Officer. The CSO and CFO consider the funding recommendation in light of criteria established by SC and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications page as soon as it is available.

Applicants are informed of the results of the competition as follows:

  1. For some competitions, applicants receive a copy of all reviews and the Scientific Officer notes as soon as they are available after the peer review committee meetings, and are informed of the rating and ranking of the application in the committee (Notice of Recommendation). This enables applicants whose applications are unlikely to be approved for funding by the SC to begin to plan a resubmission, where applicable, and to register for subsequent competitions.
  2. Once the SC has approved the grants to be funded, all applicants are sent a Notice of Decision, indicating whether or not their proposal was approved, and if approved, with what budget, which may or may not coincide with that recommended by the peer review committee. The Notice of Decision will normally be released (on ResearchNet and by mail) within three weeks following the SC meeting.

Applications that have been flagged for special attention and followed up by CIHR staff (see Section 6.2.5) are withheld as "pending". The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members if necessary prior to a final decision regarding funding.

6.4 Merit Review

Applications to certain funding opportunities are evaluated by merit review, which requires that both the scientific merit and potential impact of the projects be assessed using separate scores. Merit review is used for research programs that engage knowledge-users throughout the research process to inform the research plan, carry out the project, and apply the findings. In general, the potential impact score reflects the relevance/importance of the project to the knowledge-users and the likelihood that the project will have a substantive and sustainable impact on health outcomes, practice, programs and/or policy in the study context. Therefore, while the overall procedure for rating an application is as above, there are two separate scoring components instead of one and the internal reviewer makeup differs. The sequence of steps for programs that use merit review (PHSI, KAL, KRS and CBR-HIV/AIDS) is referenced in Appendix II. An example of the scoring of an application by merit review is given in Appendix III. Please see the relevant funding opportunity details for further information on peer review and evaluation.

7 Evaluating Grant Applications

7.1 Types of Applications

For standing competitions like the Open Operating Grant Program (OOGP), applications may be new proposals, renewals of previously funded projects, or resubmissions of an unsuccessful new or renewal application. All application types are evaluated together "on a level playing field" and the same criteria and funding cut-offs are applied to all, though peer review committee members are reminded to take the stage of career and progress made during previously funded grants into account and to vary the emphasis placed on track record and training experience appropriately. Where applicable, resubmissions may contain a two page response to previous reviews of the applications. This response, along with corresponding modifications to the research plan, is expected to improve the quality of the application. However, committee members are not provided with copies of previous reviews and must still rate resubmissions relative to other applications in the competition.

In some cases, the same peer review committee may review applications for more than one funding opportunity. For example, an OOGP committee may also review applications for a strategic funding opportunity (also called a Request for Applications, or RFA). At the conclusion of the committee meeting, these applications will be separated into their own overall ranking lists and funding decisions will be made based on the funding envelopes provided through their respective programs. Therefore, the presence of these applications will have no impact on the funding of other applications being reviewed by a committee.

7.2 Evaluation Criteria

The exact criteria for each funding opportunity are described in the funding opportunity details. All applications for Operating, Catalyst, Teams and Emerging Teams grants are evaluated against the five criteria indicated below. These five criteria are further elaborated and defined by specific factors to be considered under each criterion, although it should be noted that not all factors are necessarily relevant to every grant application. The criteria are not listed in order of importance: the relative weighting of each criterion will vary depending on the objectives of the funding opportunity, as defined by CIHR. The review of the application should be structured to justify the overall rating based on the relevant criteria.

Criterion #1: Research Approach

  • Clarity of the research question.
  • Completeness of the literature review and relevance to study design/research plan.
  • Clarity of rationale for the research approach and methodology.
  • Appropriateness of the research design.
  • Appropriateness of the research methods.
  • Feasibility of the research approach (including recruitment of subjects, project timeline, preliminary data where appropriate, etc.).
  • Anticipation of difficulties that may be encountered in the research and plans for management.

Criterion #2: Originality of the Proposal

  • Potential for the creation of new knowledge.
  • Originality of the proposed research, in terms of the hypotheses/research questions addressed, novel technology/methodology, and/or novel applications of current technology/methodology.

Criterion #3: Applicant(s)

  • Qualifications of the applicant(s), including training, experience and independence (relative to career stage).
  • Experience of the applicant(s) in the proposed area of research and with the proposed methodology.
  • Expertise of the applicant(s), as demonstrated by scientific productivity over the past five years (publications, books, grants held, etc.). Productivity should be considered in the context of the norms for the research area, applicant experience and total research funding of the applicant.
  • Ability to successfully and appropriately disseminate research findings, as demonstrated by knowledge translation activities (publications, conference presentations, briefings, media engagements, etc.).
  • Appropriateness of the team of applicants (if more than one applicant) to carry out the proposed research, in terms of complementarity of expertise and synergistic potential.

Criterion #4: Environment for the Research

  • Availability and accessibility of personnel, facilities and infrastructure required to conduct the research.
  • Suitability of the environment to conduct the proposed research.
  • Suitability of the environment (milieu, project and mentors) for the training of personnel (if applicable).

Criterion #5: Impact of the Research

  • Research proposal addresses a significant need or gap in health research and/or the health care system.
  • Potential for a significant contribution to the improvement of people's health in Canada and the world and/or to the development of more effective health services and products.
  • Appropriateness and adequacy of the proposed plan for knowledge dissemination and exchange.

Additional factors to be considered under each criterion may also be described in the funding opportunity details. Please contact your committee coordinator if you need further guidance on how to apply the individual criteria.

7.3 Internal Reviewer Report - Format

  1. A brief synopsis of the proposal
    • the purpose of the proposal;
    • the hypothesis to be tested, or the questions to be answered;
    • the objectives to be achieved and approach proposed;
    • the progress made to date.
  2. An assessment of the proposal, based on the evaluation criteria as presented in the funding opportunity details:
    • consider all factors and the strengths or weaknesses of the applications in relation to each criteria;
    • not all factors are necessarily important to each application;
    • emphasis may be placed on specific criteria in the funding opportunity details, in order to meet funding program objectives; consider this when formulating your rating;
    • focus your comments on the factors most relevant to your rating.
  3. Comments on the budget requested and a formal recommendation, including clear and detailed reasons for any recommended budget cuts.
  4. If necessary, comments on issues the reviewer feels should be flagged (Section 6.2.5). These concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the application.

The review should be clear and concise, using objective and non-inflammatory language, and include justification. Constructive advice to the applicant will allow him/her to improve the quality and efficiency of the proposed research. The applicant will receive the review as it is submitted by the reviewer. For this reason, please do not identify yourself in order to ensure the confidentiality of the review process.

7.4 The Rating

Criteria to assess the scientific merit of an application are as described above in Section 7.2. The relative weighting of these criteria depends on the program objectives as described in the funding opportunity description; if in doubt, please contact the Deputy Director responsible for the committee.

To ensure consistency, all reviewers must adhere to a common scale. It is particularly important that the full scale be used and the same conventions applied to assigned ratings. To facilitate this, the following scale and descriptors should be used:

Descriptor2 Range3 Outcome
  1. Only applications rated 3.5 or higher are eligible for CIHR funding (including partner funded programs).
  2. In the committee meetings, reviewers assign scores to one decimal place, but the final average rating is calculated to two decimal places.
Outstanding 4.5 – 4.9 May Be Funded – Will be Discussed by the Committee
Excellent 4.0 – 4.4
Very good 3.5 – 3.9
Acceptable, but low priority 3.0 – 3.4 Not Fundable – May or May Not be Discussed by the Committee
Needs revision 2.5 – 2.9
Needs major revision 2.0 – 2.4
Seriously flawed 1.0 – 1.9
Rejected 0.0 – 0.9

Merit Review employs separate scores for Potential Impact and Scientific Merit. The following scale should be used for programs assessed by Merit Review:

Potential Impact4 Range5 Scientific Merit
  1. Only those applications that exceed the threshold rating of 3.5 on both Potential Impact and Scientific Merit will be considered for funding. The mean of the two scores will be calculated for the applications with a rating of 3.5 and above on both criteria to determine the final rating and establish a ranking list.
  2. In the committee meetings, reviewers assign scores to one decimal place, but the final average rating is calculated to two decimal places.
May Be Funded Extremely Significant 4.5 – 4.9 Exceptional
Very Significant 4.0 – 4.4 Outstanding
Significant 3.5 – 3.9 Excellent, may still require revision
Not Fundable Moderate 3.0 – 3.4 Very good however needs revision to be fundable
Limited 2.5 – 2.9 Needs major revision
Negligible 0.0 – 2.4 Seriously flawed

8 Budget and Term Determinations

CIHR's objective is to provide the funds needed to allow approved research to be carried out effectively. To ensure the highest level of accountability in the process, it is critical that reviewers give the budget justification a full and thorough review so that funds are distributed as effectively as possible. The appropriate budget is very much a matter for judgment by the peer review committee. Some areas of research are more expensive than others. In addition, for a potentially fundable application, committees may recommend funds for only those parts of the proposed research deemed worthy on scientific grounds.

Peer review committees should use a zero-based approach to determine the funds required for the research thought worthy of support. In other words, the budget for each year must be built "from the ground up" with each line item justified. A percentage change from current funding is not an appropriate rationale for a budget. Committees should not make their budget recommendations in the context of perceptions of CIHR's budget; the Scientific Council itself will modify the amount for each grant if it feels it is necessary.

The sum of research staff, trainees, and materials and supplies gives a total annual operating budget recommendation. For some types of research (e.g., biomedical laboratory research) this sum is usually rolled over from year to year, while other projects (e.g., clinical trials, epidemiological studies etc.) may require differing amounts of support in different years. In the latter case, the recommended amounts, by year, should be specified.

If the peer review committee feels that the budget is not adequately justified or explained in order to assess the request appropriately, the committee may request a follow-up by CIHR staff. If the proposal is approved for funding, funds will not be released until the budget justification concerns are resolved. In rare instances with explicit justification, the budget and/or term recommended may exceed that requested by the applicant.

Consideration of the budget should include the following factors:

8.1 Research Staff

Research staff (research associates, research assistants, technicians, etc.) should be determined by the actual needs for the techniques and work required for the research. The salary scales put forward by the institution should be followed, especially if the positions of people already employed are to be continued. A starting salary should not be substituted for the higher salary of a named incumbent with a record of continued employment with the applicant. Some institutions require non-discretionary benefits packages for staff (e.g. supplementary medical and dental insurances). These are considered eligible expenses on grants and can be requested as part of operating grant budgets. Salaries for applicants (Principal Investigators or Co-Investigators) cannot be paid from the grant or any other CIHR grant, except in the case of research associates and trainees, when the salary or stipend should be addressed in the budget justification.

Graduate students may be hired as research personnel on a grant. In general this is on a part-time basis, i.e., hourly. This situation is to be distinguished from a graduate student receiving a stipend from a grant (see below), in which case the work done is part of the training of the student and constitutes the thesis or comparable academic requirement.

8.2 Trainees and Undergraduates

The assessment of requests for support of trainees should take into account the quality of the training environment. The committee's judgment of the quality of training will be influenced by such factors as the rating given the project, the nature of the research program and the project(s) on which the trainee(s) might work, the track record of the applicant in training young researchers, and the environment within which the applicant is working. It is helpful, but not essential, that the applicant has identified the project to which the trainee(s) will be assigned.

As of September 15, 2011, the three federal granting agencies' existing regulations on minimum and maximum stipend levels paid from grants will no longer apply. The agencies will also no longer restrict researchers from using some of their grant money to provide supplements to scholarship holders. These changes respond to concerns expressed by institutions and researchers about the different approaches taken by the agencies.

The removal of tri-agency regulations will offer professors more financial flexibility to pay students and postdoctoral fellows at levels that reflect the variations in costs between departments and between regions. It should be noted that stipend levels will still be governed by applicable collective agreements and labour legislation. Also, many universities have established minimum support packages, varying by department, where stipend support can be assembled from teaching assistantships, research funds, university graduate scholarships, external scholarships, and bursaries.

8.3 Material, Supplies, Services and Travel

A budget should be established for materials and supplies, consumables, animal costs, etc., to include all non-personnel requirements for funds. To facilitate budget revisions, committees may wish to build these totals by categories (e.g., animals, isotopes, chemicals or other reagents, glassware, computer software packages, payments to subjects, access to databases, data analysis, printing, publication costs, travel for research personnel or research subjects). Alternatively, committees may wish to make blanket estimates of the usual operating and travel costs per research worker (which will vary according to the type of work being done), though special costs (e.g., travel for collaboration, or unusual animal care or maintenance costs) should not be forgotten.

In general, CIHR grant funds may be used to cover only the direct costs of research and may not be used for indirect costs (e.g., library, heat and light, office furniture, regular telephone rates etc). See the Tri-Agency (CIHR, NSERC & SSHRC) Financial Administration Guide for further details. If a budget request includes amounts for what may seem to be indirect costs or "overhead," these need to be justified in terms of their direct contribution to the research. For example, a contribution to the salary of a glassware washer or technician to operate a common piece of research equipment is allowable, while a departmental "tax" to cover costs other than research expenses (e.g., library acquisitions, graduate student stipends, secretarial pool, etc.) is not allowable. The latter should normally be covered by the institution. When in doubt, such costs should be flagged for CIHR staff to follow up.

8.4 Equipment

Depending on the funding opportunity, equipment costs may be requested for identified items required to do the research. As of January 2012, funding for equipment does not have to be requested in the first year. It can be requested at any time during the grant term.

To facilitate the ability of researchers to obtain the necessary supplies for their research, CIHR has revised its definition of "equipment" effective for application deadlines of August 1, 2009 and later.

Research Equipment: Any item (or interrelated collection of items comprising a system) of nonexpendable tangible property, having a useful life of more than 1 year and a cost of $2,000 or more, which is used wholly or in part for research.

Note that all three conditions must be met for an item to be considered equipment:

  1. nonexpendable tangible property; and,
  2. useful life of more than 1 year; and,
  3. a cost of $2,000 or more.

To be considered materials or supplies an item must meet only one of the following conditions:

  1. expendable tangible property; or,
  2. useful life of 1 year or less; or,
  3. a cost of less than $2,000.

For example, a laptop computer that costs less than $2,000 would be considered as materials or supplies even though it is a nonexpendable tangible item with a useful life of more than one year.

A cost quotation must be provided for equipment or service contracts greater than $10,000. Two competitive quotes as well as letters from an appropriate institutional official documenting the availability and status of similar equipment are required for items costing more than $25,000.

8.5 Overlaps with Other Funding Sources

Peer review committees are asked to recommend budgets for grant applications irrespective of other sources of funds received or applied for, i.e., committees should not reduce recommended budgets to take into account potential overlap.

8.6 Term of Support

The term of support should be a direct reflection of the amount of time the peer review committee feels is necessary to complete the proposed work, if funded. Reviewers should be particularly mindful of the fact that it takes some time for a new investigator to build up a research program and momentum can be jeopardized by having to apply for renewal within a short time after having received a first grant. If, on its merits, a potentially fundable application requires a longer term to fulfill its objectives, it is inappropriate to limit the term to "hedge" against a new investigator's inexperience. In addition, committees may be reluctant to provide long term support for an application for which feasibility of the entire program depends on a positive outcome to initial experiments. In these cases, a 1 or 2 year term may be appropriate for an application that has a high element of risk, such that more substantive proof can be obtained, but it is inappropriate to limit the term of support simply because a proposal is untested or innovative.

Appendix I: Sequence of Steps for Peer Review of a Grant Application

  1. Initial ratings:
    Members in conflict leave the room. The two internal reviewers announce their initial rating.

    Note: ratings can be different from those previously posted on ResearchNet.
  2. Streamlining (OPTIONAL):
    Review is terminated if the following conditions are met:
    • application is not considered competitive by both reviewers (placed into bottom group by both reviewers)
    • the mean of the rating of the two internal reviewers is < 3.50
    • there is no objection from other committee members
    Committee members will not vote and no budget discussions will occur. The rating is calculated as the mean of the rating of the two internal reviewers. No SO notes will be taken.
  3. Internal Reviewers:
    • just enough summary to introduce the grant
    • review applications strengths and weaknesses (based on the evaluation criteria from the Funding Opportunity)
  4. External Reviewers (if applicable):
    • Chair identifies external reviewers
    • A committee member presents external reviewers' comments
  5. Reader (if applicable) raises additional issues
  6. Committee discussion of application should focus on:
    • factors important in rating
    • differences of view between reviewers
  7. Scientific Officer reads SO notes to the committee:
    • summary of discussions
    • strength and weaknesses of application
  8. Consensus rating by internal reviewers:
    • use full scale
    • check consistency with previous applications
    If a consensus cannot be reached, the Chair declares the consensus score usually by using the mean value of the revised respective scores after discussion.
  9. Individual ratings:
    • ± 0.5 of consensus rating
    • votes are confidential
    • the two internal reviewers are not bound to consensus rating nor their initial rating
  10. Community Reviewer (if present):
    • provides comments on a selection of lay abstracts that he/she has chosen to highlight to the committee
    • comments on the extent to which the intent and importance of the proposed research is well explained and in a language clear to members of the general public
  11. Budget:
    Not required if the application is not discussed or if the consensus rating is ≤ 2.9 following discussion.
    • annual operating base: in some applications, requested amounts may vary in different years
    • trainees: excellence of training and environment
    • equipment: justification, itemized, awarded in the first year
  12. Term of grant
  13. Issues to be flagged:
    • eligibility
    • ethics
    • human stem cells
    • environmental impact
    • budget justification
  14. Scientific Officer reads final notes (including budget comments) for review / modifications / additions by committee

Appendix II: Sequence of Steps for Merit Review of an Application

  1. Initial ratings:
    Members in conflict leave the room. The two internal reviewers announce their two initial ratings: one for the Potential Impact (PI) and one for the Scientific Merit (SM) of the research proposal.
  2. Streamlining (OPTIONAL):
    An application does not need to be discussed if:
    • the mean of the score of the two internal reviewers for both PI and SM is < 3.5
    • there is no objection from other committee members
    Committee members will not vote and no budget discussions will occur. The rating is calculated as the mean of the four respective scores (PI & SM) initially announced by the primary and secondary reviewers. No SO notes will be taken.
  3. Internal reviewers:
    • just enough summary to introduce the grant
    • review applications strengths and weaknesses (based on the evaluation criteria from the Funding Opportunity)
  4. Reader(s) raises additional issues.
  5. Committee discussion of application focusing on:
    • factors influencing the review of the application and hence the rating;
    • differences of view between reviewers
  6. Scientific Officer reads his/her SO notes to the committee:
    • summary of discussion
    • strengths and weaknesses of application.
  7. Consensus rating for both PI and SM is reached by the two reviewers by:
    • using the full rating scale
    • ensuring consistency with previous applications
    If a consensus cannot be reached for either PI or SM, the Chair declares the consensus score by using the mean value of the revised respective scores after discussion
  8. Individual ratings:
    • Committee members vote ± 0.5 of consensus ratings (PI and SM)
    • Votes are confidential
    • The two internal reviewers are not bound to the consensus ratings nor their initial ratings.
  9. Budget:
    Not required if the application is not discussed or if the consensus rating is ≤ 2.9 following discussion.
    • annual operating base: in some applications, requested amounts may vary in different years
    • trainees: excellence of training and environment
    • equipment: justification, itemized, awarded in the first year
  10. Term of grant
  11. Issues to be flagged:
    • eligibility
    • ethics
    • human stem cells
    • environmental impact
    • budget justification
  12. Scientific Officer reads final notes (including budget comments and other issues) for review / modifications / additions by committee

* Each application will be reviewed by two reviewers – a knowledge user and a researcher – and one reader. The reviewers will focus on the assessment of the Potential Impact (PI) of the project and the assessment of the Scientific Merit (SM). Readers will act as discussants and don't need to provide a written review.

** Each application will be reviewed by two reviewers and two readers for Community-Based Research (CBR) program.

Appendix III: Example Scoring for Merit Review

Potential Impact Scientific Merit
  1. Average of confidential votes—calculated by CIHR database after the meeting.
  2. Average of the two final ratings—used to establish the ranking of applications for a particular competition. Please note that to be considered for funding both the potential impact and the scientific merit ratings have to be above 3.5. In this example, the application would not be considered for funding even though its ranking rating is above 3.5.
Initial rating (e.g. 3.4) 1st reviewer Initial rating (e.g. 3.8) 2nd reviewer Initial rating (e.g. 3.6) 1st reviewer Initial rating (e.g. 4.0) 2nd reviewer
Consensus rating (e.g. 3.5) Consensus rating (e.g. 3.7)
Final rating6 (e.g. 3.45) Final rating6 (e.g. 3.82)
Ranking rating7 (e.g. 3.63)