Peer Review Manual - Foundation

Contents

  1. Section 1 – Adjudication Process for Stage 1
  2. Section 2 – Adjudication Process for Stage 2
  3. Section 3 – Adjudication Process for the Final Assessment Stage

Please note that in the context of this document, “the applicant” may refer to an individual Program Leader or multiple Program co-Leaders.

Purpose of this manual

The purpose of this manual is to provide reviewers with a step-by-step walk through of the adjudication process for the Foundation Scheme Pilots. This manual is divided into three sections which correspond to the three stages of the adjudication process as illustrated in Figure 1.

Section 1 outlines the adjudication process for Stage 1 of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: (1) preliminary review of the applications, (2) asynchronous online discussion, and (3) submission of the final rank list to CIHR.

Section 2 outlines the adjudication process for Stage 2 of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: (1) preliminary review of the applications, (2) asynchronous online discussion, and (3) submission of the final rank list to CIHR.

Section 3 outlines the adjudication process for the final stage of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: (1) pre-meeting activities, and (2) a face-to-face committee meeting.

Some reviewers may be invited to participate in both Stage 1 and Stage 2 if required. The Final Assessment Stage face-to-face committee will be composed of reviewers who have not participated as reviewers in the previous review stages. New/Early Career Investigators will be assessed and ranked against other New/Early Career Investigators at each stage of the adjudication process.

In addition to reviewing the material in this manual, it is essential that peer reviewers read and become familiar with the Foundation Scheme: 2015 2nd Live Pilot Funding Opportunity. All CIHR peer reviewers are also expected to:

For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.

Figure 1. Overview of the Foundation Scheme Adjudication Process

Long description: Figure 1

Section 1 – Adjudication Process for Stage 1

This section outlines the responsibilities of reviewers participating in the Stage 1 adjudication process. The adjudication process includes the following:

  1. preliminary review of the applications,
  2. asynchronous online discussion, and
  3. submission of the final rank list to CIHR.

1.1 Preliminary Reviews of Applications

Stage 1 focuses on the Caliber of the Applicant and the Vision and Program Direction.

Each application will be assigned to up to five reviewers, based on optimal matching between the application content and reviewer expertise. Each reviewer will receive between 10 and 16 applications. Reviewers will access their assigned applications through ResearchNet and complete their reviews remotely. It is expected that it will take approximately 1-3 hours to complete each review.

The application will be presented in a structured format to align with the adjudication criteria, outlined in the Interpretation Guidelines: Foundation Scheme Adjudication Criteria. Reviewers will also be asked to review the applicant CV(s). Through their CV(s), applicants will highlight their recognitions, funding history, activities and contributions that best demonstrate their leadership, significant contributions and productivity in the context of their research field(s).

Reviewers will be asked to provide a rating and to justify the rating by stating the strengths and weaknesses for each sub-criterion outlined in the table below. It is important for reviewers to clearly articulate the strengths and weaknesses as they will be used for two purposes: 1) to provide applicants with feedback and 2) to provide the other reviewers assigned to the application with a justification for the ratings and rankings given to the application.

The Interpretation Guidelines: Foundation Scheme Adjudication Criteria outlines a set of interpretation guidelines for each adjudication criterion. Reviewers must consult the Interpretation Guidelines for guidance on how to apply the adjudication criteria.

In some cases, there may be multiple Program Leaders for a single Foundation grant. These applicants will need to demonstrate synergy and a history of co-leading programs of research. In these cases, the assessment of each adjudication criterion should consider both the individual and joint research contributions of the Program co-Leaders.

Reviewers must take into consideration the career stage, research field and institution setting of all applicants when assessing each criterion. New/Early Career Investigators may show evidence of leadership, contributions and productivity in different ways than Mid Career/Senior Investigators. The evidence should be notable compared to peers in similar fields and career stages.

To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale acknowledging that many of the ratings are likely to fall into this range. Reviewers are encouraged to use the full breadth of the scale and should use the increased granularity within the top descriptors to express differences within these categories. To facilitate this, the following scale including descriptors and definitions is provided:

Descriptor Abbreviation Definition

Outstanding

O++

For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.

O+

O

Excellent

E++

For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.

E+

E

Good

G

For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.

Fair

F

For this sub-criterion, the application broadly addresses all relevant aspects. Major revisions are required.

Poor

P

For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

As each review is completed, ResearchNet will automatically show a ranked list of the individual reviewer's set of assigned applications based on the ratings (highest to lowest). Reviewers may make adjustments to their ratings until they are comfortable with the rankings, however, there will be no requirement to break ties at this stage of the adjudication process. Ultimately it is the application's ranking that will determine its standing within the competition and will be used to make funding decisions.

The reviewers will then be asked to submit their preliminary reviews to CIHR in order to proceed to the next step.

1.2 Online Discussion

Once the preliminary reviews are submitted, the other reviewer's reviews will be made available. Reviewers will be able to discuss each of their assigned applications with up to four other reviewers who were assigned the same application. An asynchronous online discussion tool has been developed to allow reviewers to communicate with each other to share expert perspectives and to discuss any discrepancies.

The objective of the discussion will be to discuss and understand discrepancies in reviews. There will be no requirement to reach consensus. Each reviewer's name and their preliminary review (ratings, written comments and ranking) will be visible for each application. If there are limited discrepancies in reviews, it is possible that a discussion may not be required.

The discussion period will last 72 hours and reviewers will be made aware of the planned discussion period for the competition well in advance of the deadlines. Reviewers may log in at any time during the discussion period and will not be expected to participate in online discussions simultaneously.

All five reviewers assigned to an application will be able to participate in the discussion as soon as all of the reviews for that application have been submitted. The discussion for all applications will open on the established deadline and it is therefore important to respect the established deadlines. Reviewers who have not submitted their preliminary reviews will not be able to participate in the discussion and this will be flagged in their online status.

A Virtual Chair will also be assigned to the application to moderate the discussion. The Virtual Chair will be tasked with ensuring that discussions take place if warranted, and may prompt reviewers to discuss. CIHR staff may also initiate a discussion, or provide input where necessary, to ensure that CIHR policies and procedures are followed.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings and/or rankings.

It should be noted:

  1. Every reviewer will have the ability to initiate a discussion thread. Every comment will be shared with every reviewer assigned to that application. Once comments are posted, reviewers will not be able to delete or edit them.
  2. Reviewers will be able to post comments for the attention of CIHR staff; however they will be visible by the other reviewers.
  3. A notification email will be sent daily to advise reviewers of new posts.
  4. As applications are discussed, reviewers may feel the need to update their preliminary ratings and reviews. This can happen at any time both during and after the discussion period. These modifications will not be accessible to other reviewers.
  5. A transcript of the online discussions will not be given to the applicants. However, discussion thread content will be retained by CIHR as part of the application record and this record is subject to the Government of Canada’s Access to Information Policy and Privacy Acts.

1.3 Submit the Final Rank List to CIHR

Once the asynchronous discussions are complete, reviewers will be asked to finalize their reviews and their individual ranked list. At this point, reviewers must break any ties in their individual ranked list. Once ties are eliminated, reviewers may modify the rank order of their assigned applications by moving the applications up or down the rank list. The original rank order will remain visible as a reference point. Once satisfied with the rank order of applications that they have been assigned, the reviewers will be asked to submit the final rankings to CIHR. Note that reviewers will not see the final ranked lists of other reviewers. At this point, the adjudication process for Stage 1 is complete.

Once all rankings have been received, CIHR will calculate a consolidated ranking for each application and compile a list of all applications in the competition ranked from highest to lowest. Each application will have up to five reviews, a ranking from each reviewer, a consolidated ranking and a standard deviation. CIHR will determine which applicants will be invited to complete a Stage 2 application based on the number of applications, consolidated ranking, standard deviation and funds available for the competition.

An overview of the Stage 1 adjudication process is outlined in Figure 2.

Figure 2. Stage 1 Overview

Preliminary Reviews of Assigned Applications

Reviewers will be asked to provide a rating for each adjudication criterion and to justify the rating by providing comments on the strengths and weaknesses. To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale to allow reviewers to indicate finer differences between high quality applications.

The adjudication criteria are:

Criterion 1: Caliber of the Applicant

  • Leadership (25%)
  • Significance of Contributions (25%)
  • Productivity (25%)

Criterion 2: Vision and Program Direction (25%)

Online Discussion

Once the preliminary reviews have been submitted, the other reviewer's preliminary reviews (ratings and comments) will be made available to reviewers who have been assigned the same application. Reviewers will be able to read the preliminary reviews and discuss each of their assigned applications with the four other reviewers who were assigned the same application. This is the reviewers' opportunity to share their expert perspectives, and to discuss any rating discrepancies in a virtual environment.

The Virtual Chair or CIHR staff may initiate discussions when there are large discrepancies between reviewer ratings.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings and/or rankings.

Submit the Final Ranked List to CIHR

A reviewer's set of applications will be ranked in order from highest-rated to lowest-rated based on the ratings given to the application by the reviewer.

At this stage, some applications may be tied (i.e., have the same rank). The reviewer will be required to break these ties by moving one application ahead of the other.

Once the reviewer is satisfied with the ranked list of applications, he/she will submit the ranked list to CIHR. Once all ranked applications have been received, CIHR will compile a list of all applications in the competition ranked from highest to lowest.

The number of applications, consolidated ranking, and the standard deviation of each application, as well as the funds available for the competition will be used to identify which applications will be invited to complete a Stage 2 application.

Section 2 – Adjudication Process for Stage 2

This section outlines the responsibilities of reviewers participating in the Stage 2 adjudication process. The adjudication process includes the following: (1) preliminary review of the applications, (2) asynchronous online discussion, and (3) submission of the final rank list to CIHR.

2.1 Preliminary Reviews of Applications

Stage 2 focuses on the Quality of the Program and Quality of the Expertise, Experience and Resources and is distinct from the Stage 1 application.

Each application will be assigned to up to five reviewers, based on optimal matching between the application content and reviewer expertise. Matching and assignments will be done on the basis of reviewer declarations of their ability to review and with the assistance and oversight of Virtual Chairs. Reviewers will access their assigned applications through ResearchNet and complete their reviews remotely. Reviewers at Stage 2 will not have access to the materials and reviews from Stage 1.

The application will be presented in a structured format to align with the adjudication criteria, outlined in the Interpretation Guidelines: Foundation Scheme Adjudication Criteria.

Reviewers will be asked to provide a rating and to justify the rating by stating the strengths (maximum 1,750 characters) and weaknesses (maximum 1,750 characters) for each sub-criterion outlined in the table below. The fields to outline strengths and weaknesses for each application in ResearchNet are mandatory fields. It is important for reviewers to clearly articulate the strengths and weaknesses as they will be used for three purposes: 1) to provide the other reviewers assigned to the application with a justification for the ratings and rankings given to the application, 2) to provide reviewers in the final stage with a justification for the ratings and rankings given to the application, and 3) to provide applicants with feedback.

The adjudication criteria, descriptors and criteria specific considerations are outlined in the Interpretation Guidelines: Foundation Scheme Adjudication Criteria Reviewers must consult the Interpretation Guidelines for guidance on how the adjudication criteria should be applied.

Reviewers must take into consideration the career stage, research field and institution setting of all applicants when assessing each criterion.

To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale acknowledging that many of the ratings are likely to fall into this range. Reviewers are encouraged to use the full breadth of the scale and should use the increased granularity within the top descriptors to express differences within these categories. To facilitate this, the following scale including descriptors and definitions is provided:

Descriptor Abbreviation Definition

Outstanding

O++

For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.

O+

O

Excellent

E++

For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.

E+

E

Good

G

For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.

Fair

F

For this sub-criterion, the application broadly addresses all relevant aspects. Major revisions are required.

Poor

P

For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

As each review is completed, ResearchNet will automatically show a ranked list of the individual reviewer's set of assigned applications based on the ratings (highest to lowest). Reviewers may make adjustments to their ratings until they are comfortable with the rankings, however, there will be no requirement to break ties at this stage of the adjudication process.

Reviewers will also be asked to flag applications in ResearchNet that involve any of the issues outlined below. These issues are not to be considered as criteria for adjudication unless they impact the scientific quality of the application. For detailed regulations concerning these issues, please refer to the CIHR Grants and Awards Guide.

  1. Ethical and other policy requirements: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may flag specific issues, such as the use of human participants, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  2. Human pluripotent stem cell research: Applications involving the use of human stem cells which will be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled "Certification Requirements", but it is essential that this be reviewed by reviewers.
  3. Section 56 of the Federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Reviewers should flag such applications to CIHR staff who will follow up before funds are released, if the application is funded.

2.2 Budget Assessment

Reviewers will be required to review the requested budget and justification. Applicants will be asked to submit their budget using a modular template with budget increments. Reviewers will be provided with the applicants' baseline funding amount as calculated by CIHR and as provided to applicants at the time of application. CIHR expects that the budget request for the program of research be consistent with the budget baseline established by CIHR. Any increase above the calculated baseline amount will need to be justified by the applicants. Reviewers will be required to determine if the requested budget is appropriate to support the proposed program of research, and if it is realistic and well-justified given their funding baseline. Reviewers will be asked to assess the appropriateness of justification for requests that are higher than applicants' funding baseline amounts. Appropriate budget planning should be demonstrated. The requested resources, together with any existing resources, should be proportionate to the requirements of the research proposed, and will vary depending on the research field, research approach, and scope of program activities.

Partner contributions, if applicable, should be considered when reviewing support for the proposed program of research. Partner contributions will be outlined in letters of contribution and in the budget module through ResearchNet.

Please note that the budget assessment must not be factored into the scientific assessment. Moreover, detailed scrutiny of each item will not be expected. Reviewers may recommend that the budget remain as requested or recommend an adjusted amount. If a reviewer adjusts the budget, he/she will be required to provide comments to justify their recommendation.

During the asynchronous online discussion reviewers will be required to discuss any discrepancies in their budget recommendations. In cases where a discrepancy still exists, the budget will be reconciled at the end of Stage 2 by CIHR staff and the Virtual Chair. The Virtual Chair may communicate with Stage 2 reviewers if there is a need for clarification.

The reviewers will then be asked to submit their preliminary reviews to CIHR in order to proceed to the next step.

2.3 Online Discussion

Once the preliminary reviews are submitted, the other reviewer's reviews will be accessible through ResearchNet. Reviewers will be able to discuss each of their assigned applications with the four other reviewers who were assigned the same application. An asynchronous online discussion tool has been developed to allow reviewers to communicate with each other to share expert perspectives and to discuss any discrepancies.

The objective of the discussion will be to discuss and understand discrepancies in reviews and budget recommendations. There will be no requirement to reach consensus. Each reviewer's name and their preliminary review (ratings, written comments, budget recommendation and ranking) will be visible for each application. At this point, if any reviewer has suggested a change in the budget, this should also be discussed. If there are limited discrepancies in reviews, it is possible that a discussion may not be required.

The discussion period will last 72 hours and reviewers will be made aware of the scheduled discussion period for the competition well in advance of the deadlines. Reviewers may log in at any time during the discussion period and will not be expected to participate in online discussions simultaneously.

All five reviewers assigned to an application will be able to participate in the discussion as soon as all of the reviews for that application have been submitted. The discussion for all applications will open on the scheduled discussion period start date therefore it is important to respect the established deadlines. Reviewers who have not submitted their preliminary reviews will not be able to participate in the discussion and this will be flagged in their online status.

A Virtual Chair will also be assigned to the application to moderate the discussion. The Virtual Chair will be tasked with ensuring that discussions take place if warranted, and may prompt reviewers to discuss an application or revise or expand on their review. CIHR staff may also initiate a discussion, or provide input where necessary, to ensure that CIHR policies and procedures are followed.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings, and/or rankings.

It should be noted:

  1. Every reviewer will have the ability to initiate a discussion thread. Every comment will be shared with every reviewer assigned to that application. Once comments are posted, reviewers will not be able to delete or edit them.
  2. Reviewers will be able to post comments for the attention of CIHR staff. These comments will be visible to the other reviewers.
  3. A notification email will be sent daily to advise reviewers of new posts.
  4. As applications are discussed, reviewers may feel the need to update their preliminary ratings and reviews. This can happen at any time both during and after the discussion period. These modifications will not be accessible to other reviewers.
  5. A transcript of the online discussions will not be given to the applicants. However, discussion thread content will be retained by CIHR as part of the application record and this record is subject to the Government of Canada’s Access to Information Policy and Privacy Acts.

2.4 Submit the Final Rank List to CIHR

Once the asynchronous discussions are complete, reviewers will be asked to finalize their reviews and their individual ranked list. At this point, reviewers must break any ties in their individual ranked list. Once ties are eliminated, reviewers may modify the rank order of their assigned applications by moving the applications up or down the rank list. The original rank order will remain visible as a reference point. Once satisfied with the rank order of applications that they have been assigned, the reviewers will be asked to submit the final rankings to CIHR. Note that reviewers will not see the final rank lists of other reviewers. At this point, the adjudication process for Stage 2 is complete.

Once all rankings have been received, CIHR will calculate a consolidated ranking for each application and compile a list of all applications in the competition ranked from highest to lowest. Each application will receive up to five reviews, a consolidated ranking and a standard deviation for Stage 2.

CIHR will determine which applicants will be discussed at the Final Assessment Stage based on the number of applications, consolidated ranking, standard deviation and funds available for the competition. Based on the results from Stage 2, CIHR will identify the highest ranked applications that will be considered for funding at this stage. These applications will constitute the "green zone" and do not require further discussion.

A second set of applications will move to the Final Assessment Stage. These applications will constitute the "grey zone". The "grey zone" applications are those applications that were highly ranked and have a large standard deviation due to discrepancies in reviewer ranking.

The bottom-ranked applications will be deemed unsuccessful, and will not be considered for funding. After the Final Assessment Stage, a notice of decision will be provided to all applicants.

An overview of the Stage 2 adjudication process is outlined in Figure 3.

Figure 3. Stage 2 Overview

Preliminary Reviews of Applications

Reviewers will be asked to provide a rating for each adjudication criterion, and then to justify the rating by providing comments on the strengths and weaknesses. To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale to allow reviewers to indicate finer differences between high quality applications.

The adjudication criteria are:

Criterion 1: Quality of the Program

  • Research Concept (25%)
  • Research Approach (25%)

Criterion 2: Quality of the Expertise, Experience, and Resources

  • Expertise (20%)
  • Mentorship and Training (20%)
  • Quality of Support Environment (10%)

Budget Assessment

Reviewers will be required to review the requested budget and the budget justification. Reviewers will be provided with some historical information about the applicant's open grant research funding history with CIHR. Reviewers will be required to determine if the requested budget is appropriate to support the proposed program of research, if it is realistic and well-justified.

Partner contributions, if applicable, should be considered when reviewing support for the proposed program of research.

Please note that the budget assessment must not be factored into the scientific assessment. Reviewers may recommend that the budget remain as requested or recommend an adjusted amount. If a reviewer adjusts the budget, he/she will be required to provide comments to justify their recommendation.

Online Discussion

Once the preliminary reviews have been submitted, the other reviewer's preliminary reviews (ratings and comments) will be accessible through ResearchNet to reviewers who have been assigned the same application. Reviewers will be able to read the preliminary reviews and discuss each of their assigned applications with the four other reviewers who were assigned the same application. This is the reviewers' opportunity to share their expert perspectives, and to discuss any rating discrepancies in a virtual environment.

Reviewers will be required to discuss any discrepancies in their budget recommendations. In cases where a discrepancy still exists, the budget will be reconciled at the end of Stage 2 by CIHR staff and the Virtual Chair. The Virtual Chair may communicate with Stage 2 reviewers if there is a need for clarification.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings, and/or rankings.

Submit the Final Ranked List to CIHR

A reviewer's set of applications will be ranked in order from highest-rated to lowest-rated based on the ratings given to the application by the reviewer.

At this stage, some applications may be tied (i.e., have the same rank). The reviewer will be required to break these ties by moving one application ahead of the other.

Once the reviewer is satisfied with the ranked list of applications, he/she will submit the ranked list to CIHR. Once all ranked applications have been received, CIHR will compile a list of all applications in the competition ranked from highest to lowest.

The number of applications, consolidated ranking, and the standard deviation of the applications, as well as the funds available for the competition will be used to identify which application will be discussed at the Final Assessment Stage.

Section 3 – Adjudication Process for the Final Assessment Stage

This section outlines the responsibilities of reviewers participating in the Final Assessment Stage. The adjudication process includes the following: 1) pre-meeting activities, and 2) a face-to-face committee meeting.

This stage will be completed by one multidisciplinary committee covering the full spectrum of health research. The committee will include a Chair and Scientific Officer (SO). This committee will be responsible for integrating the results of the Stage 2 reviews and making recommendations on which "grey zone" applications should be funded.

3.1 Pre-Meeting Activities

Prior to the face-to-face meeting, each committee member will be assigned a subset of applications and each application will be assigned to three committee members. For each application, the committee member will have access to information from Stage 2, including the reviews, the consolidated rankings, standard deviations and the full applications.

In order to help committee members differentiate between applications, a binning system will be used. Reviewers will be asked to assign applications to a "Yes" bin (to be considered for funding) or a "No" bin (not to be considered for funding). Each reviewer will be allocated a minimum number of applications that may be placed in the "Yes" and "No" bins. This number will depend on the funds available for a given competition. Committee members will be asked to submit their recommendations to CIHR prior to the meeting.

Based on the yes/no binning recommendations reviewers made as part of the pre-meeting activities, CIHR will rank all the Final Assessment Stage applications in order from the highest ranked to lowest ranked. The primary sort criteria will be the number of “Yes” votes received, and the secondary sort criteria will be based on the Stage 2 results.

The Chair of the committee will have an opportunity to review the recommendations from the committee members prior to the meeting. CIHR staff will prepare a schedule for the face-to-face committee meeting in consultation with the Chair and SO. An illustration of the process is provided in Figure 4.

Figure 4. Pre-Committee Meeting Overview

Reviewer assigned a subset of applications

Reviewer reads the Stage 2 reviews of their assigned applications.

Based on the reviews, the Final Assessment Stage reviewer must choose which applications should be considered for funding and which ones should not. The reviewer will receive a minimum number of applications that they may place in the "Yes" Bin (to be considered for funding), and the rest of the applications will be placed in the "No" Bin (not to be considered for funding).

Based on the yes/no binning recommendations reviewers made as part of the pre-meeting activities, CIHR will rank all Final Assessment Stage applications in order from the highest ranked to lowest ranked. The primary sort criteria will be the number of "Yes" votes received, and the secondary sort criteria will be based on Stage 2 results.

3.2 Face-to-Face Committee Meeting

The multidisciplinary committee will be asked to make recommendations on which applications in the "grey zone" should be considered for funding.

Prior to the meeting, CIHR will have compiled the results from the pre-meeting activities where committee members binned their assigned applications. At the meeting, committee members will see the grant applications ordered from highest-rated to lowest-rated according to a combination of the pre-meeting yes/no binning recommendations and Stage 2 results. Based on these results, CIHR will place the applications into one of three groups:

  1. Group A (applications that the assigned committee members recommended should be funded)
  2. Group B (applications for discussion at the meeting)
  3. Group C (applications that the assigned committee members recommended should not be funded)

The meeting will start with the committee Chair asking the committee members to validate the groupings of applications. To validate, the committee Chair will ask if any committee member would like to move any of the proposed Group A or Group C applications into Group B so that they may be discussed by the committee. Once the committee is satisfied with the groupings, the Group A and Group C applications will be locked in and will not be discussed any further at the meeting.

Please note that an estimated funding cut-off line will be displayed within the list of Group B applications. This line is calculated using both the available budget for the competition and the budget requested in the grant applications. The estimated funding cut-off line is meant to be a tool to facilitate discussion and may not be entirely accurate as budget reductions may occur.

The committee should consider the estimated funding cut-off line when adding applications to Group B for discussion. For example, if only five grants will be able to be funded from the Group B pool, then it may not be effective or efficient to move 20 grants from Group C into Group B. However, the grant application budget should not be a factor in making recommendations for funding (i.e., should not be used to vote a high-budget grant out of the funding range in favour of more, but smaller-budget, grants). CIHR is committed to funding excellent applications.

The Chair will moderate the discussion of all Group B applications, and the SO will prepare the notes that summarize the key points discussed for each application.

For each application that is discussed, the following process will be used:

  1. Any committee member in conflict with the application will be asked to leave the room.
  2. The Chair will ask the three reviewers who reviewed the application to indicate whether they voted "Yes" or "No" during the pre-meeting activities.
  3. The Chair will ask the assigned member(s) who voted the application "Yes" to start the discussion and then the member(s) who voted "No" to continue the discussion. The discussion will focus on the key points that led to their yes/no assessments.
  4. Committee members not assigned to the application will be asked to contribute to the conversation by asking questions and adding their opinions/advice/expertise to clarify points of disagreement.
  5. The SO will read the SO notes, which will capture the key elements of the discussion, to the committee for approval.

Once all of the applications in Group B have been discussed, members will be asked to vote on the Group B applications. Members will be provided with a maximum number of "Yes" votes (to be considered for funding). This maximum will be based on both the number of applications in Group B and the estimated funding cut-off line.

The committee Chair will go through the Group B applications one at a time, calling for a vote on each application. Using the electronic voting tool, committee members will vote either "Yes" or "No" for any application with which they are not in conflict. The Chair and SO will not be permitted to vote.

Committee members will be expected to adhere to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations. While committee members will not be in the room during the discussion of the applications with which they are in conflict, they will not be asked to leave during the voting process. They will not be permitted to vote on any in-conflict applications, however, they will see the final rank order of all applications.

The committee will conclude its work after all the members have agreed on the applications they recommend for funding. These recommendations will be summarized and submitted to CIHR for the final review of budgets and the funding decision. Applicants will be informed of the results of the competition once CIHR's President has approved the grants to be funded. After this stage all applicants will be sent a Notice of Decision.

An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, as well as a discussion of policy issues that may have arisen in the course of its deliberations. This discussion is important as it provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the piloted adjudication process.

Applicants, research administrators and reviewers will be asked to complete surveys after each stage to provide their feedback. Feedback received through the surveys will help refine the design of the new Open funding programs and peer review processes.

Date modified: