Streamlining of Applications at Peer Review Committee Meetings

Background

The evaluation of applications for funding occurs in two phases: (1) an in-depth "at-home" review by at least two peer reviewers to produce written evaluations, and (2) a committee meeting to discuss and rate the applications, from which CIHR generates a rank-order priority list to make funding decisions. For many programs, less than one-third of the applications are ultimately funded; thus, it is important that committees focus their discussions on the most competitive applications to ensure that an accurate rank-order list is generated. To help support this goal, a 'streamlining' process is used to eliminate non-competitive applications from the discussion process, allowing peer reviewers more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding. Applicants whose proposals are streamlined still benefit from the review process as they receive the written reviews from the assigned reviewers.

Procedure

Streamlining focuses on an "assessment of overall quality", independent of the initial rating. This assessment makes streamlining more efficient, accounts for different reviewer scoring preferences and avoids upward compression of ratings. The process is detailed below and supported by ResearchNet.

1. Prior to the committee meeting

   Reviewers will divide their applications into a top and bottom group, based on their overall quality. The top group should include those applications considered to be highly competitive and most deserving of being funded. This assessment is to be based on the reviewers' total experience with applications over the past five years. Note that this allows reviewers to make an assessment even if they have only reviewed a few applications for that committee meeting (e.g. teleconference reviewers). The proportion of applications in each group may vary depending on the overall quality of the applications reviewed, relative to the reviewers' experience. This step is not applicable to Merit Review.

2. At the committee meeting

   The assessment of each application at peer review committee meetings begins with internal reviewers announcing their initial ratings to one decimal place. An application is then streamlined if it meets the following conditions:

   For Peer Review

   1. both reviewers placed the application in their bottom group;
   2. the average of the internal reviewers' initial ratings is < 3.50;
   3. there is no objection from the other committee members that the application not be discussed.

   For Merit Review

   1. the average of the internal reviewers' initial ratings for both Potential Impact and Scientific Merit is < 3.5;
   2. there is no objection from the other committee members that the application not be discussed.

   If any of these conditions is not met, the application must be discussed. Any peer review committee member who is not in conflict with an application may flag it for discussion if he/she believes that there are factors that should be further considered or has any uncertainty regarding its quality.

   Example for Merit Review

   Candidate for Streamlining	Reviewer’s initial rating				Mean
   	Scientific Merit		Potential Impact		SM	PI
   No	2.5	3.5	3.7	3.9	3.0	3.8
   No	3.2	3.6	3.8	3.2	3.4	3.5
   Yes	3.0	3.6	3.6	3.1	3.3	3.4
   Yes*	2.0	4.0	3.4	3.4	3.0	3.4

   *Normally, given the substantial discrepancy in initial ratings for SM, the reviewers or Chair will ask to discuss the application.

3. Notes

   1. Based on their strategic objectives, other funding programs may decide to use different standards for the level of discussion required for applications.
   2. If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports (and external reviews, if any) and the Scientific Officer notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.

4. Optional Procedure

   At the discretion of the committee Chair, peer review committees may choose to consider all applications that are streamlining candidates together. The goal of this option is to remove non-competitive applications allowing committees to better budget the time spent on the remaining applications.

   This option has two requisites:

   • All reviewers must ensure they complete the top/bottom group assessments on ResearchNet at least five days prior to the meeting;
   • All reviewers must submit their initial scores at least five days prior to the meeting.
   
   The steps involved in this streamlining option are:
   
   
   1. CIHR staff will prepare a list of the applications that are streamlining candidates.
   2. Committee members who are not in conflict will vote on each application as to whether it be removed, or not, from further discussion. The vote to remove an application must be unanimous. There will be no discussion of the merits of the applications permitted at this phase. The Deputy Director also has the right to flag any application that he/she feels ought to be discussed. If an application is removed by unanimous vote reviewers will enter a T on their ratings sheet and the final rating will be calculated as the mean of the internal reviewers' initial ratings.
   3. Members in conflict with particular applications need not absent themselves from the room during this process. However, they will not participate in the voting for those applications or comment on them in any way. It is incumbent upon all reviewers to respect the need to protect the process from bias and to refrain from any commentary whatsoever regarding application quality.
   4. If a streamlining candidate was flagged for further discussion during the streamlining process, it may still be removed after a brief discussion if there does not appear to be sufficient merit for further consideration. This decision is left to the discretion of the Chair.
   5. Teleconference reviewers can participate in streamlining at the start of the meeting if they are able to join the meeting at that time; otherwise, a streamlining decision can be made during the meeting when their applications come up for review.

5. Streamlining of Applications – Best practice

   • Please refer to the Peer Review Manual and the guidelines prior to the meeting;
   • With as little as 0.01 points separating a funded application from an unfunded one, it is important that committees have sufficient time to fully discuss applications in the fundable range to ensure that the most deserving applications get funded;
   • Streamlining of applications can be done all at once or throughout the day;
   • If the committee decides to streamline applications all at once, it should be done after lunch for example. This will allow the reviewers to calibrate their scores and have a better sense of the quality of all applications;
   • If there is a significant discrepancy in the initial ratings, the application should be discussed.