Ranking and Rating Scale Meaning and Use
How does CIHR make its funding decisions?
During peer review, an application is given a rating by the committee. This rating is then used to generate a rank ordered list of applications and CIHR makes its funding decisions based on this list. Applications are funded, in rank order, until all available funding is exhausted. For clarification on how funding decisions are made when there are multiple committees within a funding program, please consult Appendix 1.
How does CIHR rank applications?
CIHR uses a scale of 0 to 4.9 to rate all applications, with 4.9 being the highest possible rating. Ratings are assigned qualitative descriptors (see Tables below) to help guide reviewers in their rating of applications. The purpose of the scale is to serve as a benchmark for peer reviewers as they rank the applications within the committee. In order to be considered for funding, an application must receive a score of 3.50 or higher.
The CIHR Rating Scale
Table 1 – CIHR's rating scale for grant programs
| Descriptor | Range | Outcome |
|---|---|---|
| Outstanding | 4.5 – 4.9 | May be Funded – Will be Discussed by the Committee |
| Excellent | 4.0 – 4.4 | |
| Very good | 3.5 – 3.9 | |
| Acceptable, but low priority | 3.0 – 3.4 | Not Fundable – May or May Not be Discussed by the Committee |
| Needs revision | 2.5 – 2.9 | |
| Needs major revision | 2.0 – 2.4 | |
| Seriously flawed | 1.0 – 1.9 | |
| Rejected | 0.0 – 0.9 |
Table 2 – CIHR's rating scale for salary & training award programs
| Descriptor | Range | Outcome |
|---|---|---|
| Outstanding | 4.5 – 4.9 | May be Funded |
| Excellent | 4.0 – 4.4 | |
| Very good | 3.5 – 3.9 | |
| Good | 3.0 – 3.4 | Not Fundable |
| Average | 2.0 – 2.9 | |
| Below average | 1.0 – 1.9 | |
| Not acceptable | 0.0 – 0.9 |
Table 3 – CIHR's rating scale for merit review:
Merit Review employs separate scores for Potential Impact and Scientific Merit.
| Potential Impact | Range | Scientific Merit | |
|---|---|---|---|
| May Be Funded | Extremely Significant | 4.5 – 4.9 | Exceptional |
| Very Significant | 4.0 – 4.4 | Outstanding | |
| Significant | 3.5 – 3.9 | Excellent, may still require revision | |
| Not Fundable | Moderate | 3.0 – 3.4 | Very good however needs revision to be fundable |
| Limited | 2.5 – 2.9 | Needs major revision | |
| Negligible | 0.0 – 2.4 | Seriously flawed |
* Only those applications that exceed the threshold rating of 3.50 on both Potential Impact and Scientific Merit will be considered for funding. The mean of the two scores will be calculated for the applications with a rating of 3.50 and above on both criteria to determine the final rating and establish a ranking list.
Meaning of Ratings
The assignment of ratings to applications achieves two goals:
- to separate the "fundable" applications (rated 3.50 and above) from the "non-fundable" (rated below 3.50); and
- to generate a rank order of applications based on their quality and scientific merit.
Each committee has a unique calibration process to rate applications relative to each other. Comparisons of absolute ratings for applications should only be made within an individual committee, for a specific competition.
It is also important to note that the rating within a committee can change from competition to competition. For example, an application that was rated 4.2 in a previous competition might be rated 4.0 or 4.4 in a subsequent competition depending on the quality of the applications within the group. In addition, CIHR policy changes have also influenced the meaning of ratings over time: these changes include raising the funding cut-off from 3.00 to 3.50 in 2005/06; and adjusting the streamlining/triage system.
Reporting Competition Results to Applicants
A great deal of emphasis has been placed historically on absolute ratings of applications, but what matters is how well one does relative to the other application (i.e., the application's rank). In a funding competition, the most relevant information to the applicant is:
- Funded or not funded
- SO notes (if provided) – basis of decision
- Reviewers' comments – feedback on proposal
- How close to funding cut-off:
- Rank in the committee
- Total number of applications reviewed by the committee
- Number of applications funded in the committee
Appendix 1
Example of Funding Allocation for a competition with multiple committees:
Some funding opportunities at CIHR have multiple committees within a funding program (e.g., the Open Operating Grant Program). The example below demonstrates how CIHR makes its funding decision in these situations. In this example, there are two committees with a total competitions budget of $800,000.
- Committee X reviewed 51 applications, and Committee Y reviewed 44. The applications in each committee are ranked based on their final ratings (which are an average of the individual members' ratings), and converted to a "percent rank" based on their rank order as a function of the total number of applications in the committee:
Committee X Committee Y # Rating % Rank # Rating % Rank 1 4.37 1.96 1 4.68 2.27 2 4.16 3.92 2 4.59 4.55 3 3.94 5.88 3 4.44 6.82 4 3.92 7.84 4 4.39 9.09 5 3.86 9.80 5 4.25 11.36 6 3.76 11.76 6 4.04 13.63 7 3.74 13.73 7 3.99 15.91 8 3.45 15.69 8 3.87 18.18 … … … … … … 51 1.90 100.00 44 2.25 100.00 - The committees are combined to create a single rank-order list, based on the percent rank of each application in its committee, and the cumulative cost of the applications is calculated:
# % Rank Committee Rating Cumulative Cost 1 1.96 X 4.37 $110,000 2 2.27 Y 4.68 $207,696 3 3.92 X 4.16 $264,750 4 4.55 Y 4.59 $421,124 5 5.88 X 3.94 $546,544 6 6.82 Y 4.44 $695,120 7 7.84 X 3.92 $788,100 8 9.09 Y 4.39 $862,521 9 9.80 X 3.86 $973,406 10 11.36 Y 4.25 $1,149,306 … … … … … 94 100.00 X 1.90 … 95 100.00 Y 2.25 … - Applications are listed top-down by percent rank to calculate the cumulative cost. The percent rank cutoff is then set, based on what can be afforded within the competition. This cutoff is subsequently applied to each committee and grants ranked above the cutoff will be funded. For this example, if $800,000 were available, the first 4 applications in Committee X and the first 3 applications in Committee Y would be funded. This results in 4/51 (7.84%) of applications from Committee X and 3/44 (6.82%) of applications from Committee Y will be funded.