Project Grant Program: Results

Before the committee meeting

Each application is assigned to reviewers (number of assigned reviewers may vary based on committee requirements) who provide the following:

• a summary of the application;
• a preliminary rating;
• a justification of the rating by stating the strengths and weaknesses of the application based on the evaluation criteria;
• top (competitive) or bottom (non-competitive) group selection;
• comments on the integration of sex and/or gender in the application.

Reviewers provide these preliminary ratings and comments prior to the committee meeting. Please see section 4.2.1: Conducting reviews in the Project Grant peer review manual for more information.

During the committee meeting

The preliminary ratings from all assigned reviewers are used by the committee members to discuss the application. Following the committee discussion, the assigned reviewers are asked to reach a consensus score. Each committee member is then asked to vote +/- 0.5 of the consensus score, and these votes are averaged to calculate the application’s final score. Furthermore, members calibrate scores based on all the applications being reviewed by their committee. It is possible – and normal – for the preliminary ratings provided by the assigned reviewers in the reviews, to be different from the final scores.

Please see section 4.3: The Peer Review Committee Meeting in the Project Grant peer review manual for more information.

CIHR asks committees to focus their discussion on the top 40% of applications based on preliminary ratings. The top (competitive) / bottom (non-competitive) group selection is also used to inform which applications will be discussed at the committee meeting. The remaining applications will be placed on a streamline candidate list and will only be removed from the list of those being discussed at the committee meeting (i.e., removed from committee discussions) if there is no objection from any committee member regarding the fact that they will not be discussed. Please note that, ultimately, the committees can discuss any application.

Please see section 4.3.1: Streamlining of Applications in the Project Grant peer review manual for more information.

The Notice of Recommendation is uploaded to ResearchNet before funding decisions are finalized and contains key information about the application’s standing within the competition. It includes the final rating (which may differ from the preliminary ratings provided by assigned reviewers in the individual reviews), percent rank within the committee, and the recommended average annual budget (before the across-the-board cut is applied). The NOR is provided in advance of the Notice of Decision (NOD) to offer applicants approximately two extra weeks to decide whether to resubmit an unsuccessful application to the following competition.

As a rule of thumb, applications ranked within the top 15 per cent of applications reviewed by their committee have a greater likelihood of being funded. For example, in a committee where 40 applications are reviewed, the top 6 ranked applications would have a greater likelihood of getting funded than those applications that are lower ranked. However, every committee is different, and there are a number of factors that need to be taken into consideration when looking at your percent ranking: the number of applications in the committee, possible score ties within a committee, whether your application is subject to the equalization exercise (for early career researchers, female applicants and applicants submitting applications written in French), or if it is a request for a large grant. Furthermore, the range for applications that are focused on Indigenous Health Research (IHR) and go through the iterative review process will be different.

Using the NOR to infer if the application will be successful or not is a reasonable approach to determine if the applicant should register for the next competition. However, it is possible that applications will not receive the decision applicants are expecting because of the factors mentioned above.

The Notice of Decision, including the final recommended budget in the case of funded grants, will be uploaded to ResearchNet at the end of the peer review process, following approval by the President of CIHR. The NOD contains the same information as the NOR but includes final funding decisions taking into account equalization measures.

As noted above, the preliminary ratings in the reviews may differ from the final rating on the NOD. The NOD document represents the final funding decisions for the application.

If you have any questions, please get in touch with the Contact Centre at support-soutien@cihr-irsc.gc.ca.

Any given Project Grant competition involves approximately 2300 applications reviewed across approximately 58 peer review committees. In order to make funding decisions, CIHR needs to be able to compare peer review results across committees. To do this, CIHR calculates each application’s percent rank.

The percent rank of an application is calculated using its standing within its assigned committee. For example, an application that was ranked 5/57 (based on its score) within its committee has been ranked higher than all but 4 of the other 56 applications in the committee and therefore has a percent rank of 92.9% (percent rank = 1-(4/56)). Percent rank is used in this exercise because it stretches the scale so that the first-ranked application in every committee receives a percent rank of 100%.

For the Project Grant program, funding decisions across committees are made in a similar fashion to how they were calculated under the former Open Operating Grants Program (OOGP). Scores for each application were converted to within-committee rankings, which were then used to calculate each application’s percent rank. This allows CIHR to account for scoring differences across the committees, and it also allows us to fund an approximately proportional number of applications across each committee. The number is approximate due to a number of factors that are explained in detail below; however, it is important to note that this is not dictated by pillar or area of research. Read more about the relationship between application rank within a panel, the percent rank score, and funding cutoffs.

The number of applications per committee generally ranges between 20 and 80, which accounts for some of the variability (because of the denominator’s impact on the percent rank calculation) of success rate by committee. The committee success rates are close to the overall success rate but fluctuate around it. It is not possible to create committees of comparable size without forcing applications into inappropriate panels (i.e., that are not the best expertise match to review the application).

In addition to the differences in application pressure, a number of factors may cause individual committee success rates to fluctuate.

Large grants

Within the overall competition budget, there is a specific funding envelope for large grants (i.e., those within the top 2% of the total grant amounts requested). CIHR implemented this policy in the OOGP in 2010-11 to strike a balance between investing in highly ranked applications with large budgets and the imperative to maximize the total number of grants funded. The threshold for being considered a large grant fluctuates from one competition to the next but in past competitions, it has been any application that has requested more than $2M to $2.3M. Large grants are reviewed in their assigned committees; however, they are combined and treated as a separate cohort for the purpose of making funding decisions. Using this methodology means that it is possible for a large grant to be highly ranked in its committee (i.e., the ranking is above the committee cutoff) but ultimately not funded (i.e., the large grant budget gets exhausted before all the large grants above their respective committee cutoffs are funded). In this scenario, such an application would receive a bridge grant—but since bridge grants are not included in the calculation of success rates, this would reduce the committee’s success rate overall.

Ties

If applications are tied (e.g., two applications have the same final score within a committee and are therefore both equally ranked), it is CIHR’s policy to fund both or neither. Occasionally, a tie will fall right at the committee cutoff. In such a scenario, and if there are sufficient funds, both applications are funded. This would increase an individual committee's success rate, as the committee gets to fund one grant more than it otherwise would have been able to support.

Equalization of early career researchers

CIHR monitors the competition results to ensure that the proportion of ECRs funded is at least equal to the proportion of ECR applicants to the competition^{Footnote 1}. A separate pot of money, provided to CIHR through Budget 2016, is reserved for this equalization and if it is not fully needed, it is reinvested within the Project Grant competition.

To complete the equalization process, CIHR creates the funding scenario as described above using the competition budget available. We then assess if the proportion of ECR grants funded is at least equal to the proportion of applications submitted by ECRs . If it is not, additional ECR applicants are funded based on their percent rank. ECRs funded through this process are combined and treated as a separate cohort for the purpose of making funding decisions. This means that the intervention is completed at the competition level and not the committee level to ensure that the top applicants are selected.

Equalization of female Nominated Principal Investigators

This process is being implemented for the first time in the Spring 2021 Project Grant competition. Like the ECR equalization process, CIHR will ensure that the proportion of female NPIs funded is at least equal to the proportion of female NPI applicants to the competition.

This process and its impacts are the same as with the ECR equalization described above.

Equalization of French language applications

This process was being implemented for the first time in the Spring 2021 Project Grant competition. Like the ECR equalization process, CIHR will ensure that the proportion of French-language applications funded is at least equal to the proportion of French-language applications submitted to the competition.

This process and its impacts are the same as with the ECR equalization described above.

CIHR aims to strike the right balance between full grants and bridge grants for the Project Grant program, but the numbers may vary by competition because they are not predetermined.

In the past, we have received questions about the rationale for offering approximately 58 bridge grants (i.e., the same number of bridge grants as committees) instead of supporting one extra applicant per committee. It is important to note that, even if there were funds to support funding for this number of bridge grants, they would not be distributed across the committees equally due to differences in committee size.

Applications that are adjudicated by the Indigenous health research (IHR) committee are done as part of the iterative review process. Further, CIHR has committed 4.6% of the Project Grant budget to support IHR applications as part of its commitment to invest 4.6% of its total budget in Indigenous health research.

CIHR Institutes, Initiatives and Partners may leverage the Project Grant program through Priority Announcements (PAs). PAs are additional sources of funding for highly ranked applications aligned to specific research areas. PAs are primarily funded by Institutes and do not affect CIHR’s investment in investigator-initiated research. PAs are usually CIHR Institute priorities that use the Project Grant peer review process to identify and fund areas of research within the Institutes’ mandates. These will vary from one competition to the next; however, all follow the same funding-decision process.

The process for selecting applications for funding involves four steps:

1. Once the preliminary results of the Project Grant competition are established, a list of applications eligible for a PA that are below the Project Grant cut-off but above 3.5 is produced.
2. The eligible applications are assessed for relevance to the objectives of the Priority Announcement.
3. All applications that are deemed relevant are then ranked as a cohort.
4. Applications are funded in rank order as far as budgets will allow.