Reforms of open programs and peer review – Questions and answers – Peer Review Process

Questions

  1. How will the new design reduce peer review burden?
  2. How will CIHR be matching applications to expert reviewers?
  3. What types of experts will be part of the face-to-face interdisciplinary committee(s) and how will they be selected?
  4. Will applicants have access to reviewer comments?
  5. What role will knowledge users play in application reviews?
  6. Can applicants nominate at least one expert reviewer to review their grant?
  7. Will reviewers still write reviews or just tick boxes?
  8. How will the ranking of individual reviews be tabulated if the total number of applications assigned to a reviewer varies?

1. How will the new design reduce peer review burden?

In designing the new Open Suite of Programs, CIHR aims to implement design elements that will reduce the overall time a reviewer spends reviewing, discussing, and providing feedback on an application. To achieve this, CIHR is implementing a number of design elements:

  1. A multi-stage competition process that, for the Foundation Scheme, involves a two-stage screening process prior to face-to-face review. Stage 1 applications are short, and designed to be reviewed in a relatively short period of time. Peer reviewer burden is also actively managed by controlling the number of applicants invited to participate in Stage 2, where a more detailed application is required.
  2. Structured review criteria provides peer reviewers with clearly defined review criteria and relevant application information in order to evaluate success. This will help reduce the amount of time reviewers spend analyzing an application package, and providing free form feedback to applicants.
  3. Conducting screening reviews and conversations in a virtual space (internet-assisted discussions) will help reduce reviewer burden by alleviating the need for reviewers to travel to Ottawa, freeing time to conduct research or engage in other activities. This model will also facilitate international expert review by supporting cost-effective access to international research leaders.

As part of a multi-phase competition process, we want to be thoughtful about how and when to use face-to-face committee meetings. We know that a lot of committee time is spent discussing applications that everyone agrees should be funded or that everyone agrees have fatal flaws. There is, however, always a "grey zone" – where reviewer views are varied for a number of different reasons. The intent is to leverage the strengths of structured review and available technology to encourage early recognition of outstanding applications and triage of non-competitive applications. The intent is to concentrate face-to-face discussions in fewer, broader, more multi-disciplinary committees on these "grey zone" applications.

Each design element highlighted has the potential to reduce the burden felt by applicants and peer reviewers.

2. How will CIHR be matching applications to expert reviewers?

Matching applications to appropriate reviewers is essential to implementing application-focused review. We are currently implementing various IT-based tools to assist with the matching processes. These tools are intended to support human judgment and experience in validating matching assignments. External experts will help support this process. Pilots are being designed to ensure these IT-based tools are effective and are appropriately integrated with expert guidance.

3. What types of experts will be part of the face-to-face interdisciplinary committee(s) and how will they be selected?

The interdisciplinary committee(s) will consist of recognized leaders with a broad understanding of CIHR and the Canadian and international health research landscapes. CIHR envisions there will be one committee for the Foundation Scheme, and several committees for the Project Scheme. Project committees will be formed based on the different types of applications received in each competition.

4. Will applicants have access to reviewer comments?

All applicants will receive structured feedback from each assigned reviewer at each stage of the multi-stage competition process.

5. What role will knowledge users play in application reviews?

CIHR will include knowledge users in the review of applications to the new open funding schemes, where appropriate. Knowledge users will be an important expertise pool in the College of Reviewers and have been given a high priority in the development of the College of Reviewers recruitment strategy. A strong knowledge users base will be critical to ensure that CIHR assigns the right mix of expertise in the review of CIHR applications; especially as CIHR expects to attract a diversity of programs and projects at any stage: discovery, demonstration, validation and/or application of health-related knowledge.

6. Can applicants nominate at least one expert reviewer to review their grant?

Yes. CIHR will continue to offer applicants the opportunity to suggest reviewers (domestic or international) in their applications. These suggested reviewers are considered in the reviewer identification and assignment process. In the future, we recognize that the reviewers suggested by applicants may or may not be registered College members. In the latter case, applicants may use the College of reviewer nomination process to put forward their expert reviewer nomination. It is CIHR's intent to have a process in place to capture and fast track nominations made through the application process so that staff may have timely access to the appropriate reviewers.

7. Will reviewers still write reviews or just tick boxes?

Reviewers will be required to provide written comments, in addition to ratings, for each sub-criterion for each assigned application. Writing a formal, fully-articulated long review will not be required, although partaking in this exercise may be helpful to some as preparation for submitting formal comments to CIHR. In the spirit of reducing peer reviewer burden, reviewers will only be asked to provide their expert perspective on the key strengths and weaknesses.

8. How will the ranking of individual reviews be tabulated if the total number of applications assigned to a reviewer varies?

The new adjudication process is founded on relative quality as opposed to absolute scores. The relative ranking is applied at the individual reviewer level on their own pool of assigned applications. This approach is intended to address differences in reviewer behavior across disciplines, and provide greater consistency in peer review. Rankings from all assigned reviewers for each application will be consolidated into a final competition ranking. At the outcome of the competition, the top-ranked applications across a broad base of health-related research will be selected for funding.

Date modified: