Update on the Fall 2017 Project Grant competition

July 10, 2017

Dear colleagues,

Since CIHR announced the delay to the Fall 2017 Project Grant competition in the May 5, 2017 email to the community, we have had in-depth discussions on the upcoming Project Competition (registration deadline August 15, 2017 and application deadline September 15, 2017) with the CIHR Scientific Directors, the Chairs of the College of Reviewers ("College Chairs"), and the University Delegates. We also sought quick feedback from selected community leaders.

The Scientific Directors, Anne Martin-Matthews (Acting VP Research) and I all agreed that we must return to face-to-face grants panels for the evaluation of the project grants for this fall competition. I therefore asked the College Chairs, composed of 17 research leaders and led by Paul Kubes (University of Calgary) to become the major advisory body to CIHR on Grants Programs and Peer Review, in addition to their role in advising CIHR on constructing the College of Reviewers.

A 1.5 day meeting of the College Chairs was held last week, and was also attended by nine competition chairs from the previous Project Grant competition as well as two members of the University Delegates Executive Committee. This meeting led to very thoughtful suggestions on how to improve the upcoming Project Grant competition, as well as subsequent competitions, and those suggestions constitute the major part of the changes described below.

In addition, while on a "Listening Tour" to 10 universities across Canada over the past several weeks, I also heard many excellent comments about peer review (and much else!), many of which are also incorporated in the changes below. I want to thank all of you who participated in these discussions for your input and feedback.

The changes below are only a first step in improving the Project Grant competition. We had only about six weeks to consider and make these changes, but I am confident that they will substantially improve the competition. However, this system must continue to improve for the Spring 2018 competition. In particular, we must determine the best approach to the review of multidisciplinary applications that do not fit well with any of the historical panels, an issue that CIHR will work with the community to address for the Spring 2018 competition.

For the Fall 2017 Project Grant competition, the following changes have been made:

  1. The grant applications will be reviewed by face-to-face panels, with no online or other prior evaluations
    • Altogether, the panels will cover the full spectrum of health research. As with the historical CIHR grants panels, each will have a name, mandate, and a face-to-face meeting.
    • Fifty-three panels will be struck. Depending on the scientific expertise required, these panels may be larger in size than they have been historically, to ensure that the reviewer expertise is sufficient to cover the breadth of applications received.
    • The panel mandates from the Spring 2015 Open Operating Grant Program (OOGP) competition will serve as the starting point for the development of the Fall 2017 panels.           
    • Substantial changes to the OOGP-based panel mandates were not possible in the time CIHR has had to deliver this Fall 2017 competition, but we will seek input from the College Chairs, University Delegates, Panel Chairs, Scientific Officers, reviewers, and applicants to evaluate and adjust the roster of panels for the Spring 2018 competition. There is a clear desire from the community, and a firm commitment from CIHR, to update the panels to best address the breadth of CIHR's mandate, the increasingly multidisciplinary nature of research, and the challenge of reviewing highly innovative grants. The evolution of the panel mandates will be a priority for CIHR for the Spring 2018 competition and the process to provide input will be shared with the community in the coming weeks.
    • The iterative review process that has been used for applications related to Indigenous Health Research will be retained.
  2. Applicants will be asked to choose the panel most relevant to their research
    • Each applicant must identify the panels (1st and 2nd choices) that they think best fits their application. This choice must be made at the registration stage. Please read the full mandates of each panel because the mandates have changed slightly from the OOGP.
    • In addition to indicating their choices, applicants must write a strong, clear research Summary of the Research Proposal. A good summary is essential to guiding the decision of the Chairs and Scientific Officers as to whether a specific panel has the required expertise to fairly and knowledgeably review an application. If an application does not fit, then the Summary will help the Chairs and Scientific Officers identify which panel is best suited to review it, or to decide whether their panel should be expanded to provide the necessary expertise.
    • Changes to the Summary of the Research Proposal will be allowed at the application stage, but the overall thrust of the application must remain the same so that reassignment of an application to a different panel will not be required.
  3. Reviewers will be recruited for a single panel
    • Each panel will include an experienced Chair and two Scientific Officers, who will help CIHR staff identify reviewers with the necessary expertise.
    • The Chairs and Scientific Officers will participate in the selection of reviewers as well as the assignment of applications to the panel members.
    • Once the panel has been recruited and confirmed, each member will be asked to complete the Conflict and Ability to Review exercise for every application in their panel. The results of this exercise—which will use the High/Med/Low/Not Enough Expertise assessment—will allow Chairs, Scientific Officers, and CIHR staff to generate the final peer review assignments.
    • Each reviewer will receive an average of 10 applications to review, and each application will have three reviewers assigned to it, a primary and two secondary reviewers, each of whom will write a review.
    • Reviewers will score the applications using the 0-4.9 scale.
    • If you receive an invitation to review for the fall competition, please accept it. The quality of peer review depends entirely on the community's participation in the process. The invitations to review will begin going out the week of July 10th, 2017.
  4. The Project Grant application form has changed only slightly
    • The full request for applications, registration instructions, and application instructions will be available on July 11, 2017 in the CIHR Funding Opportunities database.
    • The research proposal can be a maximum of 10 pages (including figures and tables). The structure of the proposal will be entirely at the discretion of the applicant. Applicants can therefore write this section of the application in the way that they deem most appropriate to address the adjudication criteria.
    • The adjudication criteria and weighting will be as follows:
      • Criterion 1. Concept (25%)
        • Significance and Impact of the Research (25%)
      • Criterion 2. Feasibility (75%)
        • Approaches and Methods (50%)
        • Expertise, Experience, and Resources (25%)
    • Applicants are now invited to include unlimited attachments, but reviewers are not required to read those attachments.
    • Applicants now also have the option of including a response to previous reviews (maximum two pages).
  5. All health-related applications, including multidisciplinary applications, are welcome
    • The Chairs and Scientific Officers will work with CIHR to ensure that the appropriate expertise is available to evaluate a multidisciplinary application in any given panel.

The Future Evolution of Peer Review at CIHR

The changes outlined above are only a first step and CIHR is committed to continue working iteratively with the community to ensure that the peer review system continues to evolve. The College Chairs have made other important recommendations to CIHR on Project Grants, but unfortunately it is not possible to introduce these additional changes into the current competition due to operational limitations (e.g. changes to the ResearchNet system would not be possible in time to deliver the competition). These recommendations will be given strong consideration by the Scientific Directors, Anne Martin-Matthews, and myself for implementation in subsequent Project Grant competitions:

  • In addition to the Summary of Research Proposal to be submitted at the registration stage, require that applicants provide a justification for their panel selection and aim, to the extent possible, to allow applicants to remain in the panel of choice. If this is not possible, notify the applicant of the decision to transfer their application to a different panel.
  • Implement a recurring, systematic and evidence-informed process to update the composition of panels and their mandates to ensure that they cover the breadth of CIHR's mandate.
  • Consider how best to review multidisciplinary applications and applications that do not fit easily into the panel structure.
  • Implement the 0-100 point scale for scoring applications.
  • When an applicant responds to previous reviews, the previous reviews should also be included.
  • Panels should aim to focus their discussions on the top 40% of applications, based on initial reviewer scores and as determined by the panel Chairs and SOs.
  • The number of attachments that a reviewer should be required to review should be limited (e.g., 5 pages).
  • Individuals who receive CIHR funding should be required to participate in the peer review system.

Changes to the Foundation Grant program in the future

Finally, please note that the above comments pertain only to the Project Grant competition and not the Foundation Grant program. Discussion with the Scientific Directors, College Chairs, University Delegates and the research community clearly indicates that we must also make changes to the Foundation Grant program in the future, as also suggested by the International Peer Review Expert Panel report. Before any changes are made, however, a thoughtful evaluation of this program must first be undertaken by the CIHR Scientific Directors, College Chairs, University Delegates and selected community leaders; such an analysis will take time. For the upcoming Foundation Grant competition, we may only be able to make small changes, if any. Thus, the intake for the 2017-18 Foundation Grant competition (registration deadline August 8, 2017 and stage 1 application deadline September 12, 2017) will remain open. We will provide more information about the Foundation program the week of July 17.

Thank you again for your participation in improving peer review at CIHR.

Sincerely,
Roderick R. McInnes, C.M., O.Ont., MD, PhD, FRSC
Acting President, Canadian Institutes of Health Research

Date modified: