Designing for the Future: The New Open Suite of Programs and Peer Review Process - Town hall presentation

Winter-Spring 2013

Table of Contents


Objectives

The objectives of this presentation are to:

  • Review why changes to the Open Suite of Programs are necessary;
  • Provide an overview of the new design of the Open Suite of Programs and Peer Review process;
  • Outline the implementation plan and draft timelines; and
  • Answer questions from the community.

The Rationale

Objectives of the reform

The objectives of the reform of CIHR's investigator-driven strategy are to:

  • Capture excellence across all four research pillars, from knowledge creation to knowledge translation
  • Capture innovative, original and breakthrough research
  • Integrate new talent to sustain Canada's pipeline of health researchers
  • Improve sustainability of the long-term research enterprise

In meeting these objectives, the reform is also meant to address a number of current operational challenges:

  • Workload and costs for applicants
  • Peer review burden
  • Lack of consistency and efficiency of peer review process
  • Growing discrepancy between research evolution and committee structure
  • Program complexity

Workload and costs for applicants

  • Writing grant applications can be extremely costly to researchers and institutions. A study conducted in Australia (NHMRC) last year estimated the total cost of applications to be over $17,000 (Australian dollars) per application submitted.
  • On average, a CIHR OOGP application takes some 169 hours to complete which is comparable to the NHMRC benchmark.
  • The estimated applicant cost to prepare an OOGP application is $10,878. With the costs of administration and peer review included the total cost is approximately $14,000.

Peer review burden

  • Currently, CIHR populates 53 standing peer review committees with over 2,300 reviewers. These committees represent a number of research areas, disciplines, living systems, populations, etc.
  • The number of committees keeps on growing over time and requests continue to be made to create new standing committees to review research that is seen as not easily reviewed in the current committee structure.
  • It is becoming increasingly difficult to recruit appropriate reviewers as the correct expertise often resides with members in conflict, currently applying for funds, or sitting on a different CIHR peer review committee.

Lack of consistency and efficiency of peer review process

Results from the 2010 Ipsos Reid survey indicate a high degree of dissatisfaction amongst Stakeholders in certain areas of peer review.

Stakeholder Satisfaction – Peer Review (Percent of respondents who provided an opinion)

Figure 1 long description

Growing discrepancy between research evolution and committee structure

  • There is a high degree of effort currently going into trying to ensure that each application is reviewed by the most appropriate committee.
  • Inflow and outflow of applications to standing committees for the last 4 OOGP competitions is illustrated below:

CIHR OOGP - Inflow and Outflow (Spring 2010 - Fall 2011)

An example...

Overview of The Design

  • Feedback from the research community on the proposed changes to the Open Suite of Programs and peer review process was collected from February 8, 2012 to May 1, 2012:

  • Work was completed through the summer and the fall to assess all of the feedback and enhance the overall design.

  • The updated design was released in December of 2012.

Key elements of the design have been endorsed by CIHR's Governing Council and Science Council:

  1. Two separate, complimentary funding schemes
    • Project Scheme
    • Foundation Scheme
  2. A peer review process that includes:
    • Application-focused review
    • Multi-stage review
    • Structured review criteria
    • Remote review of applications at the initial stage(s)
  3. A College of Reviewers that will support excellent peer review across the spectrum of health research

Investigator-driven operating grants budgetary envelope (OOGP)


Figure 5 long description

2011-12:

  • # new grants: 802
  • # applications: 4,578
  • Average multi-year grant size: ~$600k
  • Success rate: 17.5%

2007-08:

  • # new grants: 816
  • # applications: 3,625
  • Average multi-year grant size:~$540k
  • Success rate: 22.6%

CIHR is committed to maintaining approximately the same number of Nominated Principal Investigators that are currently supported in the system today.

  • A number of scenarios were considered to determine a sustainable intake of grantees over the longer term.
  • The current planning scenario is:

    Steady State Scenario for the new Open Funding Schemes
    Foundation Project Total
    Proportion of CIHR's Open Funding Grants Budget 45% 55% 100%
    Annual intake
    (number of grants)
    114 939 1,053
    Steady-state
    (number of grants)
    750 2,200 2,950
  • The expectation is that grant values in both schemes will vary and be commensurate with the needs of each individual program of research or project.

  • Modeling predicts a distribution of foundation and project grant values as follows:

The Foundation Scheme

The Foundation Scheme is designed to contribute to a sustainable foundation of health research leaders. It is expected to:

  • Support a broad base of research leaders across career stages, areas and disciplines relevant to health;
  • Develop and maintain Canadian capacity;
  • Provide flexibility to pursue new, innovative lines of inquiry;
  • Contribute to the creation and use of health-related knowledge.

The Foundation Scheme will have one competition a year.

The specific review criteria are still under development. The requirements for enhanced institutional support are also still evolving.

The Project Scheme

The Project Scheme is designed to capture ideas with the greatest potential for important advances It is expected to:

  • Support a diverse portfolio of health-related research and knowledge translation projects at any stage, from discovery to application, including commercialization;
  • Promote relevant collaborations across disciplines, professions and sectors;
  • Contribute to the creation and use of health-related knowledge.

The Project Scheme will have two competitions per year.

As a result of community feedback, the Project Scheme will use a two-stage process for review.

The specific review criteria are still under development.

College of Reviewers

  • To deliver on the vision and objectives, CIHR will work with funding partners to build a College of Reviewers
  • It will be a centrally-managed, national resource.
  • The College will be made up of faculties of experts from both within and outside Canada.
  • Specific roles will be defined to support the peer review processes (e.g. moderators and chairs) and to support the peer review system as a whole (e.g. faculty chair).
  • Specific programs will be developed to train and mentor reviewers, and to monitor performance.
  • The recruitment of reviewers to the College will be a staged process
    • The transition period for the College of Reviewers is expected to occur from spring 2013 to 2016.
    • The target is to recruit approximately 8,000 members to the College.
    • CIHR currently has a base of 2,300 active reviewers supporting the OOGP and has accessed approximately 5,800 individuals to participate in peer review across all programs over the last 5 years.
    • The first waves will focus on the recruitment of active and recently active reviewers.
    • Additional waves will allow for the targeted recruitment of reviewers to address gaps in expertise, and to increase the number of international reviewers.

The Project Scheme will require between 500-675 reviewers per competition and the Foundation scheme will require between 275-350 per competition (assumes each application is reviewed by 5 reviewers and each reviewer receives between 15–20 applications).

Rationale for individual design elements

Design Element - Multi-stage competition process

  • Effective screening of applications
  • Decrease applicant burden and reviewer burden
  • Focus reviewer attention on specific criteria for each stage of review

Design Element - Application-Focused Review

  • Avoid "force fitting" applications into standing committee structure
  • Assign appropriate expertise to each application

Design Element - Structured Review Criteria

  • Minimize inconsistent/inappropriate application of review criteria
  • Improve transparency of review process
  • Decrease peer review burden

Design Element - Remote (virtual) Screening/Review

  • Facilitate access to expertise, including international
  • Improve cost-effectiveness of the process
  • Minimize group dynamics and committee culture biases

Transitioning to the New Schemes

The transition to the new Open Suite of Programs and peer review processes, will occur over a number of years.

The transition strategy includes three phases:

  1. Piloting key peer review design elements
  2. Gradually phasing-in the new funding schemes
  3. Gradually phasing-out the existing Open funding program

Given the scope of the proposed changes, and the intention to learn from the results of the pilots, course corrections and adjustments to timelines may be required.

1. Piloting key peer review design elements

  • CIHR will conduct various pilot studies and tests for the new peer review process between early 2013 and mid-2016. Pilots will include:

    Design Elements being Piloted Existing CIHR Program
    Some elements of remote review, multi-phased competition process and application binning (completed) New Investigator Program
    Remote review, asynchronous discussion, structured review (two pilots – spring and fall of 2013) Fellowship Program
    First Project Scheme Pilot: Short structured application, new criteria, structured review, remote review, five reviewers per application (fall 2014). KT Synthesis Program
    Second Project Scheme Pilot: all elements of Project Scheme through a strategic program Strategic Program (TBD)
    Matching Pilot: Will be conducted in the background of the next few OOGP competitions and will pilot methods for matching applications to reviewers OOGP

2. Gradually phasing-in the new funding schemes

  • The phase-in of the new Open funding schemes will occur over the course of several competition cycles.
  • The Foundation Scheme will be launched through two "live pilot" competitions with application deadlines scheduled for fall 2014 and fall 2015.
  • The first regular Foundation competition application deadline is scheduled for fall 2016.
  • The first Project competition application deadline is scheduled for spring 2016.
  • We are committed to ensuring that adequate support is available to applicants, reviewers, and academic institutions throughout the transition process to help the research community navigate through this transition period with minimal disruption.

3. Gradually phasing-out the existing Open funding program

  • CIHR recognizes the importance of carefully managing the implications for grantees who are currently supported by CIHR's existing Open Suite of Programs.
  • CIHR will hold three more Open Operating Grant Program competitions (spring 2013, fall 2013, and spring 2014).
  • There will also be a transitional Open Operating Grant Program competition in spring 2015, which will be held in parallel to the first "live pilot" of the Foundation Scheme.
  • Other existing open programs will be phased-out after the launch of the first Project Scheme competition.