Designing for the Future: The New Open Suite of Programs and Peer Review Process – Questions and Answers

1. If the changes to the Open Suite of Programs and peer review process are successful, what will CIHR have achieved?

Our goal in designing the new Open Suite of Programs and peer review processes is to develop a high quality, flexible and sustainable system capable of identifying and supporting excellence in research and knowledge translation in all areas of health.

The design of the new Open Suite of Programs is intended to:

• Reduce funding program complexity
• Improve access to funding for the diverse health-related applications CIHR receives
• Reduce applicant workload
• Streamline application processes to capture the correct information
• Provide support for new/early career investigators
• Support knowledge user collaborations, where appropriate
• Ensure an appropriate base of qualified experts to uphold an excellent peer review system
• Improve the quality, fairness and transparency of peer review
• Reduce conservatism in peer review
• Reduce peer reviewer workload

Through the re-design of the new Open Suite of Programs, CIHR and its research community will have created a future-oriented system that is poised to accelerate the advancement of health knowledge and improvements to health research, the health care system, and health outcomes.

2. Why is CIHR changing the Open Suite of Programs and supporting peer review processes?

As the major federal funder of health research in the country, CIHR has a role in ensuring the long-term sustainability of the health research enterprise. This includes having an Open Suite of Programs capable of developing and supporting a well-trained base of investigators with the skills and expertise needed to design and conduct innovative and diverse research and knowledge translation activities aimed at improving health. It also includes an expert peer review system that is well-managed, fair, and transparent in the selection and support of the most innovative and cutting-edge health-related proposals.

Over the past several years, we have become increasingly aware of the need to modernize existing frameworks and systems to better capitalize on Canada's health research strengths, and to better capture the evolution of the health research landscape. We have also heard that there are a number of challenges with the current funding framework and peer review process that need to be addressed. CIHR anticipates the new design will improve the efficiency and effectiveness of CIHR's funding and peer review processes by:

• Reducing funding program complexity
• Improving access to funding for the diverse health-related applications CIHR receives
• Reducing applicant workload
• Streamlining application processes to capture the correct information
• Providing support for new/early career investigators
• Supporting knowledge user collaborations, where appropriate
• Ensuring an appropriate base of qualified experts to uphold an excellent peer review system
• Improving the quality, fairness and transparency of peer review.
• Reducing conservatism in peer review
• Reducing peer reviewer workload

3. Is the design process finished?

Although the document outlines the key elements of the design of the new Open Suite of Programs and peer review processes, it is not meant to signal the end of the design process. There are still details to be confirmed before we implement the design, and the transition plan identifies a number of important pilots.

CIHR remains committed to keeping the community involved and informed as the next level of design detail is developed and implementation plans progress for the new Open Suite of Programs and peer review processes.

4. What evidence was considered to inform the design?

In developing this new design for the Open Suite of Programs, CIHR took into consideration the literature on research funding and peer review; examined national and international research funding programs; and engaged in thoughtful reflection on what new design elements would best work for health researcher and stakeholder communities and for CIHR. We have listened closely to our stakeholders, and based on our own analyses and the existing evidence, have identified a number of changes that could transform the current Open Suite of Programs to better deliver on the full spectrum of CIHR's mandate.

In reviewing what evidence is available, CIHR found that there is not a large base to draw upon in this area. Through designing and implementing the new Open Suite of Programs, there is an opportunity to contribute to the evidence base for funding program and peer review design. CIHR is in the process of developing a Research Plan to assess the outcomes of the new peer review process, gain a greater understanding of the complexities of peer review and contribute to the body of literature on peer review (Annex D, Designing for the Future: The New Open Suite of Programs and Peer Review Process).

Annex 1 of the Design Discussion Document lists the sources that CIHR used to inform the design of the new Open Suite of Programs and Peer Review Process.

5. Will knowledge users not registered at an eligible institution be allowed to apply to the Project Scheme?

Knowledge users are eligible to apply to the Project Scheme as a Project Leader. In cases where knowledge users are not registered at an eligible institution, a parallel application for institution eligibility will be required during the competition process. The current process is being reviewed to better accommodate these situations.

6. Can a single Project Leader hold multiple Project grants?

There will be no restriction on the number of Project grants that may be held by a single investigator during the transition period. This may be revised after transition.

7. Are Project grants renewable?

As per current practices, project grantees interested in building on the results of previous research and knowledge translation projects can submit a new application, emphasizing sequential lines of inquiry, to the Project Scheme competition.

CIHR is moving away from the term renewal primarily because it does not apply to all research career trajectories that span CIHR's mandate. This does not preclude researchers from applying for consecutive funding to advance a line of inquiry that is linked to a previous project.

8. Is there a cap on Project grant values?

Project grant values and durations will be commensurate with the requirements of the project proposed, and will vary depending on the field, proposed approach, and scope of activities. Applicants will be eligible to apply to the Project Scheme for grants within the range of approximately $25,000 to $750,000 per year, with grant durations ranging between approximately 1 to 5 years, reflecting the wide variety of projects presently funded in various Open programs. It is expected that the funding profile of Project grantees will broadly reflect the current Open Operating Grants Program distribution.

9. How many competitions will there be per year for the Project Scheme? And what are the application deadlines?

There will be two Project competitions per year. The competition launch dates, application deadline dates, and funding release dates for the Project Scheme are as follows:

10. Is the Foundation Scheme only for the super-elite?

No, the Foundation Scheme is designed to support health research leaders at all career stages, and from all health-related fields.  CIHR will award Foundation grants with varying durations and values to support diverse health-related programs of research.

As described in the February 2012 Design Discussion Document, a separate stream will be established for new/early career investigators to capture emerging research leaders.

11. Will a Program Leader in the Foundation Scheme be eligible to apply to the Project Scheme as a Project Leader?

A Foundation Grant Program Leader (e.g., Nominated Principal Investigator or Principal Investigator from the Foundation Scheme) will not be eligible to apply to the Project Scheme as a Project Leader (e.g., Nominated Principal Investigator or Principal Investigator). However, as CIHR supports collaborative approaches to health research and knowledge translation, there is no restriction on Foundation Grant Program Leaders participating in Projects as collaborators.

12. Is there a cap on Foundation grant values?

Foundation grant values will be commensurate with the requirements of the research proposed, and will vary depending on the research field, research approach, and scope of program activities. Investigators will be able to apply to the Foundation Scheme for grants within the range of approximately $50,000 to $1.5 million per year. Established investigators will be awarded 7-year grants, and new/early career investigators will be awarded 5-year grants. It is expected that an applicant's CIHR funding history will be an important factor in determining future levels of funding.

13. If I only hold one CIHR grant today, am I eligible to apply to the Foundation Scheme?

Eligible applicants to the Foundation Scheme will include new and established independent researchers with a demonstrable track record of excellence and impact in their field of study. These researchers must be affiliated with an eligible institution.

Applicants may include researchers with or without one or more existing sources of CIHR funding (e.g. Open Operating Grant Program, Open Knowledge Translation Programs, Strategic Funding), and future Project grantees. Note that once funded, Foundation grantees can only hold one active Foundation grant at a time.

14. What does CIHR envision as the requirements for enhanced institutional support?

Enhanced Institutional Support is expected to build upon current expectations outlined in the Federal Indirect Cost Program and the Memorandum of Understanding on the Roles and Responsibilities in the Management of Federal Grants and Awards. The level and type of enhanced institutional support required will depend on the nature, scope and critical success factors of the program of research.

Discussions are underway with academic institutions on how to best define and implement the requirement for enhanced institutional support.

15. How many competitions will there be per year for the Foundation Scheme? And what are the application deadlines?

There will be one Foundation competition per year. The competition launch date, application deadline date, and funding release date for the Foundation Scheme are as follows:

16. Who qualifies for the new/early career investigator stream in the Foundation Scheme?

Eligible applicants to the Foundation Scheme's new/early career investigator stream will include applicants who, at the time of application, have assumed his/her first independent academic position (e.g., faculty appointment) no more than 5 years (60 months) ago.

Discussions are underway to determine the requirements for a self-assessment eligibility checklist.

17. How will CIHR be matching applications to expert reviewers?

Matching applications to appropriate reviewers is essential to implementing application-focused review. We are currently assessing various IT-based tools to assist with the matching processes. These tools are intended to support human judgment and experience in validating matching assignments. We envision that external experts will help support this process. A number of pilots are being designed to ensure these IT-based tools are effective and are appropriately integrated with expert guidance.

18. What was CIHR's rationale for including a remote (virtual) screening process?

CIHR is committed to implementing application-focused review, which is intended to match the most appropriate expert reviewers to individual applications in order to improve the quality and fairness of reviews. Given the number and different types of applications CIHR receives through its Open programs, application-focused review can only be logistically feasible and successful if supported by internet-assisted technology. The remote (virtual) screening process was proposed as a means to bring together the appropriate expertise to inform peer review, gain cost-effective access to a broader base of expertise (including international experts), reduce biases that occur in face-to-face discussions, and reduce the burden of travel demands imposed on peer reviewers' time.

19. What types of experts will be part of the face-to-face interdisciplinary committee(s) and how will they be selected?

The interdisciplinary committee(s) will consist of recognized leaders with a broad understanding of CIHR and the Canadian and international health research landscapes. CIHR envisions there will be one committee for the Foundation Scheme, and several committees for the Project Scheme. Project committees will be formed based on the different types of applications received in each competition.

These committees will assess "grey zone" applications from a portfolio perspective rather than an individual application perspective.

20. Will applicants have access to reviewer comments?

All applicants will receive structured feedback from each assigned reviewer at each stage of the multi-stage competition process.

21. Why are the adjudication criteria listed in Annexes A and B still in early draft form?

CIHR is currently working with a variety of stakeholders to further develop and refine details for the proposed adjudication criteria. Discussions are ongoing, and continue to shape the evolution of the proposed Project/Foundation Scheme adjudication review criteria.

22. It appears that the current OOGP committees will no longer exist in the new design. What are they being replaced with?

Given the number and diversity of applications CIHR receives, CIHR is moving towards an application-focused peer review process that will match applications to individual reviewers with the most appropriate expertise. However, CIHR values the work of its peer review committees, and recognizes the need to maintain face-to-face discussions as part of the new peer review process. It is CIHR's intention to make more judicious use of face-to-face committee meetings by using broader, multidisciplinary committees at the final assessment stage of the multi-stage competition process. Committee discussions will focus on "grey zone" applications, with particular emphasis on applications with large variances in independent reviewer rankings.

23. What are the competition timelines for the Foundation and Project Schemes?

The competition timelines for the Foundation and Project Schemes are represented in the following table:

24. Who is eligible to apply to the Foundation "live pilots"?

The phase-in of the Foundation Scheme would start with two "live pilot" competitions. "Live pilot" competitions will be regular competitions where CIHR will implement specific application intake strategies to manage application pressure.

Existing grant holders with grant term-end dates during the transition period, along with new/early career investigators and researchers who have never before held CIHR grant funding, will be invited to apply to the first "live pilot" competition.

Successful applicants from the first "live pilot" competition will be the first Foundation grantees. Applicants and reviewers will be asked to provide feedback on the competition process, which will inform adjustments to the second Foundation Scheme "live pilot." Similar intake strategies will be applied to the second "live pilot" to manage application pressure.

A transitional Open Operating Grant Program competition (spring 2015) will be held in parallel to the "live pilot" competition to allow those who are unsuccessful after Stage 1 of the Foundation Scheme competition an opportunity to apply to the Open Operating Grant Program competition.

25. Why are Open knowledge translation programs being phased out?

The new Project Scheme was designed to reduce barriers and support a diverse suite of projects, including those which have been funded through Open knowledge translation programs. We recognize the importance of sustaining the Open knowledge translation programs until the Project Scheme is fully implemented. As a result, the Open knowledge translation funding mechanisms will have a delayed phase-out planned for the fall of 2016.

In designing the new Project and Foundation Schemes, a detailed program by program assessment of all Open knowledge translation program objectives, adjudication criteria, application requirements, and eligible expenses was undertaken to ensure that projects and programs of research presently supported by these funding mechanisms will be eligible and competitive in the new schemes. A monitoring process will be put in place to ensure that the types of projects supported by the Open knowledge translation programs are supported in the new schemes.

26. How will CIHR manage grantees whose term end dates fall within the gap between the last Open Operating Grant Program competition in spring 2015 and the first Project Scheme competition in winter 2016?

With the resources required to launch and fund the new Open funding scheme competitions, CIHR cannot support two regular Open Operating Grant Program competitions during the phase-in period of the transition. As such, there will be no competition with a fall 2014 application deadline. We recognize that this may impact currently funded researchers whose Open Operating Grant term-end dates fall within that period, as well as unfunded researchers seeking support. We are in the process of analysing a number of options to mitigate this effect. Discussions are underway to determine the most feasible and appropriate option for CIHR and its research community.

CIHR is committed to giving the research community enough time to plan their applications to CIHR.

27. If a current researcher who holds more than 1 OOGP grant applies for the first Foundation grant competition and is unsuccessful, will they be allowed to keep their remaining OOGP grants?

Current thinking suggests that successful applicants to the Foundation Scheme with multiple sources of Open funding would have their existing, ongoing funds rolled-up into the budget of their new Foundation grant. This process would apply to current grant holders with multiple sources of Open funding (e.g., Open Operating Grant Program; Open Knowledge Translation funding programs), as well as future Project grantees interested in consolidating their individual Projects into a cohesive program of research.

During the transition period, applicants with more than one active Open Operating Grant whose application to the Foundation Scheme is unsuccessful will not lose their existing funding before their term-end date.

28. Given the recent issues with the Common CV system, how will CIHR ensure a smooth implementation of future system changes?

The Common CV experience has prompted CIHR to extend its transition period. A major component of the piloting stage will include the piloting, and rigorously testing, of any new or modified systems and processes.

CIHR is currently in the process of assessing various IT-solutions to support the new funding schemes and peer review processes.

29. What type of training, support and resources will CIHR offer to applicants and reviewers?

Ensuring that the research community is able to navigate through this transition period with minimal disruption is a priority for us all. To achieve this, CIHR is developing an orientation program that will:

• Provide orientation to all applicants, reviewers and research administrators about the new schemes (objectives, application requirements, review criteria, etc.);
• Provide guidance on roles and responsibilities for College members and institutions;
• Support the mentoring of new/early career reviewers.

Orientation and development programs will be supported by short, self-directed on-line learning modules, as well as by instructor-led webinars. We are also discussing approaches that will involve institutional administrative staff who currently support researchers through the application process.

As part of our orientation activities, CIHR will provide the research community with information, systems, and tools that will enable them to experience program requirements and competition processes on a trial basis. These features will also be made available to institutions who wish to provide specific training to their students and researchers on how to use CIHR systems.

30. How will Foundation grantees transition out of the Foundation program?

At Year 6 of the Foundation grant, grantees will have an opportunity to signal whether they wish to continue their program of research by applying to the next Foundation Scheme competition. If no application for continued support is submitted, then end-of-grant reporting will be expected within 18 months of the term-end date of the Foundation grant.

31. Does CIHR have a contingency plan if the results of a pilot or a new measure indicate that the expected outcome was not achieved?

We are committed to continuous assessment of the outcomes of the new peer review process. We will conduct research projects that will involve rigorous analytics to gain a greater understanding of the complexities of peer review, and of how best to measure the ongoing performance of the new Open Suite of Programs. It is our intent to share our findings with the research community.

These pilots will be conducted using competitions of smaller size so that we can properly train applicants, orient reviewers, and monitor outcomes in a managed fashion. Participants will be asked to provide feedback on the design elements through surveys and focused discussions.

Given the scope of the proposed changes, CIHR recognizes that course corrections and adjustments to timelines may be required along the way.

32. What technology will CIHR use to support the two new funding schemes and peer review process?

We are currently assessing various IT-based solutions to support the new funding schemes and peer review processes. A major driver for the piloting stage of the transition strategy is the need to select and test the most appropriate IT solutions for the proposed peer review enhancements.

33. Will CIHR provide feedback to reviewers on their peer review performance?

It is CIHR's intent, through the College of Reviewers Recognition and Performance Management Program, to recognize members' contributions and performance, as well as identify opportunities for further development. We are exploring approaches to collect quantitative and qualitative feedback on performance. Consultations on this topic will be required.

34. Will CIHR be consulting with Chairs and Scientific Officers before inviting existing peer reviewers to the College of Reviewers?

CIHR's current base of reviewers is invaluable, and provides a strong foundation for the initial recruitment waves for the College of Reviewers. CIHR will work with a number of key stakeholders, including Chairs and Scientific Officers of existing CIHR peer review committees, to identify and invite existing peer reviewers, as well as to nominate potential candidates, to the College of Reviewers.

35. What are the next steps in the implementation of the reforms?

CIHR's next steps towards implementing the reforms include:

• Developing and implementing pilots to assess aspects of the design in order to evaluate the changes to the peer review process.
• Develop a comprehensive training approach for applicants, peer reviewers and institutions.
• Visiting institutions across the country beginning in late January 2013 to continue the discussions on the reforms with the research community. The proposed format of the visit will include a meeting with senior administration staff followed by a town hall forum open to researchers from across CIHR's mandate.

36. Where can I obtain further information on CIHR's reforms?

Further information on the Reforms can be found on CIHR's website. You can also contact CIHR with questions by sending an email to: Roadmap-Plan.Strategique@cihr-irsc.gc.ca