CIHR Peer Review Manual for Clinician Scientist Award Applications
Revised: January 2014
Table of Contents
Section I – Policy & Guiding Principles
- 1. Purpose of the Manual
- 2. Peer Review at CIHR
- 3. Principles of Peer Review
- 4. Policies Impacting Peer Review
Section II – Peer Review
- 1. Peer Review Committee Members
- 2. Types of Applications
- 3. Peer Review Process
- 3.1 Initializing the Review Process
- 3.2 Evaluating the Applications
- 3.2.1 Evaluation Criteria
- 220.127.116.11 Evaluation Criteria for Training Award Application
- 18.104.22.168 Evaluation Criteria for Salary Award Applications
- 3.2.2 Relevance Review
- 3.2.3 Reviewing Applications in Preparation for Meeting
- 3.3 During the Meeting
- 3.3.1 Streamlining
- 3.3.2 Rating of applications
- 3.3.3 Flagging of applications for special attention
- 3.3.4 End of Meeting review
- 3.3.5 Funding Decision
- Appendix I: Sequence of Steps for Peer Review of a Clinician Scientist Award Application
- Appendix II: Reviewer Worksheet for Clinician Scientist Training Award Applications
- Appendix III: Reviewer Worksheet for Clinician Scientist Salary Award Applications
Section I: Policy & Guiding Principles
1. Purpose of the Manual
On behalf of CIHR, we would like to thank you for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like yourself who generously give of their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.
The peer review process is described in detail in this manual and on CIHR's website. It is essential that committee members read and be familiar with this Manual and the Funding Opportunity for which you review. Concise information on the role of each committee member and their responsibilities is also available on the CIHR website: please refer to the Peer Review Committee Members Role page.
The purpose of this manual is to provide information on CIHR's objectives, governance and policies; to outline the roles and responsibilities of peer review committee members; and to define the policies and procedures for peer review of applications.
For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.
2. Peer Review at CIHR
The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:
"To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system."
The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also provides accountability, not only to the Government of Canada and the Canadian taxpayer – the source of CIHR funding – but to the research community at large. Peer review is carried out by committees of experts that encompass all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health).
Peer review is overseen by CIHR's Science Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with CIHR's 5-year Strategic Plan. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC's responsibilities.
For more information on the different types of Peer Review and meeting formats, please refer to Types of Review at CIHR.
3. Principles of Peer Review
Confidentiality is information about a person that shall not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. Thus, information contained in applications, reviewer reports, names of reviewers and committee discussions are all strictly confidential. The use of this information for any other purpose than what is outlined here is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.
Committee members must not discuss with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the chances of success or failure. Applicants must not contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank within committee, etc.). All requests for information on an application or a reviewer report should be referred to the Deputy Director at CIHR responsible for the committee in question.By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision and CIHR will not edit the reviews provided. The identity of the reviewers will not be revealed to the applicants under any circumstances. However, a list of peer review committee members will be published on the CIHR website 60 days after the Science Council approves funding for a competition cycle.
3.2 Conflict of Interest
CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a Conflict of Interest means a conflict between a Participant's duties and responsibilities with regard to the Review Process, and a Participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the Participant:
- would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
- has a professional or personal relationship with an Applicant or the Applicant’s institution; or
- has a direct or indirect financial interest in a funding opportunity or application being reviewed.
A conflict of interest may be deemed to exist or perceived as such when review committee members, external reviewers or observers:
- are a relative or close friend, or have a personal relationship with the applicants;
- are in a position to gain or lose financially/materially from the funding of the application;
- have had long-standing scientific or personal differences with the applicants;
- are currently affiliated with the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
- are closely professionally affiliated with the applicants, as a result of having in the last six years:
- frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
- been a supervisor or a trainee of the applicants;
- collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
- been employed by the institution, when an institution is the applicant; and/or
- feel for any reason unable to provide an impartial review of the application.
All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff and the Chair are responsible for resolving areas of uncertainty during the committee meeting.
All committee members must read and agree to abide by the COIC policy prior to viewing any application information.
Success of the peer review system is critically dependent upon the willingness and ability of all committee members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application. In programs where written reviews are required, these reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that your review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, which includes helping the applicant by pointing out deficiencies that could be repaired in a resubmission, will help to convince a disappointed applicant that you provided a fair assessment of the proposal.
4. Policies Impacting Peer Review
Some policies may not apply to all salary award programs. Please contact your committee coordinator for more information.
4.1 International Collaborations
As stated in the CIHR Act, one of the ways CIHR fulfills its mandate is by "pursuing opportunities and providing support for the participation of Canadian scientists in international collaborations and partnerships in health research." As a result, CIHR accepts applications for research to be carried out in, or in collaboration with applicants based in, other countries. The international nature of the research should not be a factor in the scientific assessment of the proposal, beyond how it relates to the feasibility of the proposed research and the quality of the research question. Reviewers should also not be influenced by the funding obtained or requested for the international components when recommending a budget for the Canadian component(s).
For detailed information on applying for funding with an international partnership component, please refer to the subsections titled Global Health Research and International Collaborations in the Grants and Awards Guide.
4.2 Knowledge Translation
Knowledge translation is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants and research allowances.
For end of grant KT, many means of dissemination exist and the onus is on the researcher/trainee to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified. Please consult About Knowledge Translation for more information.
4.3 Gender, Sex and Health Research
Applicants are encouraged to demonstrate the use of gender and sex-based analysis in applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands understanding of health determinants in both sexes and results in improvements in health and health care. For more information on how peer reviewers can assess whether gender and/or sex are appropriately integrated into CIHR applicants' proposed research designs, please refer to Integrating Gender and Sex in Health Research: A Tool for CIHR Peer Reviewers.
4.4 Official Language Minority Communities
Federal agencies are required to take positive measures to ensure the support and recognition of minority language communities in Canada. For CIHR, this means an obligation to promote health research in these communities. For further information, please refer to the initiative on Official Language Minority Communities. Research proposals in these areas should still be subject to the same rigorous peer review process as any other application. However, the justification for promoting health research in minority language communities should not be a factor in the assessment.
4.5 Publications and Productivity
An important evaluation criterion in all funding programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing the quality of publications, peer review committees should focus on the quality of a publication's content and NOT simply the number of publications nor the quality or impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.
CIHR funds researchers in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."
When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer review committees must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.
Section II – Peer Review
1. Peer Review Committee Members
Peer Review committee membership will vary depending on the type of peer review. In general, individual committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment. For more details, please refer to the Peer Review Membership Guidelines. Committees as a whole should also satisfy the need to cover the range of research areas for which the committee is responsible, to appropriately represent the Canadian health research community, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members. For more details, please refer to the Procedure for Selection of Peer Review Committee Members.
For further information on peer reviewers at CIHR, please consult the Peer Review Committee Members Role page.
Complete instructions for peer review committee members are available at the Instructions for Committee Members page.
2. Types of Applications
Clinician Scientist Award applications may be new applications, resubmissions of a previously unsuccessful applications, or renewal applications. All application types are evaluated together "on a level playing field" and the same criteria and funding cut-offs are applied to all, though peer review committee members are reminded to take the stage of career and previous progress made into account and to vary the emphasis placed on track record appropriately. Note that Phase One and Phase Two applications are evaluated independently using separate evaluation criteria and cut-offs.
3. Peer Review Process
The prime responsibilities of a peer review committee are to evaluate applications submitted for a particular competition and to rank them in order of excellence using CIHR's rating scale. The evaluation of applications is finalized by a committee meeting to discuss and rate the applications, from which CIHR generates a rank-order priority list to make funding decisions.
3.1 Initializing the Review Process
All eligible applications received by the appropriate deadline date are entered into the competition. Applications must be complete at the time of submission; otherwise they are withdrawn from the competition.
CIHR staff review the applications for eligibility and compliance. Applications are assigned to a peer review committees with the mandate that most closely aligns with the applicant's training, credentials and area of research. Committee members will then be given access to the applications to declare conflicts of interest and indicate their level of expertise (if applicable). It is important to note that many candidates will likely be conducting research outside of the reviewer's research specialty. Reviewers should keep in mind that they are to review the application with a generalist's perspective and assess the overall quality of the research proposed by the candidate. However, should a reviewer feel that his/her level of comfort of reviewing an application is unacceptably low; he/she may identify their expertise as "low" in Research-Net.
Chairs, Scientific Officers and/or CIHR staff will assign the applications to committee members. Efforts will be made to ensure a balanced workload, taking into consideration potential conflicts, language capabilities and areas of expertise. The final authority for the assignment of applications rests with CIHR. Any committee member who has a conflict of interest with an application (as defined in Section 3.2, above) must not take part in the evaluation of that application. All committee members are then given access to the full applications assigned to their committee, four to six weeks before the peer review committee meeting.
3.2 Evaluating the Applications
All applications submitted to a funding opportunity are treated equally for evaluation; the same criteria and funding cut-offs are applied to all. Reviewers evaluate applications in reference to the evaluation criteria listed in the funding opportunity details, which vary by program and by funding opportunity.
3.2.1 Evaluation Criteria
22.214.171.124 Evaluation Criteria for Training Award Application
The evaluation of Clinician Scientist Training Awards - Phase One applications is based on the following three criteria:
- Achievements and Activities of the Candidate
- Characteristics and Abilities of the Candidate
- Research Training Environment
According to the objectives of the Clinician Scientist Training Award funding program, the evaluation of these applications should be focused on the applicant and the predictors of successful post-training research activity. Here are the points to consider for each criterion:
1. Achievements and Activities of the Candidate
- Candidate's plans (Training expectations). A description of the applicant's career intentions and proposal for achieving them. Look for clarity and logic in the explanation of the candidate's plans for a research career and the relevance of the proposed training.
- Proposed research project. A carefully planned, systematic study aimed at clearly answering a question in health research. The ideal project is one that is best for the candidate given their education, experience and interests. It is the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training. Bear in mind that it is not the project per se that is being assessed. It is the project as an integral part of the candidate's development as a researcher.
- Honours, awards and academic distinction. Official recognition or prizes signifying special qualities of the recipient. Includes accomplishments in terms of formal education and scholarship. Take into consideration the career path that they have followed to date. Assess the number, importance and breadth of the candidate's special distinctions relative to their education, training and work experience. Note relevance to research and whether the recognition is regional, national or international. Note the length of time required to complete academic programs and any indications of special academic distinction.
- Publications and related research achievements such as articles, chapters or books published (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights. Look for:
- Evidence of achievements in research relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., time spent raising children).
- Publications, observe the number of co-authors and the position of the candidate's name in the authors list (note that the importance of this position can vary depending on the discipline, etc.). Note the candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (article, chapter, book, etc.).
Try to get a sense of the entire body of work and its likely impact. Note the publication dates and relate them to the candidate's education and training. Consider the list of abstracts as an indication of conference presentation activities. Note the candidate's other professional activities. Consider any patents or copyrights to which the candidate contributed.
The information required to review this criterion can be found in the following sections of the application:
- Training expectations
- Research Project
- Candidate's Common CV and Contributions Details
- Dean of Faculty/Research Director Letter
- University Nomination Letter
2. Characteristics and Abilities of the Candidate
- A perspective on the candidate provided by persons who are familiar with their characteristics and abilities. Look for:
- Evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with career research achievement. Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not. Read the supporting text carefully, taking note of the extent to which they justify the scores.
- Indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent.
- Indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.
- Mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing, if the candidate has had an opportunity to conduct research.
The information required to review this criterion can be found in the following sections of the application:
- Sponsor Forms
3. Research Training Environment
- Elements of the research environment ((research activity, resources and mentorship) that will contribute directly or indirectly to the quality of the candidate's research training experience. Look for:
- Review information on the research experience, qualifications, honours and awards of the supervisor. Examine the supervisor's publication record to get a sense of productivity; impact and collaboration (as noted above, please consider the different disciplines and their impacts on these).
- Determine the research environment, including space, facilities, and personnel support available. Review the information on grants currently held, noting the extent to which the supervisor was either listed as a principal or co-applicant for the funds. Get a sense of the resources available and the overall level of activity
- Review the supervisor's training record. Note for each person listed: the level of training, length of time with the supervisor, degree received (if applicable) and current position. Your assessment should take into consideration the career stage and discipline of the supervisor. Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.
The information required to review this criterion can be found in the following sections of the application:
- Space, Facilities and Personnel Support Information
- Supervisor(s) Common CV and Contribution Details
Please refer to the Reviewer Worksheet available in Appendix II to assist in preparing your review. This document is for your use only and therefore will not be forwarded to applicants.
126.96.36.199 Evaluation Criteria for Salary Award Applications
The evaluation of Clinician Scientist Salary Award – Phase 2 applications is based on the following three criteria:
- Applicant Track Record
- Research Plan
- Environment and Support
Here are the points to consider for each criterion:
According to the objectives of the Clinician Scientist Salary Award funding program, the evaluation of these applications should be focused on the applicant, rather than the proposed research project.
1. Track Record of the Candidate
The candidate should demonstrate that they have already established a track-record of significant achievement that shows their commitment to research and the originality and impact of their research relative to their career stage. It should also demonstrate that the candidate is forging an international reputation for excellence in their fields and exceptional promise for the future.
Reviewers should consider the following in their evaluation of the candidate's track record:
- Assess the quality and diversity of the academic and research training received by the applicant. Are the candidate's prior training and research experience relevant and appropriate for this award?
- What awards or acknowledgements of academic and research achievement has the applicant received?
- What peer-reviewed grant funding has been obtained or applied for? Consider the applicant's role in obtaining multi-investigator grants and the numbers and categories of grants that support their research.
- Does the applicant demonstrate leadership in the field through knowledge translation activities (e.g. commercialization, clinical activities, public health activities, invited speaker or moderator, etc), and contributions to research training, mentorship and supervisory activities, contributions to professional activities (e.g. peer review committee, review of scientific manuscripts, review of grant/award applications, graduate committees, departmental and extra-departmental administrative, etc).
- Does the applicant have a good publication record in peer-reviewed journals? To what extent does the applicant appear to have contributed to the work published? Patents and other significant contributions should be considered. Consider publications or papers accepted for publications, demonstrated capacity to publishing as principal author without mentors, the quality of journals and the continuity of production since the start of the career. Please refer to the Publications and Productivity section of this guide.
- What research has been accomplished to date and has the applicant clearly demonstrated independence and originality? Significant contributions to team research should also be considered.
- Are the sponsor letters from three reputable individuals in the field, and do they show evidence that the candidate has the characteristics and skills that correlate with career research achievement?
- Common CV and Contribution Details (if applicable)
- 5 year Research Plan to evaluate the applicant's role on other grants (if applicable)
- Attached publications (if applicable)
Please refer to the Reviewer Worksheet available in Appendix III to assist in preparing your review. This document is for your use only and therefore will not be forwarded to applicants.
Additional factors to be considered under each criterion may also be described in the funding opportunity details. Please contact your committee coordinator if you need further guidance on how to apply the individual criteria. All aspects of the application listed above should be reviewed taking into consideration the career stage and discipline of the applicant.
2. Research Plan
The candidate's research plan should cover the full duration of the salary award. A well prepared research plan should include details on the expected goals and their rationale, an explanation of how the goals are likely to be achieved (methodology), a rationalization as to where the research will be carried out (environment and resources), in collaboration with whom (team and collaborators) and finally details on the timeframe.
Reviewers should consider the following in their evaluation of the research plan:
- Is the research plan relevant to the candidate's research career objectives?
- Are the ideas put forward in the research plan innovative and/or original?
- Does the research plan have the potential of significantly advancing our understanding of the area?
- Is the proposed research feasible, given the resources and support available to the investigator?
- Has preliminary data been accumulated to support the 5 year research plan?
- Have strong research interactions and collaborations been established? What are the candidate's contributions to the proposed collaborations? Are the proposed collaboration of high quality and what level of impact could they have on the professional growth of the candidate?
- Will the quality and extent of proposed dissemination and outreach activities be within and/or beyond the academic community?
- Is the applicant's leadership role clearly outlined in his/her research plan? Does the research plan provide evidence of the applicant's leadership in the design and conduct of the proposed research? Leadership can be demonstrated through the applicant's engagement as a mentor, their ability to manage research, to contribute novel ideas to their research program, to make decisions that are crucial to the success of the research program, to lead his/her research collaboratively, have excellent working relationships with those around him/her, etc.
- For New Investigators, has the applicant demonstrated independence or shown promise to become independent from former supervisors?
- 5 year Research Plan
- Summary of Progress
- Letters of collaboration (if applicable)
- Common CV to evaluate the applicant's supervisory experience
- Common CV and Contribution Details (if applicable) to evaluate dissemination and outreach activities
Note: The Research Proposal and Research Proposal Appendix should be viewed as a reference to the 5 year Research Plan.
3. Environment and Support
It is imperative that the salary award application demonstrates a strong institutional/organizational commitment to the continued scientific development and productivity of the candidate. The application should reveal a clear commitment from the institution/organization to ensure that the majority of the candidate's efforts will be devoted directly to the research plan, detailed in the application, with the remaining efforts being devoted to an appropriate balance of teaching, administrative, and clinical responsibilities.
Reviewers should consider the following in their evaluation of the environment and support:
- What is (or will be) the applicant's position within the institution?
- What space, operating funds, infrastructure and/or other resources will be available to the candidate and are they adequate?
- Has the institution demonstrated a commitment to enable the candidate to devote full time to research and related duties by releasing the applicant from teaching, administration, clinical work and/or other responsibilities?
- Has the institution demonstrated support for the scientific development of the candidate and their independent research program?
- Does the institution or organization demonstrate leadership in the candidate's chosen field?
- Will the candidate receive adequate scientific and career guidance?
- Dean of Faculty/Research Director letter
- Appendix 2A (part 1, 2 and 3) completed by the Head of Department
3.2.2 Relevance Review
The Relevance Review Process is used by strategic leads and/or partners to assess the alignment of an application with a specific research theme described in the funding opportunity (FO). As the name implies, the process is used when it is important for applications to be relevant to (or in alignment with) targeted research components of the FO. This review approach is generally reserved for strategic FOs and Priority Announcements (PAs).
3.2.3 Reviewing Applications in Preparation for Meeting
Read all of your assigned applications before rating any of them. Note that you are reviewing two different types of applications: Training Awards, these are the Phase One applications; and Salary Awards, these are the Phase Two applications. As such, ensure that you rate each application using the appropriate selection criteria described in Section 3.2.1 of this guide. It is recommended that reviewers separate their training awards and salary awards applications and devote their attention to one type of application at a time.
As you examine each application, take some notes to capture your impressions. The Reviewer Worksheets, located in Appendix II (for training award applications) and Appendix III (for salary award applications) of this guide, provides a template that you may use. Note that the worksheets will not be filed with CIHR.
Be alert to unconscious bias related to gender, discipline or geographic location. Remember that:
- Career interruptions for child bearing and raising can influence opportunity for knowledge production, publications and related variables;
- Different disciplines and environments offer different opportunity for publication; and,
- The reputation of institutions should not affect your view of candidates or their research training environment.
It is still the responsibility of all peer review committee members to familiarize themselves in advance of the meeting with all applications to be assessed by their committee. This will assist peer reviewers to prepare their internal reviewer's reports.
In advance of the meeting, reviewers are required to complete the following tasks on ResearchNet:
- Upload reviews: Reviewers are required to provide a written review of each application assigned to them. To ensure that a reviewer's time is used in the most efficient way, a structured review process is being used. For each above-mentioned criterion, reviewers will be asked to comment on the strengths and weaknesses only of that particular aspect of the applications. The review should be clear and concise, using objective and non-inflammatory language, and include justification. Constructive advice to the applicant will allow him/her to improve the quality and efficiency of the proposed research. The applicant will receive the review as it is submitted by the reviewer. For this reason, please do not identify yourself in order to ensure the confidentiality of the review process.
Please refer to the template below to assist you in preparing your written reviews.
|1. Achievements and Activities of the Candidate||Strengths
|2. Characteristics and Abilities of the Candidate||
|3. Research Training Environment||
|1. Environment and Support||
|2. Research Plan||
|3. Track Record of the Candidate||
Note Reviewers are to consider each of the three review criteria above in the determination of their overall score.
- Provide a numerical rating for the application. To ensure consistency, all reviewers must adhere to a common rating scale. Reviewers rate applications between 0.0 and 4.9 (in increments of 0.1, with 4.9 being the highest and 0 being the lowest); applications with a rating of 3.5 or higher are considered for funding. For additional information, please see Ranking and Rating Scale Meaning and Use.
- Divide the applications reviewed into a top and bottom group based on their overall quality. The top group should include the applications considered to be highly competitive and most deserving of being funded. This assessment is to be based on the reviewers' experience and will be used during the streamlining phase of the review (Section 3.3.1). The proportion of applications in each group may vary depending on the overall quality of the pool of applications reviewed relative to the reviewers' experience.
The deadline for uploading your reviews to ResearchNet, along with your assessments of overall quality and initial ratings, is one week before the meeting date. Reviews can be saved as drafts, by selecting "save draft copy" on ResearchNet prior to submission. In order to access the other reviews for the applications you were assigned, which would allow you to be better prepared for the discussion at the committee meeting; you must select "submit final review" on ResearchNet. Afterwards, you will no longer be able to modify it prior to the meeting. If you wish to revise your reviews after the committee meeting, you will have one week to directly upload your changes to ResearchNet.
Please contact your committee Program Delivery Coordinator if you encounter any technical issues.
3.3 During the Meeting
The prime responsibilities of a peer review committee are to evaluate applications submitted for a particular competition and to rate them so that they may be ranked in order of priority and excellence. It is important that committees follow defined procedures in order to function in a consistent manner. For a summary of the review procedure, please see Appendix I.
Any committee member who has a conflict of interest with an application (as defined in Section 3.2) must not take part in the discussion of that application. Committee members in conflict must leave the room/teleconference line before the application is discussed. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.
The evaluation of applications for funding occurs in two phases: (1) an in-depth "at-home" review by at least two peer reviewers to produce written evaluations, and (2) a committee meeting to discuss and rate the applications, from which CIHR generates a rank-order priority list to make funding decisions. For many programs, less than one-third of the applications are ultimately funded; thus, it is important that committees focus their discussions on the most competitive applications to ensure that an accurate rank-order list is generated. To help support this goal, a streamlining process is used to eliminate non-competitive applications from the discussion process, allowing peer reviewers more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding. Applications that are streamlined still benefit from the review process as they receive the written reviews from the assigned reviewers. For detailed information on streamlining, please see streamlining on the CIHR website.
The assessment of each application at peer review committee meetings begins with internal reviewers announcing their initial ratings to one decimal place. An application is then streamlined if it meets the following conditions:
- both reviewers placed the application in their bottom group;
- the average of the internal reviewers' initial ratings is <3.50;
- there is no objection from the other committee members that the application not be discussed.
Notes: If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports and the Scientific Officer notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.
3.3.2 Rating of applications
If an application is not streamlined, the discussion proceeds as follows:
- The primary reviewer presents his/her assessment, describing strengths and weaknesses of the application;
- The secondary reviewer follows, concentrating on points of agreement or disagreement and elaborating points not addressed by the first reviewer;
- The Chair leads the discussion of the application by all committee members;
- The Scientific Officer reads back the Scientific Officer notes, capturing the key elements of the discussion to be considered when rating the application;
- The Chair seeks a "consensus rating" from the two internal reviewers. The internal reviewers may revise their initial ratings as they see fit. If a consensus cannot be reached, the mean value of the ratings of the two internal reviewers is used (round up, if necessary, to obtain a single decimal point);
- All committee members, including the two internal reviewers but excluding the Chair and Scientific Officer, then cast individual confidential votes within ±0.5 of the consensus rating. The internal reviewers are not bound to the consensus rating. The rating assigned to the application is the average of these confidential votes. A vote is taken even if the consensus rating is <3.5 (i.e., not in the fundable range).
3.3.3 Flagging of applications for special attention
Any concerns in the following areas should be discussed and, if necessary, flagged for CIHR staff to address. These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please see the Grants and Awards Guide.
- Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria to receive CIHR funding.
- Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may comment on specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
- Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled "Certification Requirements", but it is essential that this be verified by committee members.
- Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the "Controlled Drugs and Substances Act" are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded. For more information please refer to the Pending Grants and Awards page in the Grants and Awards Guide.
3.3.4 End of Meeting review
Once all applications have been reviewed, if the peer review committee feels that any application(s) has been treated inconsistently, a re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, a consensus rating is determined by the two internal reviewers and voting proceeds as before. The committee does not review the overall rankings of all applications at the end of the meeting as individuals with conflicts of interest would inevitably be present.
An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This discussion provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.
3.3.5 Funding Decision
Following peer review, CIHR staff generates a final rank list based on the committee recommendations, to be reviewed by CIHR's Chief Scientific Officer (CSO) and Chief Financial Officer (CFO). Applications will be funded from the top down in order of ranking as far as the budget will allow. The CSO and CFO consider the funding recommendation in light of criteria established by Science Council (SC) and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications webpage as soon as it is available.
Once the SC has approved the list of applicants to be funded, all applicants are sent a Notice of Decision, indicating whether or not their application was approved. They will also receive a copy of all reviews, the Scientific Officer notes (if applicable) and an Offer of Award that details the budget, duration and conditions of funding.
Applications that have been flagged for special attention and followed up by CIHR staff are withheld as "pending". The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members if necessary prior to a final decision regarding funding.
Appendix I: Sequence of Steps for Peer Review of a Clinician Scientist Award Application
1. Initial ratings: Members in conflict leave the room. The two internal reviewers announce their initial rating.
Note: ratings can be different from those previously posted on ResearchNet
2. Streamlining (optional):
Review is terminated if the following conditions are met:
- the application is not considered competitive by both reviewers (placed into bottom group by both reviewers).
- the mean of the rating of the two internal reviewers is <3.50
- there is no objection from other committee members
Committee members will not vote. The rating is calculated as the mean of the rating of the two internal reviewers. No SO notes will be taken.
3. Internal Reviewers:
The primary reviewer presents his/her assessment, describing strengths and weaknesses of the application followed by the secondary reviewer, who should concentrate on points of agreement or disagreement and elaborate on points not addressed by the first reviewer.
4. Reader raises additional issues (if applicable)
5. Committee discussion of application should focus on:
- factors important in rating
- differences of view between reviewers
6. Scientific Officer reads SO notes to the committee:
- summary of discussions
- strength and weaknesses of application
7. Consensus rating by internal reviewers:
- use full scale
- check consistency with previous applications
If a consensus cannot be reached, use mean of internal reviewers' ratings.
8. Individual ratings:
- ± 0.5 of consensus rating
- Votes are confidential
- The two internal reviewers are not bound to consensus rating
9. Issues to be flagged:
- human stem cells
- Section 56 of the federal Controlled Drugs and Substances Act
10. Scientific Officer reads final notes for review / modifications / additions by committee
Appendix II: Reviewer Worksheet for Clinician Scientist Training Award Applications
|NAME OF APPLICANT
|1. Achievements and activities of the candidate (Overall)|
|a) Candidate's plans (Training expectations)|
|b) Proposed research project|
|c) Honours, awards and academic distinction|
|d) Publications and related research achievements|
|2. Characteristics and abilities of the candidate (Overall)|
|a) Candidate exhibits the characteristics and skills that correlate with career research achievement|
|b) Sponsors perceive the candidate as an investigative type|
|c) Sponsors perceive the candidate as both energetic and capable of being highly focused|
|d) Creativity in setting research goals|
|e) Candidate has had an opportunity to conduct research|
|3.Research training environment (Overall)|
|a) Research experience, qualifications, honours and awards of the supervisor|
|b) Determine the research environment, including space, facilities, and personnel support available|
|c) Supervisor's training record|
Appendix III: Reviewer Worksheet for Clinician Scientist Salary Award Applications
|NAME OF APPLICANT
|1. Applicant Track Record|
|a) Applicant demonstrates that they have already established a track-record of significant achievement|
|b) Applicant demonstrates the originality and impact of their research relative to their career stage|
|c) Applicant demonstrates he/she is building international reputation for excellence in his/her fields and exceptional promise for the future|
|2. Research Plan|
|a) The candidate's research plan encompasses the entire research plan for the first 5 years of the salary award|
|3. Environment and Support|
|a) Application demonstrates a strong institutional/organizational commitment to the career development of the candidate|