CIHR Reviewers' Guide for Clinician Scientist Training & Salary Awards

October 2014

Table of Contents


Introduction

On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide instructions on the peer review process specific to the CIHR Clinician Scientist Training & Salary Award programs.

Peer Review at CIHR

Information on CIHR's objectives, governance and policies; an outline of the roles and responsibilities of peer review committee members; and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting the reviews.

Summary of the Peer Review Process

The CIHR Clinician Scientist Training & Salary Award programs receive two types of applications: Training Awards and Salary Awards. Applications received for these funding opportunities are "renewal applications", meaning that the applicants already hold either a Clinician Scientist Training Award or a Clinician Scientist Salary Award.

Each funding opportunity has its separate independent adjudication criteria and cut-offs. Although there are two types of applications, they are all reviewed by the same committee. The same criteria for funding cut-offs are applied to all applications within each funding level (i.e. training vs salary). Application types (i.e. new vs renewal) within each funding level are however evaluated together "on a level playing field". As always though, reviewers are reminded to take into account the stage of career and previous progress made and to vary the emphasis placed on track record appropriately.

This program uses an unstructured peer review process, using the online ResearchNet platform. The review process is completed in two (2) stages: an individual preliminary review of an assigned set of applications, followed by a teleconference committee meeting. The committee will be composed of one Chair, one Scientific Officer (SO) and internal reviewers. The number of internal reviewers will be determined based on the number of applications received.

Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:

Stage 1: Individual Review
Step 1: Read the pertinent documentation
Step 2: Determine ability to review and identify conflicts
Step 3: Conduct preliminary review
Step 4: Submit initial reviews and ratings on ResearchNet

Stage 2: The Committee Review
Step 1: Familiarize yourself with all applications
Step 2: Attend committee meeting

Stage 1: Individual Review

Step 1: Read the pertinent documentation

The peer review process for this program is described in details in this document. It is essential to read the document and be familiar with it. It is also important to read the following:

Step 2: Determine ability to review and identify conflicts

To determine the ability to review and identify conflicts of interests, reviewers are to follow these steps:

  • Log into ResearchNet.
  • On the home page, click on the link of their assigned committee to open the main task list.
  • Complete the task "Review Confidentiality and Conflict of Interest Guidelines" (once completed, it will "open" the other tasks).
  • Open the "Manage Conflicts/Ability to Review" task.
  • For each of the assigned application, use the information provided to indicate their ability to review (high-medium-low) or if they have a conflict of interest including conflict of language. If there is a conflict, CIHR will assign the application to another reviewer.

It is important to note that many candidates will likely be conducting research outside of the reviewer's research specialty. Therefore, they should review the application with a generalist's perspective and assess the overall quality of the research proposed by the candidate, using the appropriate adjudication criteria. However, if a reviewer feels that their level of comfort reviewing an application is unacceptably low, they should choose the expertise option "Not Enough Expertise".

Assignment of Applications

Based on the ability to review/conflict information, CIHR, in collaboration with the committee's Chair and Scientific Officer, will proceed with assigning the applications to reviewers. Efforts will be made to ensure a balanced workload, taking into consideration potential conflicts, language capabilities and areas of expertise. The final authority for the assignment of applications rests with CIHR.

Each application will be assigned to two (2) reviewers and one (1) reader to ensure the reliability of the rankings.

Step 3: Conduct preliminary review

Once the individual assignments are released, reviewers will have access to the full content of their assigned applications under the task "Conduct Reviews". They should then follow the steps below.

3.1 Review the adjudication criteria

Reviewers should first become familiar with the adjudication criteria for this funding opportunity. They can be found at the end of this document in Appendix A (for training award applications) and Appendix B (for salary award applications). These appendices identify which elements of the application to review for each criterion.

It is important to note that, for this program, expectations should differ based on the research area/discipline of the candidate. For example, publication productivity can vary when comparing a biomedical researcher, a clinician or a social scientist; where a health professional with a minimal amount of publications may have a publication record of superior quality than a biomedical researcher with numerous publications.

3.2 Read the assigned applications

Reviewers should read all of their assigned applications in details before rating any of them; and jot down notes to capture their impressions. It is recommended to separate the training awards and salary awards applications and concentrate on one type of application at a time.

The CIHR Clinician Scientists Reviewer Worksheets in Appendix C (for training award applications) and Appendix D (for salary award applications) provide templates that reviewers may wish to use. These worksheets are strictly for the reviewers' own personal use and will not be filed with CIHR.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (e.g. publications via PubMed, etc.) on the candidate, the supervisor (if applicable) or the proposed research institution. They are however free to consult published lists of journal impact factors when assessing the candidate's research accomplishments. It is important to note that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.

Reviewers should also be alert to unconscious bias related to gender, discipline or geographic location as detailed in the CIHR Peer Review Guide for Training and Salary Awards.

3.3 Rate the assigned applications

Reviewers are then asked to rate their assigned applications against each of the adjudication criteria described in Appendix A and Appendix B, using CIHR's traditional rating scale (below). It is particularly important that the full scale be used.

Reviewers are not bound by this initial rating and can change it at the peer review committee meeting.

Descriptor Range Outcome
Outstanding 4.5 – 4.9 May be Funded
Excellent 4.0 – 4.4
Very Good 3.5 – 3.9
Good 3.0 – 3.4 Not Fundable
Average 2.0 – 2.9
Below Average 1.0 – 1.9
Not Acceptable 0.0 – 0.9

For further details on CIHR's rating scale, please consult the web page entitled Ranking and Rating Scale Meaning and Use.

Please note that only applications rated 3.5 or higher are eligible for CIHR funding. Therefore, applications rated below 3.5 are not eligible for CIHR funds, including those from partnerships/priority announcement programs.

3.4 Provide a written assessment for each assigned application

Reviewers are asked to provide a short written assessment for each assigned applications that supports their ratings. The written reviews will serve to initiate the discussion of applications with other reviewers. They also provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training.

Comments should focus on the strengths and weaknesses of each adjudication criterion:

  • Keep it simple;
  • Use familiar descriptors that align with your rating;
  • Include justification, context and an explanation of your comments, if applicable, for each topic introduced;
  • Be clear and concise;
  • While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
  • Use objective and non-inflammatory language;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.

The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the comments and should not identify themselves in order to ensure the confidentiality of the review process.

3.5 Flag issues for CIHR's attention

Any concerns regarding eligibility, ethics, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments. For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards. Concerns may be expressed by email at ClinicianScientists@cihr-irsc.gc.ca, or by using the Issues for CIHR Attention Form.

These issues should not be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, reviewers should refer to the Grants & Awards Guide.

Step 4: Submit your reviews and ratings on ResearchNet

As the reviewers perform their evaluation, the reviews can be saved as drafts by selecting "Save draft copy" on ResearchNet. This allows them to make changes at a later time. However, in order to submit the reviews and ratings to CIHR, they must select "Submit Final Review". Afterwards, they will no longer be able to modify them prior to the meeting. After the committee meeting, they will have one week to revise their written comments only.

It is important for reviewers to respect the deadline provided by submitting their reviews and scores in ResearchNet by the date specified via correspondence with CIHR staff responsible for this program. Delays in the peer review process will jeopardize CIHR's ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that he/she may not be able to submit his/her reviews on or before the deadline, he/she must contact CIHR staff as soon as possible.

Stage 2: Committee Review

Stage 2 consists of a teleconference meeting of peer review members. The objective of this meeting is to discuss and rate the applications so that they may be ranked in order of excellence so that CIHR can generate a rank-order priority list to make funding decisions.

Step 1: Become familiar with all applications

It is the responsibility of all reviewers to familiarize themselves in advance of the meeting with all applications to be assessed at the meeting in order to facilitate the committee discussion.

Step 2: Attend the committee meeting

During the teleconference, any committee member who has a conflict of interest with an application cannot take part in the discussion of that application and will be asked to leave the teleconference for the duration of the discussion. The Chair and CIHR staff will be responsible for monitoring conflicts and for resolving areas of uncertainty.

The discussion of each application will proceed as follows:

  1. The two reviewers assigned to the application announce their initial ratings to one decimal place.
  2. The application is streamlined (not discussed) if it meets the following conditions:
    • The average of the internal reviewer's initial rating is below 3.5;
    • There is no objection from the other reviewers that the application not be discussed1.

If the application is not streamlined, the discussion proceeds as follows:

  1. The primary reviewer assigned to the application summarizes his/her assessment, describing strengths and weaknesses of the application.
  2. The second reviewer follows, concentrating on points of agreement or disagreement, and elaborate on points not addressed by the first reviewer.
  3. The reader will then concentrate on points of agreement or disagreement, and elaborate on points not addressed by the first and second reviewers.
  4. The Chair leads the discussion of the application by all committee members.
  5. The Scientific Officer reads the SO notes, capturing the key elements of the discussion to be considered when rating the application.
  6. The Chair seeks a consensus rating from the two reviewers assigned to the application. The internal reviewers may revise their initial ratings as they see fit. If a consensus cannot be reached, the mean value of the ratings of the two internal reviewers is used (round up, if necessary, to obtain a single decimal point).
  7. All committee members, including the two reviewers and the reader but excluding the Chair and Scientific Officer, cast individual confidential votes within +/- 0.5 of the consensus rating (they are not bound by the consensus rating). The final rating to be assigned to the application will be the average of these confidential votes. A vote will be taken even if the consensus rating is below 3.5 (i.e. not in the fundable range).

At the end of the meeting, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the teleconference. Following the discussion, a consensus rating is determined by the two reviewers and voting proceeds as before. The committee does not review the overall ranking of all applications at the end of the meeting as individuals with conflicts of interests would inevitably be present.

After the committee meeting, the reviewers will have one week to revise their written comments only should they wish to do so.

Feedback

An important component of the peer review process is the review of the committee's effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.

This discussion will occur at the end of meeting. Feedback can also be provided to the committee coordinator by email at ClinicianScientists@cihr-irsc.gc.ca.

Appendix A – CIHR Clinician Scientist Training Awards Adjudication criteria

The evaluation of Clinician Scientist Training Award applications is based on the following three criteria. According to the objectives of the Clinician Scientist Training Award funding program, the evaluation of these applications should be focused on the applicant and the predictors of successful post-training research activity. Here are the points to consider for each criterion.

Criterion Information Source Notes and advice for reviewers on how to review
1. Achievements and Activities of the Candidate
a) Training Expectations (candidate's plan)

Training expectations

Abstract

Research Project

Candidate's Common CV and Contributions Details

Dean of Faculty/Research Director Letter

University Nomination Letter

This section provides description of the applicant's career intentions and proposal for achieving them.

  • Assess the clarity and logic of the candidate's plans for a research career and the relevance of the proposed training.
  • Description of how the training they expect to acquire will contribute to their productivity and to the research goals they hope to achieve, and how this award will enable them to establish themselves as independent investigators.
b) Proposed Research Project

Training expectations

Abstract

Research Project

Candidate's Common CV and Contributions Details

Dean of Faculty/Research Director Letter

University Nomination Letter

This section provides a research project summary: a carefully planned, systematic study aimed at clearly answering a question in health research. It should be completed in collaboration with the proposed supervisor(s) and be written in general scientific language.

Note: it is not the project per se that is being assessed. The project should be viewed as an integral part of the candidate's development as a researcher.

  • Determine if the proposed project is adequate to the candidate given their education, experience and interests. Is the project the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training?
  • The proposed research project summary should:
    • Include the specific hypothesis of the research and describe the candidate's role on the project;
    • Provide a concise account of the subject matter, an overview of each part of the research plan, specific project aims and the methodology;
    • Reflect the significance of the project.
c) Honours, Awards and Academic Distinction

Training expectations

Abstract

Research Project

Candidate's Common CV and Contributions Details

Dean of Faculty/Research Director Letter

University Nomination Letter

This section provides a list of official recognitions received signifying special qualities of the candidate (i.e. citations, distinctions, Honours and Prizes/Awards), including accomplishments in terms of formal education and scholarship.

  • Take into consideration the career path that they have followed to date.
  • Assess the number, importance and breadth of the candidate's official recognitions and special distinctions relative to their education, training and work experience.
  • Note the length of time required to complete academic programs and any indications of special academic distinctions received.
  • Determine relevance to research and whether the recognition is regional, national or international.
d) Publications and Related Research Achievements

Training expectations

Abstract

Research Project

Candidate's Common CV and Contributions Details

Dean of Faculty/Research Director Letter

University Nomination Letter

This section provides a list of publications such as papers, articles, chapters or books (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights. Look for:

  • Evidence of research achievements relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., health professional career, time spent raising children, etc.).
  • For publications, observe the number of co-authors and the position of the candidate's name in the authors list (note that the importance of this position can vary depending on the discipline, etc.).
  • The candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (e.g., paper, article, chapter, book, etc.).
  • Try to get a sense of the entire body of work and its likely impact.
  • Note the publication dates and relate them to the candidate's education and training.
  • Consider the list of abstracts as an indication of conference presentation activities.
  • The candidate's other professional activities. Consider any patents or copyrights to which the candidate contributed.
2. Characteristics and Abilities of the Candidate

A perspective on the candidate provided by persons who are familiar with their characteristics and abilities. Keep in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors. With the Sponsor Assessment Form:

  • Look for evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with research career achievement.
  • Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not.
  • Read the supporting text carefully, taking note of the extent to which the sponsors justify their scores.
  • Look for indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent.
  • Look for indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.
  • If the candidate has had an opportunity to conduct research, look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.
  • Consider the following:
    • Do the detailed comments support the ratings outlined on the first page of the assessment?
    • How long has the sponsor known the candidate?
    • What is the relationship of the sponsor to the candidate?
3. Research Training Environment
Research Training Environment

Space, Facilities and Personnel Support Information

Supervisor(s) Common CV and Contribution Details

This section describes elements of the research environment (research activity, resources and mentorship) that will contribute directly or indirectly to the quality of the candidate's research training experience. It should demonstrate the commitment of the proposed supervisor(s) and their institution to support the development of the candidate's research project (funding, facilities, equipment, etc.) and professional development.

  • Look at the supervisor(s) research experience, qualifications, honours and awards. Examine their publication record to get a sense of productivity, impact and collaboration taking into consideration the different disciplines and their impacts on these.
  • Determine if the research environment, including space, facilities, and personnel support available is appropriate.
  • Get a sense of the resources available and the overall level of activity by reviewing the information on grants currently held, noting the extent to which the supervisor(s) was either listed as a principal or co-applicant for the funds.
  • Get a sense of the resources available and the overall level of activity.
  • Review the supervisor(s) training record. Note for each person listed: the level of training, length of time with the supervisor(s), degree received (if applicable) and current position.
  • Your assessment should take into consideration the career stage and discipline of the supervisor(s). Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.

Appendix B – CIHR Clinician Scientist Salary Awards Adjudication criteria

The evaluation of Clinician Scientist Salary Award applications is based on the following three criteria. According to the objectives of the Clinician Scientist Salary Award funding program, the evaluation of these applications should be focused on the applicant, rather than the proposed research project.

All aspects of the applications listed below should be reviewed taking into consideration the career stage and discipline of the candidate.

Criterion Information Source Notes and advice for reviewers on how to review
1. Applicant Track Record

Common CV

5-year Research Plan

Attached publications (if applicable)

The candidate should demonstrate that they have already established a track-record of significant achievement that shows their commitment to research and the originality and impact of their research relative to their career stage. It should also demonstrate that the candidate is forging an international reputation for excellence in their fields and exceptional promise for the future.

  • Assess the quality and diversity of the academic and research training received by the applicant. Are the candidate's prior training and research experience relevant and appropriate for this award?
  • What awards or acknowledgements of academic and research achievement has the applicant received?
  • What peer-reviewed grant funding has been obtained or applied for? Consider the applicant's role in obtaining multi-investigator grants and the numbers and categories of grants that support their research.
  • Does the applicant demonstrate leadership in the field through knowledge translation activities (e.g. commercialization, clinical activities, public health activities, invited speaker or moderator, etc.), and contributions to research training, mentorship and supervisory activities, contributions to professional activities (e.g. peer review committee, review of scientific manuscripts, review of grant/award applications, graduate committees, departmental and extra-departmental administrative, etc.)?
  • Does the applicant have a good publication record in peer-reviewed journals? To what extent does the applicant appear to have contributed to the work published? Patents and other significant contributions should be considered. Consider publications or papers accepted for publications, demonstrated capacity to publishing as principal author without mentors, the quality of journals and the continuity of production since the start of the career.
  • What research has been accomplished to date and has the applicant clearly demonstrated independence and originality? Significant contributions to team research should also be considered.
  • Are the sponsor letters from three reputable individuals in the field, and do they show evidence that the candidate has the characteristics and skills that correlate with career research achievement?
2. Research Plan

5 year Research Plan (references may be added to the Research proposal appendix)

Summary of Progress

Letters of collaboration (if applicable)

Common CV to evaluate the applicant's supervisory experience

Common CV and Contribution Details (if applicable) to evaluate dissemination and outreach activities

The candidate's research plan should cover the full duration of the salary award. A well prepared research plan should include details on the expected goals and their rationale, an explanation of how the goals are likely to be achieved (methodology), a rationalization as to where the research will be carried out (environment and resources), in collaboration with whom (team and collaborators) and finally details on the timeframe.

Note: the Research Proposal and Research Proposal Appendix should be viewed as a reference to the 5-year Research Plan.

  • Is the research plan relevant to the candidate's research career objectives?
  • Are the ideas put forward in the research plan innovative and/or original?
  • Does the research plan have the potential of significantly advancing our understanding of the area?
  • Is the proposed research feasible, given the resources and support available to the investigator?
  • Has preliminary data been accumulated to support the 5 year research plan?
  • Have strong research interactions and collaborations been established? What are the candidate's contributions to the proposed collaborations? Are the proposed collaboration of high quality and what level of impact could they have on the professional growth of the candidate?
  • Will the quality and extent of proposed dissemination and outreach activities be within and/or beyond the academic community?
  • Is the applicant's leadership role clearly outlined in his/her research plan? Does the research plan provide evidence of the applicant's leadership in the design and conduct of the proposed research? Leadership can be demonstrated through the applicant's engagement as a mentor, their ability to manage research, to contribute novel ideas to their research program, to make decisions that are crucial to the success of the research program, to lead his/her research collaboratively, have excellent working relationships with those around him/her, etc.
  • For New Investigators, has the applicant demonstrated independence or shown promise to become independent from former supervisors?
3. Environment and Support

Dean of Faculty/Research Director letter

Appendix 2A (parts 1, 2 and 3) completed by the Head of Department

It is imperative that the salary award application demonstrates a strong institutional/organizational commitment to the continued scientific development and productivity of the candidate. The application should reveal a clear commitment from the institution/organization to ensure that the majority of the candidate's efforts will be devoted directly to the research plan, detailed in the application, with the remaining efforts being devoted to an appropriate balance of teaching, administrative, and clinical responsibilities.

  • What is (or will be) the applicant's position within the institution?
  • What space, operating funds, infrastructure and/or other resources will be available to the candidate and are they adequate?
  • Has the institution demonstrated a commitment to enable the candidate to devote full time to research and related duties by releasing the applicant from teaching, administration, clinical work and/or other responsibilities?
  • Has the institution demonstrated support for the scientific development of the candidate and their independent research program?
  • Does the institution or organization demonstrate leadership in the candidate's chosen field?
  • Will the candidate receive adequate scientific and career guidance?

Appendix C – CIHR Clinician Scientist Training Awards Reviewer Worksheet

The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.

Applicant Name:
Application #:

Criterion Reviewer's Comments
1. Achievements and activities of the candidate (overall)
  • Candidate's plan (Training Expectations)

Strengths:

Weaknesses:

  • Proposed research project
  • Honours, awards and academic distinctions
  • Publications and related research achievements
2. Characteristics and abilities of the candidate (overall)
  • Candidate exhibits the characteristics and skills that correlate with career research achievement.

Strengths:

Weaknesses:

  • Sponsors perceive the candidate as an investigative type.
  • Sponsors perceive the candidate as both energetic and capable of being highly focused.
  • Creativity in setting research goals
  • Candidate has had an opportunity to conduct research.
3. Research training environment (overall)
  • Research experience, qualifications, honours and awards of the supervisor.

Strengths:

Weaknesses:

  • Determine the research environment, including space, facilities and personnel support available.
  • Supervisor's training record.
Score:

Appendix D – CIHR Clinician Scientist Salary Awards Reviewer Worksheet

The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.

Applicant Name:
Application #:

Criterion Reviewer's Comments
1. Applicant Track Record
  • Academic and research training received by the applicant

Strengths:

Weaknesses:

  • Prestigious awards or acknowledgements of academic achievement
  • Publications and related research activity
  • Research accomplished to date - has the applicant clearly demonstrated independence and originality? Contributions to team research?
  • To what extent does the applicant demonstrate leadership in the field? Quality of the applicant's training, mentorship and supervisory activities.
  • Record of funding
2. Research Plan
  • Are the ideas put forward innovative and/or original?

Strengths:

Weaknesses:

  • Could the research plan extend significantly our understanding of the area?
  • Is the project feasible, given the resources and support available to the investigator?
  • Have strong research interactions and collaborations been established?
  • Has the applicant shown evidence of leadership?
  • Has the applicant demonstrated independence or shown promise to become independent from former supervisors? (New Investigators)
3. Environment and Support
  • What is (or will be) the applicant's position within the institution?

Strengths:

Weaknesses:

  • Institutional commitment to space, operating funds and/or infrastructure.
  • Has the institution demonstrated a commitment to protect the candidate's research time?
  • Does the institution or organization demonstrate leadership in the candidate's chosen field?
  • Letters of support
Score:

Footnotes

Footnote 1

If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports and the SO notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.

1