CIHR Reviewers’ Guide for Fellowship Awards

Table of Contents

Introduction

On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide instructions on the peer review process specific to the CIHR Fellowship Awards program.

Peer Review at CIHR

Information on CIHR’s objectives, governance and policies; an outline of the roles and responsibilities of peer review committee members; and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting the reviews.

Summary of the Peer Review Process

The CIHR Fellowship Awards program uses an individual structured review process, using the online ResearchNet platform. The review process is completed in two (2) stages: an individual review of an assigned set of applications, followed by a virtual discussion of the applications. Interactive learning sessions which include training on components of the review process are available online to guide you. There is no committee meeting for this funding opportunity.

There are five (5) peer review committees for the Fellowship Awards program. Each application received will be assigned to the committee with the mandate that most closely aligns with the applicant’s training, credentials and area of research. The link to the committees’ mandates is listed under the section “Read the pertinent documentation”.

All eligible applications received will be assigned to three (3) reviewers.

Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:

  • Stage 1: Individual Review
    • Step 1: Read the pertinent documentation
    • Step 2: Identify conflicts of interests
    • Step 3: Conduct preliminary review
  • Stage 2: Virtual Discussion
    • Step 1: Access the other reviewer’s evaluation
    • Step 2: Discuss application
    • Step 3: Edit and confirm reviews
    • Step 4: Submit final reviews

Stage 1: Individual Review

Step 1: Read the pertinent documentation

The peer review process for this program is described in details in this document. It is essential to read the document and be familiar with it. It is also important to read the following:

Step 2: Identify conflicts of interests

To identify conflicts of interests, reviewers are to follow these steps:

  • Log into ResearchNet.
  • On the home page, click on the link of their assigned committee to open the main task list.
  • Complete the task “Conflict of Interest and Confidentiality Policy and Reviewer Consent” (once completed, it will “open” the other tasks).
  • Open the “Manage Conflicts/Ability to Review” task.
  • For each of the assigned application, use the information provided to indicate if they are able to review or if there is a conflict. If there is a conflict, CIHR will reassign the application to another reviewer.

Step 3: Conduct preliminary review

Once conflicts have been identified, the full content of the remaining assigned applications will be available under the task “Conduct Reviews”. Reviewers should then follow the steps below.

3.1 Review the adjudication criteria

Reviewers should first become familiar with the adjudication criteria for this funding opportunity. They can be found at the end of this document in Appendix A. This appendix provides an interpretation of each criterion and identifies which elements of the application to review for that criterion.

It is important to note that, for this program, expectations should differ based on the research area/discipline of the candidate. For example, publication productivity can vary when comparing a biomedical researcher, a clinician or a social scientist; where a health professional with a minimal amount of publications may have a publication record of superior quality than a biomedical researcher with numerous publications.

3.2 Read the assigned applications

Reviewers should read all of their assigned applications in details before rating any of them; and jot down notes to capture their impressions. The CIHR Fellowship Awards Reviewer Worksheet in Appendix B provides a template that they may wish to use. This worksheet is strictly for the reviewers’ own personal use and will not be filed with CIHR.

It is important to note that many candidates will likely be conducting research outside of the reviewer’s research specialty. Therefore, they should review the application with a generalist’s perspective and assess the overall quality of the research proposed by the candidate, using the appropriate adjudication criteria. However, if a reviewer feels that their level of comfort reviewing an application is unacceptably low, they should inform CIHR staff immediately and the application will be assigned to an alternate reviewer.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (e.g. publications via PubMed, etc.). They are however free to consult published lists of journal impact factors when assessing the candidate’s research accomplishments. It is important to note however that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.

Reviewers should also be alert to unconscious bias related to gender, discipline or geographic location as detailed in the CIHR Peer Review Guide for Training and Salary Awards.

3.3 Rate their assigned applications

Reviewers are then asked to rate their assigned applications against each of the adjudication criteria described in Appendix A. This funding opportunity uses an adjudication scale based on descriptors (below). It is particularly important that the full scale be used and the same convention applied to assign ratings.

Descriptor Abbreviation Definition
Outstanding O++ For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.
O+
O
Excellent E++ For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.
E+
E
Good G For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.
Fair F For this sub-criterion, the application broadly addresses all relevant aspects. Major revisions are required.
Poor P For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

ResearchNet will provide you with the application details needed for the adjudication of each criterion. Start by selecting the criterion you want to evaluate from a drop-down list. The rating scale will then appear and you can choose the letter score that best represents the application content provided. The ratings that you submit for each criterion will then be weighted automatically by CIHR in the calculation of an overall ranking. The purpose of the scale is to serve as a benchmark for peer reviewers as they rate the applications.

3.4 Provide a written assessment for each assigned application

Reviewers are asked to provide a short written assessment for each assigned applications that supports their ratings. The written reviews will serve to initiate the discussion of applications with other reviewers. They also provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training.

Comments should focus on the strengths and weaknesses of each adjudication criterion:

  • Keep it simple;
  • Use familiar descriptors that align with your rating;
  • Include justification, context and an explanation of your comments, if applicable, for each topic introduced;
  • Be clear and concise;
  • While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
  • Use objective and non-inflammatory language;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.

The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the comments and should not identify themselves in order to ensure the confidentiality of the review process.

3.5 Flag issues for CIHR’s attention

Any concerns regarding eligibility, ethics, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments or discussed through the discussion thread. For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards. Concerns may be expressed by email at Fellowships@cihr-irsc.gc.ca, or by using the Issues for CIHR Attention Form.

These issues should not be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please refer to the Grants & Awards Guide.

3.6 Submit preliminary reviews and ratings

It is important for reviewers to respect the deadline provided by submitting their reviews and scores via ResearchNet by the date specified via correspondence with CIHR staff responsible for this program. Delays in the peer review process will jeopardize CIHR’s ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that he/she may not be able to submit his/her reviews on or before the deadline, he/she must contact CIHR staff as soon as possible.

Stage 2: Virtual Discussion

Step 1: Access the other reviewers' evaluation

Once the preliminary reviews and ratings are submitted, reviewers will have access to the reviews and ratings of the other reviewers. They should compare their rating of the application with those of the other reviewers and read the other reviews to identify topic(s) of agreement and disagreement.

Step 2: Discuss application

At this stage, reviewers will be invited to discuss their assigned applications with the other reviewers assigned to the same application. The online discussion function will become accessible on a set date, or once all reviewers assigned to the application have submitted their reviews, whichever comes first. The following statuses will help them navigate the online discussion functions in ResearchNet:

Status Description
Pending The discussion cannot begin yet. Reviewer must wait for the other reviewer to submit their preliminary ratings for this application or after the set date for the discussion to start.
Start Discussion Reviewer is allowed to be the first to post a comment.
Active The discussion has begun.
Active*
(with an asterix)
A reviewer posted a new comment since the last time you accessed the discussion.
Closed All reviewers submitted their final ratings.
Suspended The online discussion is closed until further notice by CIHR.

Once a comment is posted, it cannot be deleted or edited. Comments can be posted for the attention of CIHR staff; however, it will be visible by the other reviewers. All communications with CIHR staff, including replies, should be flagged as such.

A notification email will be sent daily to advise reviewers of new posts on the online discussion board.

Step 3: Edit and confirm reviews

3.1 Edit and confirm reviews

At the end of the online discussion, reviewers may modify their reviews by changing their ratings and editing their comments. Whether or not changes are made, they are required to submit all reviews by clicking on “Submit Final Reviews”.

After this point, reviewers will no longer be able to modify their written comments.

3.2 Break ties

Once the reviews have been submitted for ranking, an initial rank order will automatically be calculated for each of the reviewer’s assigned application. Should a tie occur in the ranking, they will have to break the tie by changing the rank order position up or down of the applications. The original rank order will remain visible as a reference point.

3.3 Modify the rank list

Once all ties have been eliminated, a final rank order will be generated. Reviewers will have one last opportunity to modify the order by examining the final ranking list and determine if they are satisfied with the rank order of applications. If they’re not satisfied, they can modify the order by moving the applications up or down the rank list. Again, the original rank order will remain visible as a reference point but the other reviewers will not see the rank list.

When reviewers are satisfied, they may proceed to the next step of submitting their final reviews.

Step 4: Submit final reviews

Once reviewers are satisfied with the final ranking list of their assigned applications, they must submit their reviews and ranking to CIHR.

Feedback

An important component of the peer review process is the review of the committee’s effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR’s ongoing efforts to maintain an effective and high quality peer review system.

Since there is no face-to-face or teleconference meeting, the reviewers’ feedback should be communicated to the committee coordinator by email at Fellowships@cihr-irsc.gc.ca.

Appendix A – CIHR Fellowship Awards Adjudication Criteria

There are three adjudication criteria for the Fellowship program and they are as followed.

Criterion Notes/advice to reviewers on how to review
1. Achievements and Activities of the Candidate (60%)

a) Training Expectations

Weight in overall score for this sub-criterion: 10%

This section provides an overview of how the candidate's previous training relates to the present proposal and elaborates on career goals.

  • Assess the clarity and logic of the candidate's plans for a research career and the relevance of the proposed training.
  • Description of how the training they expect to acquire will contribute to their productivity and to the research goals they hope to achieve, and how this award will enable them to establish themselves as independent investigators.
  • Justification of why they chose the proposed training location and what they expect to learn from the training experience.
  • Justification if they are planning to hold this award in the same research environment, and/or with the same supervisor as where they completed their doctorate degree. (i.e. research institution or its affiliate).

b) Proposed Research Project

Weight in overall score for this sub-criterion: 10%

This section provides a research project summary which should be completed in collaboration with the proposed supervisor(s) and be written in general scientific language.

  • Determine if the proposed project is adequate to the candidate given their education, experience and interests. Is the project the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training?
  • Note: it is not the project per se that is being assessed. The project should be viewed as an integral part of the candidate's development as a researcher.
  • The proposed research project summary should:
    • Include the specific hypothesis of the research and describe the candidate’s role on the project;
    • Provide a concise account of the subject matter, an overview of each part of the research plan, specific project aims and the methodology;
    • Reflect the significance of the project.

c) Honours, Awards and Academic Distinction

Weight in overall score for this sub-criterion: 10%

This section provides a list of official recognitions (i.e. citations, distinctions, Honours and Prizes/Awards) received by the candidate.

  • Assess the number, importance and breadth of the candidate's official recognitions and special distinctions relative to their education, training and work experience.
  • Note the length of time required to complete academic programs and any indications of special academic distinctions received.
  • Determine relevance to research and whether the recognition is regional, national or international.

d) Publications and Related Research Achievements

Weight in overall score for this sub-criterion: 30%

This section provides a list of publications such as papers, articles, chapters or books (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights. Look for:

  • Evidence of research achievements relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., health professional career, time spent raising children, etc.).
  • For publications, observe the number of co-authors and the position of the candidate's name in the authors list (note that the importance of this position can vary depending on the discipline, etc.).
  • The candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (e.g., paper, article, chapter, book, etc.)
  • Try to get a sense of the entire body of work and its likely impact. Note the publication dates and relate them to the candidate's education and training. Consider the list of abstracts as an indication of conference presentation activities.
  • The candidate's other professional activities. Consider any patents or copyrights to which the candidate contributed.
2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities

Weight in Overall Score: 20%

In this section, three sponsors provide an assessment of the candidate. These assessments should come from individuals under whom the candidate has trained, who are familiar with the candidate’s characteristics and abilities and/or who have had a good opportunity to assess their potential for research. Keep in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors. With the Sponsor Assessment Form:

  • Look for evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with research career achievement.
  • Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not.
  • Read the supporting text carefully, taking note of the extent to which the sponsors justify their scores.
  • Look particularly for indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent.
  • Indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.
  • If the candidate has had an opportunity to conduct research. Look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.
  • Consider the following:
    • Do the detailed comments support the ratings outlined on the first page of the assessment?
    • How long has the sponsor known the candidate?
    • What is the relationship of the sponsor to the candidate?
3. Research Training Environment

Weight in Overall Score: 20%

This section describes elements of the research environment that will contribute directly or indirectly to the quality of the candidate's research training experience that are available. It should demonstrate the commitment of the proposed supervisor(s) and their institution to support the development of the candidate's research project (funding, facilities, equipment, etc.) and professional development.

  • Look at the supervisor(s) research experience, qualifications, honours and awards. Examine their publication record to get a sense of productivity, impact and collaboration taking into consideration the different disciplines and their impacts on these.
  • Determine if the research environment, including space, facilities, and personnel support available is appropriate.
  • Get a sense of the resources available and the overall level of activity by reviewing the information on grants currently held, noting the extent to which the supervisor(s) was either listed as a principal or co-applicant for the funds.
  • Review the supervisor(s) training record. Note for each person listed the level of training, length of time with the supervisor(s), degree received (if applicable) and current position.
  • Your assessment should take into consideration the career stage and discipline of the supervisor(s). Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.

Appendix B – CIHR Fellowship Awards Reviewer Worksheet

The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.

Applicant Name: ___________________________________ Application #: __________________

Criterion Rating
(O++, O+, O, E++, E+, E, G, F, P)
Reviewers Comments
1. Achievements & Activities of the Candidate (60%) a) Training Expectations (10%) Strengths:
Weaknesses:
b) Proposed Research Project (10%)
c) Honours, Awards and Academic Distinction (10%)
d) Publications and Related Research Achievements (30%)
2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities (20%) Strengths:
Weaknesses:
3. Research Training Environment (20%) Strengths:
Weaknesses: