CIHR Reviewers’ Guide for Doctoral Research Awards

Table of Contents

Introduction

On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide instructions on the peer review process specific to the CIHR Doctoral Research Awards (DRA) program.

Peer Review at CIHR

Information on CIHR’s objectives, governance and policies; an outline of the roles and responsibilities of peer review committee members; and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting the reviews.

Summary of the Peer Review Process

The CIHR DRA program uses an individual structured review process using the online ResearchNet platform. The review process is completed in one stage: an individual review and rating of an assigned set of applications (there is no committee meeting). All eligible applications received will be assigned to two (2) reviewers.

There are two (2) peer review committees for this program. Each application received will be assigned to the committee with the mandate that most closely aligns with the applicant’s theme of the proposed research activities. A link to the committees’ mandates is listed under the section “Read the pertinent documentation”.

Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:

Step 1: Read the pertinent documentation
Step 2: Identify conflicts of interests
Step 3: Conduct in-depth review of assigned applications
Step 4: Submit reviews and ratings on ResearchNet
Step 5: Be prepared for a re-review

Step-by-Step Instructions

Step 1: Read the pertinent documentation

The peer review process for this program is described in details in this document. It is essential to read the document and be familiar with it. It is also important to read the following:

Step 2: Identify conflicts of interests

To identify conflicts of interests, reviewers are to follow these steps:

  • Log into ResearchNet.
  • On the home page, click on the link of their assigned committee to open the main task list.
  • Complete the task Review Confidentiality and Conflict of Interest Guidelines (once completed, it will “open” the other tasks).
  • Open the Manage Conflicts task.
  • For each of the assigned application, use the information provided to indicate if they are able to review or if there is a conflict. If there is a conflict, CIHR will reassign the application to another reviewer.

Step 3: Conduct in-depth review of assigned applications

Once conflicts have been identified, the full content of the remaining assigned applications will be available under the task “Conduct Reviews”. Reviewers should then follow the steps below.

3.1 Review the adjudication criteria

Reviewers should first become familiar with the adjudication criteria for this funding opportunity. They can be found at the end of this document in Appendix A. This appendix identifies which elements of the application to review for each criterion.

3.2 Read the assigned applications

Reviewers should read all of their assigned applications in details before rating any of them; and jot down notes to capture their impressions. The CIHR Doctoral Research Awards Reviewer Worksheet in Appendix B provides a template that they may wish to use. This worksheet is strictly for the reviewers’ own personal use and will not be filed with CIHR.

It is important to note that many candidates will likely be conducting research outside of the reviewer’s research specialty. Therefore, they should review the application with a generalist’s perspective and assess the overall quality of the research proposed by the candidate, using the appropriate adjudication criteria.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research on the candidate, the supervisor or the proposed research institution (e.g. publications via PubMed, etc.).

3.3 Unconscious bias

As reviewers read the applications, they should be alert to unconscious bias related to gender, cultural (including Indigenous and geographic bias), age, language (including official language and minority communities bias) and institution and remember that:

  • Career interruptions for child bearing and raising can influence opportunity for knowledge production, publications and related variables;
  • Different disciplines and environments offer different opportunities for research contributions, publication and other research related activities;
  • The reputation of institutions should not affect the reviewer’s view of applicants or their research training environment;
  • It is important to take steps to mitigate bias in reviewers’ thought process about difference in culture (e.g. Indigenous Peoples of Canada); and,
  • A point should be made of respecting Canada’s linguistic duality by recognizing the value of research in French and the value of research on Francophone minority communities.

To learn more about unconscious bias, reviewers are encouraged to complete the related learning module.

3.4 Rate the assigned applications

Reviewers are then asked to rate their assigned applications against each of the adjudication criteria described in Appendix A, using CIHR’s traditional rating scale (below). It is particularly important that the full scale be used.

Descriptor Range Outcome

Outstanding

4.5 – 4.9 May be Funded
Excellent 4.0 – 4.4
Very Good 3.5 – 3.9
Good 3.0 – 3.4 Not Fundable
Average 2.0 – 2.9
Below Average 1.0 – 1.9
Not Acceptable 0.0 – 0.9

Only applications rated 3.5 or higher are eligible for CIHR funding. Therefore, applications rated below 3.5 are not eligible for CIHR funds, including those from partnerships/priority announcement programs.

3.5 Provide a written assessment for each assigned application

Reviewers are asked to provide a short written assessment for each assigned application that supports their ratings. The written reviews will provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training.

Comments should highlight the strengths and weaknesses of each adjudication criterion:

  • Keep it simple;
  • Use familiar descriptors that align with your rating;
  • Include justification, context and an explanation of your comments, if applicable, for each topic introduced;
  • Be clear and concise;
  • While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
  • Use objective and non-inflammatory language;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.

The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the comments and should not identify themselves in order to ensure the confidentiality of the review process.

3.6 Flag issues for CIHR’s attention

Any concerns regarding eligibility, ethics, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments. For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards. Concerns may be expressed by email at PeerReview.Recruitment@cihr-irsc.gc.ca (note that this email is strictly meant for Peer Review activities).

These issues should not be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, reviewers should refer to the Grants & Awards Guide.

Step 4: Submit reviews and ratings on ResearchNet

As the reviewers perform their evaluation, the reviews can be saved as drafts by selecting “Save draft copy” on ResearchNet. This allows them to make changes at a later time. However, in order to submit the reviews and ratings to CIHR, they must select “Submit Final Review”. Afterwards, they will no longer be able to modify them.

It is important for reviewers to respect the deadline provided by submitting the reviews and scores in ResearchNet by the date specified via correspondence with CIHR staff responsible for this program. Delays in the peer review process will jeopardize CIHR’s ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that he/she may not be able to submit his/her reviews on or before the deadline, he/she must contact CIHR staff as soon as possible.

Step 5: Be prepared for a re-review

Once all the scores are submitted, CIHR will perform a discrepancy review by calculating the final rating for each application. CIHR will then identify applications which are at risk of an unfair decision because of a wide spread between the two reviewers’ scores. In such cases, CIHR will ask both reviewers to reconsider their initial assessment and resubmit scores. To do so, they will be asked to get in contact with each other to discuss the application. This second review usually reduces the gap between scores to an acceptable size. If the discrepancy persists, CIHR will obtain a third review.

For this purpose, it is recommended that reviewers keep their working notes on file until few weeks after the competition results have been announced.

Feedback

An important component of the peer review process is the review of the committee’s effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR’s ongoing efforts to maintain an effective and high quality peer review system.

Since there is no face-to-face or teleconference meeting, the reviewers’ feedback should be communicated to the committee coordinator by email at PeerReview.Recruitment@cihr-irsc.gc.ca (note that this email is strictly meant for Peer Review activities).

Appendix A – CIHR Doctoral Research Awards Adjudication criteria

The following criteria are to be used for evaluating Doctoral Research Award applications:

Criterion

Information Source

Notes/advice for reviewers on how to review

  1. Achievements and Activities of the Candidate (weight in overall score: 35%)
  1. Publication Activity

Common CV completed by the candidate

“Publication List” attachment, if provided by the candidate

Weight in overall score for this sub-criterion: 10%

  • Review the list of articles and other publications produced by the candidate. Consider presentations in the assessment of other research activity (see below).
  • Assess the publication activity of the candidate relative to your expectations of someone with their academic experience.
  • Consider: breadth of science covered and the frequency of publication
  • In considering the candidate's contribution to the publications, take into account the number of co-authors for each paper and the prominence of the candidate's name on the list of authors.
  • Bear in mind that publication activity patterns vary among health science disciplines.
  1. Other Research Activity

Common CV completed by the candidate

Weight in overall score for this sub-criterion: 10%

  • Review information on presentations, research prizes and other indicators of the candidate's research productivity.
  • Assess other research activity of the candidate relative to your expectations of someone with their academic experience.
  • Consider: breadth of science covered, size and importance of meetings involved, frequency of conference presentations and research honours or awards.
  1. Academic Performance

Candidate’s academic transcripts

Weight in overall score for this sub-criterion: 15%

  • Review undergraduate academic transcripts and, if available, graduate transcripts.
  • Consider:
    • Type of program and courses pursued
    • Course load
    • Grades obtained
    • Relative standing (if available)
    • Overall average
    • Trend (give credit for a steadily improving or consistently good performance)
  1. Characteristics and Abilities of the Candidate
  • Critical thinking
  • Independence
  • Perseverance
  • Originality
  • Organizational Skills
  • Interest in discovery
  • Research Ability
  • Leadership

Sponsor’s Assessments

Weight in overall score for this sub-criterion: 40%

  • Keep in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors.
  • Examine the assessments, recognizing that positive comments are common while negative ones are not.
  • Provide a score based on your overall impression. Consider:
    • the extent to which the box scores and narratives are consistent.
    • whether forms are from sponsors under whom the candidate has trained and/or who have had a good opportunity to assess their potential for research.
    • the length of time that the sponsor has known the candidate.
    • the relationship of the sponsor to the candidate.
  1. Research Training Environment (weight in overall score: 25%)
  1. Training program for the candidate

Application Details (including Training Expectation and Research Proposal Summary) and Space, Facilities and Personnel Support completed by the candidate and/or supervisor(s)

Weight in overall score for this sub-criterion: 10%

  • Review the candidate's training expectations and proposed doctoral research program, including project, resources available and planned non-research activities.
  • Most candidates will be conducting research outside your research specialty. From a non-specialist's perspective, assess the intellectual challenge and excitement of the research in which the candidate will be involved.
  • Consider the extent to which the training program appears to fit with the candidate's training expectations and the resources available for the candidate's project.
  1. Scientific Activity

Common CV completed by the research supervisor(s)

Weight in overall score for this sub-criterion: 5%

  • Review the research supervisor's publication record, significant contributions to research, and honours or awards.
  • In assessing whether the scientific environment is one in which a doctoral student will be inspired and challenged, focus on the scientific productivity and impact of the research supervisor(s) as indicated by publications, presentations and research prizes or awards.
  • Bear in mind that publication activity patterns vary among health science disciplines.
  1. Research Resources

Common CV completed by the research supervisor(s)

Weight in overall score for this sub-criterion: 5%

  • Review the research resources available in the laboratory in which the candidate will train.
  • In assessing the extent to which the level of research resources in the training environment will enhance the scientific development of the candidate, focus on the adequacy of peer-reviewed research funding secured by the research supervisor.
  • Bear in mind that availability of funding varies among health science disciplines.
  1. Training Record

Common CV completed by the research supervisor(s)

Weight in overall score for this sub-criterion: 5%

  • Review the supervisory experience of the researcher under whom the candidate proposes to pursue doctoral studies.
  • Consider the number of master's students, doctoral students and postdoctoral fellows that the research supervisor has trained or is currently training.
  • In assessing the likelihood that students will be inspired to continue on in research, consider any information on the subsequent positions of persons who trained with the research supervisor.
  • Bear in mind that the opportunity to train students will vary with the length of time that has elapsed since the supervisor(s) completed his/her own research training.

Appendix B – CIHR Doctoral Research Awards Reviewer Worksheet

The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.

Applicant Name: ___________________________________________ Application #:__________________

Criterion and Weight

Notes for Reviewers

Reviewer’s Comments

  1. Achievements and Activities of the Candidate
  1. Publication Activity (10%)
  • List of articles and other publications produced by the candidate.
  • Publication activity of the candidate relative to your expectations of someone with their academic experience.
  • Breadth of science covered and the frequency of publication.
  • Number of co-authors for each paper and the prominence of the candidate's name on the list of authors.

Strengths:

Weaknesses:

  1. Other Research Activity (10%)
  • Presentations, research prizes and other indicators of the candidate's research productivity.
  • Other research activity of the candidate relative to your expectations of someone with their academic experience.
  • Breadth of science covered, size and importance of meetings involved, frequency of conference presentations and research honours or awards.

Strengths:

Weaknesses:

  1. Academic Performance (15%)
  • Undergraduate academic transcripts and, if available, graduate transcripts.
  • Consider: type of program and courses pursued, course load, grades obtained, relative standing (if available), overall average, trend (give credit for a steadily improving or consistently good performance).

Strengths:

Weaknesses:

  1. Characteristics and Abilities of the Candidate (40%)

Examples: critical thinking, independence, perseverance, originality, organizational skills, interest in discovery, research ability, leadership.

  • Keep in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors.
  • Provide a score based on your overall impression.
  • Assess to which extent the box scores and narratives are consistent, recognizing that positive comments are common while negative ones are not.
  • Assess if forms are from sponsors under whom the candidate has trained and/or who have had a good opportunity to assess their potential for research.
  • Assess the length of time that the sponsor has known the candidate.
  • Assess the relationship of the sponsor to the candidate.

Strengths:

Weaknesses:

  1. Research Training Environment
  1. Training program for the candidate (10%)
  • Training expectations and proposed doctoral research program, including project, resources available and planned non-research activities.
  • Intellectual challenge and excitement of the research in which the candidate will be involved.
  • Training program fits with candidate's training expectations and the resources available for the candidate's project.

Strengths:

Weaknesses:

  1. Scientific Activity (5%)
  • Research supervisor's publication record, significant contributions to research, and honours or awards.
  • Scientific productivity and impact of the research supervisor(s) as indicated by publications, presentations and research prizes or awards.
  • Bear in mind that publication activity patterns vary among health science disciplines.

Strengths:

Weaknesses:

  1. Research Resources (5%)
  • Research resources available in the laboratory in which the candidate will train.
  • Adequacy of peer-reviewed research funding secured by the research supervisor.
  • Bear in mind that availability of funding varies among health science disciplines.

Strengths:

Weaknesses:

  1. Training Record (5%)
  • Research supervisor’s experience as a supervisor.
  • Number of master's students, doctoral students and postdoctoral fellows that the research supervisor has trained or is currently training.
  • Information on the subsequent positions of persons who trained with the research supervisor.
  • Bear in mind that the opportunity to train students will vary with the length of time that has elapsed since the supervisor(s) completed his/her own research training.

Strengths:

Weaknesses:

Score:

Date modified: