Guide for Reviewers - CIHR Fellowships

Contents

  1. What’s New?
  2. Introduction
  3. Peer Review Principles
  4. Step by Step Instructions
  5. Step 1: Read this Guide and the funding opportunity
  6. Step 2: Review Your Assigned Applications
    1. Become knowledgeable of the Adjudication Criteria
      1. Achievements and Activities of the Candidate
      2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities
      3. Research Training Environment
    2. Read the Applications
    3. Rate the Application
    4. Write Comments Supporting Your Rating
    5. Flag Issues for CIHR's Attention
  7. Step 3: Submit Preliminary Ratings and Reviews
  8. Step 4: Discuss the Application with the Other Reviewers
    1. Accessing Other Reviewers Reviews and Online Discussion
    2. Using the Online Discussion
  9. Step 5: Confirm your reviews by submitting Ready for ranking
  10. Step 6: Break Ties
  11. Step 7: Modify the Rank List
  12. Step 8: Submit Final Reviews to CIHR

Annex 1: Adjudication Criteria

Annex 2: Adjudication Criteria Interpretation

  1. Achievements and Activities of the Candidate
    1. Training Expectations
    2. Proposed Research Project
    3. Honours, Awards and Academic Distinction
    4. Publications and Related Research Achievements
  2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities
  3. Research Training Environment

1) What’s New?

As part of CIHR’s continuous commitment to improve peer review, and in the context of the Health Research Roadmap, CIHR is enhancing the peer review process for the Fellowship program. The enhancements include:

  1. On-line Interactive Learning Lessons are available which include training on components of the review process;
  2. Increasing the number of reviewers per application to three;
  3. A new rating scale using letters (A, B, C, D, E) instead of numbers;
  4. Reviewers will comment on the strengths & weaknesses and provide a letter rating for each criterion;
  5. Submitting preliminary assessments to enable the discussion;
  6. Discussing applications via asynchronous online discussion;
  7. Opening the discussion on a set date or once everybody submits their preliminary reviews;
  8. Breaking ties and modifying application rank prior to submitting final reviews to CIHR;
  9. Eliminating the re-review and third review process;
  10. Eliminating the calibration exercise and the benchmark application;
  11. Elimination of criteria weight variation between candidates with and without a PhD.

2) Introduction

On behalf of CIHR, we would like to thank you for agreeing to serve as a reviewer. The success of the peer review process is made possible by dedicated people like yourself who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The peer review process is described in detail in this Guide for Reviewers and in the Peer Review Manual for Salary & Training Award Applications.

The purpose of these documents is to provide information on CIHR’s objectives, governance and policies; to outline the roles and responsibilities of reviewers; and to define the policies and procedures for peer review of applications.

For detailed regulations concerning all aspects of CIHR funding programs, please see the Grants and Awards Guide.

3) Peer Review Principles

  • Avoid conflict of interest;
  • Respect the confidentiality of applications;
  • Respect deadlines.

4) Step by Step Instructions

Step 1: Read this Guide and the funding opportunity

Step 2: Review your assigned applications

  1. Become knowledgeable of the adjudication criteria
  2. Read the applications
  3. Rate the applications
  4. Write strengths & weaknesses to support your ratings
  5. Flag issues for CIHR's attention

Step 3: Submit preliminary reviews (ratings and comments)

Step 4: Discuss the applications with the other reviewers

Step 5: Confirm your reviews by submitting Ready for ranking

Step 6: Break ties

Step 7: Modify your rank list

Step 8: Submit final reviews to CIHR

5) Step 1: Read this Guide and the funding opportunity

It is essential that reviewers read and be familiar with this document and the objectives of the funding opportunity.

6) Step 2: Review Your Assigned Applications

A) Become knowledgeable of the Adjudication Criteria

There are three adjudication criteria for the Fellowship program:

  1. Achievements and Activities of the Candidate
  2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities
  3. Research Training Environment

It is important to note that expectations may differ by research area/discipline. For example, a health professional with minimal publications may have a superior publication record than a biomedical researcher with numerous publications. (e.g., publication productivity can vary while comparing a biomedical researcher, a clinician or a social scientist).

1. Achievements and Activities of the Candidate

The Achievements and Activities of the Candidate criteria include these four sub-criteria:

  1. Training Expectations
  2. Proposed Research Project
  3. Honours, Awards and Academic Distinction
  4. Publications and Related Research Achievements

Reviewers should consider the following in their assessment of the achievements and activities of the candidate:

  • Assess the clarity and logic in the explanation of the candidate's plans for a research career and the relevance of the proposed training.
  • Determine if the proposed project is adequate to the candidate given their education, experience and interests. Is the project the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training? Note: it is not the project per se that is being assessed. It is the project as an integral part of the candidate's development as a researcher.
  • Assess the number, importance and breadth of the candidate's special distinctions relative to their education, training and work experience. Also determine if the distinction is relevant to their research and if the recognition is regional, national or international.
  • Note the length of time required to complete academic programs and any indications of special academic distinction.
  • Look for evidence of achievements in research relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., time spent raising children).
  • For publications, observe the number of co-authors and the position of the candidate's name in the authors list. Note that the importance of this position can vary depending on the discipline, etc. Also note the candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (article, chapter, book, etc.).

2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities

To review the Sponsor’s Assessment of the Candidate’s characteristics and abilities, we ask you to take a look at the Sponsor Assessment forms. They should offer a perspective on the candidate from individuals who are familiar with their characteristics and abilities. Recognize that positive comments are common while negative ones are not. Read the supporting text carefully, taking note of the extent to which they justify the ratings. Keep in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors.

To perform an accurate review of these assessments, the following should be considered:

  • Do the detailed comments support the ratings outlined on the first page of the assessment?
  • How long has the sponsor known the candidate?
  • What is the relationship of the sponsor to the candidate?

3. Research Training Environment

Finally, in your review of the Research Training Environment, consider:

  • Information on the supervisors’ research experience, qualifications, honours and awards.
  • The supervisors’ publication record to get a sense of productivity, impact and collaboration (consider the different disciplines and their impacts on these).
  • If the research environment, including space, facilities, and personnel support available is adequate. Review the information on grants currently held, noting the extent to which the supervisors were either listed as a principal or co-applicant for the funds. Get a sense of the resources available and the overall level of activity.

Review the supervisors’ training record. Note for each listed trainee: the level of training, length of time with the supervisors, degree received (if applicable) and current position. Your assessment should take into consideration the career stage and discipline of the supervisors.

B) Read the Applications

Read all of your assigned applications before rating any of them. It is important to note that many candidates will likely be conducting research outside of your research specialty. We ask that you provide an overall assessment of the quality of the application. However, if you feel that your level of comfort of reviewing an application is unacceptably low, you should inform CIHR staff and they will assign the application to an alternate reviewer.

As you read the applications, be alert to unconscious bias related to gender, discipline or geographic location. Remember that:

  • Career interruptions for child bearing and raising can influence opportunity for knowledge production, publications and related variables;
  • Each discipline and environment offer different opportunities for research contributions, publication and other research related activities; and,
  • The reputation of institutions should not affect your view of applicants or their research training environment.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (i.e. publications via PubMed, etc.). You are however free to consult published lists of journal impact factors when assessing the candidate's research accomplishments. Note that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.

C) Rate the Application

To rate the application, you will use the new rating scale, which consists of A, B, C, D, and E, with A being the highest possible rating. The ResearchNet reviewer worksheet will provide you with the application details needed for the adjudication of each criterion.

Start by selecting the criteria you want to evaluate from a drop down list. The rating scale will appear and you will choose the letter score that best represents the application content provided. Each criterion is described in details in Annex 1. The ratings that you submit for each criterion, on the A to E scale, will then be weighted automatically by CIHR in the calculation of an overall ranking. The purpose of the scale is to serve as a benchmark for peer reviewers as they rank the applications.

Rating Descriptor
A For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal. Its numerical value is 20.
B For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible. Its numerical value is 16.
C For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary. Its numerical value is 12.
D For this sub-criterion, the application broadly addresses all relevant aspects. Considerable improvements are required. Its numerical value is 8.
E For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps. Its numerical value is 4.

D) Write Comments Supporting Your Rating

Write brief comments highlighting the strengths and weaknesses that support your rating of each adjudication criteria. The written reviews will serve to initiate the discussion of applications with other reviewers. They also provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training.

  • Keep it simple;
  • Use familiar descriptors that align with your rating;
  • Provide comments on the strengths and weaknesses only for each adjudication criteria of the application;
  • Include justification, context and an explanation of your comments, if applicable, for each topic you introduce;
  • Be clear and concise, while expressing complete thoughts;
  • Use objective and non-inflammatory language;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant, or inappropriate in any way.

E) Flag Issues for CIHR's Attention

Any concerns in the following areas should not be noted in the written comments or discussed through the discussion thread but must be brought to CIHR's attention for follow-up:

  • Eligibility
  • Responsible Conduct of Research
  • Ethics
  • Human pluripotent stem cell research

Please report these concerns to CIHR staff by email at Fellowships@cihr-irsc.gc.ca. Questions or interpretations of CIHR policy, as well as allegations of responsible conduct of research, including misrepresentation of information in an application, should not affect the review of an application and must not appear in any written assessment of an application. For detailed regulations concerning these issues, please see the Grants and Awards Guide.

7) Step 3: Submit Preliminary Ratings and Reviews

Submit your reviews once you have reviewed, rated and written comments for every application assigned to you.

8) Step 4: Discuss the Application with the Other Reviewers

A) Accessing Other Reviewers Reviews and Online Discussion

Once your preliminary ratings and reviews are submitted, you will have access to the submitted ratings and written reviews from other reviewers. Read their reviews and identify topic of agreement and disagreement. Also compare your rating of the application with those of the other reviewers.

The online discussion function will only be available on a set date, or once all reviewers assigned to an application have submitted their preliminary ratings and written comments.

B) Using the Online Discussion

The online discussion function will become accessible on a set date, or once all reviewers assigned to the application have submitted their reviews, whichever comes first. The following statuses will help you navigate the online discussion function in ResearchNet:

  1. Pending: You cannot begin the discussion. You must wait for the other reviewers to submit their preliminary ratings for this application or after set date for discussion to start.
  2. Start Discussion: Allows you to be the first to post a comment.
  3. Active: The discussion has begun.
  4. Active with star symbol (*): A reviewer posted a new comment since the last time you accessed the discussion.
  5. Closed: All reviewers submitted their final ratings.
  6. Suspended: The online discussion is closed until further notice by CIHR.

Once a comment is posted, you cannot delete or edit this comment. Comments can be posted for the attention of CIHR staff; however it will be visible by the other reviewers. All communication with CIHR staff, including replies, should be flagged as such.

A notification email will be sent daily to advise reviewers of new posts on the online discussion board.

9) Step 5: Confirm your reviews by submitting Ready for ranking

At the end of the online discussion, you may modify your reviews by changing your rating and editing your comments. Reviewers are required to submit all applications for Ready for Ranking whether or not the review was modified following the online discussion period.

10) Step 6: Break Ties

Reviewers must break ties in the rank order of applications assigned to them. Ties can be eliminated by changing the application rank order position up or down as the original rank order will remain visible as a reference point.

11) Step 7: Modify the Rank List

Once all ties have been eliminated, reviewers may modify the rank order of applications by moving the applications up or down the rank list. The original rank order will remain visible as a reference point. Note that the other reviewers will not see your rank list.

12) Step 8: Submit Final Reviews to CIHR

Once satisfied with the applications' rank order, the reviewer confirms the ranking by submitting the reviews and ranking to CIHR.

Annex 1: Adjudication Criteria

Overview of the six adjudication criteria

The alpha ratings that you submit for each criterion will be weighted automatically by ResearchNet in the calculation of an overall rating and as such, the weighting of each criterion should not influence your rating.

Criteria Weight Overall Weight
Total 100%
1. Achievements and Activities of the Candidate 60%
a) Training Expectations 10%
b) Proposed Research Project 10%
c) Honours, Awards and Academic Distinction 10%
d) Publications and Related Research Achievements 30%
2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities 20%
3. Research Training Environment 20%

Annex 2: Adjudication Criteria Interpretation

1. Achievements and Activities of the Candidate

a) Training Expectations

Working Definition

This section provides an overview of how the candidate's previous training relates to the present proposal and elaborates on career goals.

What to Look For
  • Clarity and logic of the candidate's plans for a research career and the relevance of the proposed training;
  • Description of how the training they expect to acquire will contribute to their productivity and to the research goals they hope to achieve, and how this award will enable them to establish themselves as independent investigators;
  • Justification of why they chose the proposed training location and what they expect to learn from the training experience;
  • Justification if they are planning to hold this award in the same research environment, and/or with the same supervisor as where they completed their doctorate degree. (i.e. research institution or its affiliate).

b) Proposed Research Project

Working Definition

This section provides a research project summary which should be completed in collaboration with the proposed supervisor(s) and be written in general scientific language.

What to Look For
  • The ideal research project is one that is best for the candidate given their education, experience and interests. It is the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training.
  • Bear in mind that it is not the project per se that is being assessed. Instead, the project should be viewed as an integral part of the candidate's development as a researcher.
  • The proposed research project summary should:
    • Include the specific hypothesis of the research and describe the candidate's role on the project;
    • Provide a concise account of the subject matter, an overview of each part of the research plan, specific project aims and the methodology;
    • Reflect the significance of the project.

c) Honours, Awards and Academic Distinction

Working Definition

This section provides a list of official recognitions (i.e. citations, distinctions, Honours and Prizes/Awards) received by the candidate.

What to Look For

Assess the number, importance and breadth of the candidate's official recognitions relative to their education, training and work experience. Note the length of time required to complete academic programs and any indications of special academic distinctions received. Finally, note relevance to research and whether the recognition is regional, national or international.

d) Publications and Related Research Achievements

Working Definition

This section provides a list of publications such as papers, articles, chapters or books (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights.

What to Look For
  • Evidence of research achievements relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., health professional career, time spent raising children, etc.).
  • For publications, observe the number of co-authors and the position of the candidate's name in the authors list (note that the importance of this position can vary depending on the discipline, etc.).
  • The candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (e.g., paper, article, chapter, book, etc.)
  • Try to get a sense of the entire body of work and its likely impact. Note the publication dates and relate them to the candidate's education and training. Consider the list of abstracts as an indication of conference presentation activities.
  • The candidate's other professional activities. Consider any patents or copyrights to which the candidate contributed.

2. Sponsor’s Assessment of the Candidate’s Characteristics and Abilities

Working Definition

In this section, three sponsors provide an assessment of the candidate. These assessments should come from people under whom the candidate has trained and/or who have had a good opportunity to assess their potential for research. Keep in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors.

What to Look For

  • Evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with research career achievement.
  • Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not.
  • Read the supporting text carefully, taking note of the extent to which the sponsors justify their scores.
  • Look particularly for indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent.
  • Indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.
  • If the candidate has had an opportunity to conduct research. Look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.

3. Research Training Environment

Working Definition

This section describes elements of the research environment that will contribute directly or indirectly to the quality of the candidate's research training experience that are available. It should demonstrate the commitment of the proposed supervisor(s) and their institution to support the development of the candidate's research project (funding, facilities, equipment, etc.) and professional development.

What to Look For

  • The supervisor(s) research experience, qualifications, honours and awards. Examine their publication record to get a sense of productivity, impact and collaboration taking into consideration the different disciplines and their impacts on these.
  • Determine if the research environment, including space, facilities, and personnel support available is appropriate.
  • Get a sense of the resources available and the overall level of activity by reviewing the information on grants currently held, noting the extent to which the supervisor(s) was either listed as a principal or co-applicant for the funds.
  • Review their training record. Note for each person listed the level of training, length of time with the supervisor(s), degree received (if applicable) and current position.
  • Your assessment should take into consideration the career stage and discipline of the supervisor(s). Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.