Guide for Reviewers - CIHR Fellowships

Table of Contents

Responsibilities of Reviewers
Confidentiality
Avoid Conflict of Interest
Respect the Confidentiality of Applications
Be Aware of the Reviewer Community's Scoring of the Benchmark Application

Reviewing the Assigned Applications
Read the Applications
Rate the Applications
Issues for CIHR's Attention
Feedback for the Applicant
Send Review Results to CIHR via ResearchNet
Be Prepared for a Re-Review Request from CIHR

Annex 1: Benchmark Application
Annex 2: Criteria and Rating Scales
Annex 3: Calibrating Scores
Annex 4: Reviewer Worksheet
Annex 5: Guide to Locating Information
Annex 6: Issues to be Reported for CIHR's Attention

Responsibilities of Reviewers

• Avoid conflict of interest;
• Respect the confidentiality of applications;
• Be aware of the reviewer community's scoring of the benchmark application;
• Review and rate each assigned application using the criteria provided by CIHR;
• Submit ratings and feedback to CIHR via ResearchNet by the deadline specified;
• Provide a re-review if requested by CIHR.

Confidentiality

Confidentiality is the obligation not to disclose directly or indirectly to anyone else, information about a person without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. Thus, information contained in applications, reviewer reports, names of reviewers and committee discussions are all strictly confidential. The use of this information for any other purpose than what is outlined here is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.

Committee members must not discuss with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the chances of success or failure. Applicants must not contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank within committee, etc.). All requests for information on an application or a reviewer report should be referred to the Deputy Director at CIHR responsible for the committee in question.

By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision and CIHR will not edit the reviews provided. The identity of the reviewers will not be revealed to the applicants under any circumstances. However, a list of peer review committee members will be published on the CIHR website 60 days after the Scientific Council approves funding for a competition cycle. Please refer to CIHR's policy on Conflict of Interest and Confidentiality in the context of Merit, Relevance and Peer Review (CCIP).

Avoid Conflict of Interest

• As soon as you receive the applications to be reviewed, look over the candidates' names, the names of their proposed supervisors and their institutions, and indicate using ResearchNet your ability to review each application that has been assigned to you.

You must not be involved in the review if the applicant or the proposed research supervisor:

• Is from your institution (unless they are located in another campus);
• Has collaborated with you within the last five years;
• Has been supervised by you within the last ten years;
• Is a close personal friend or relative;
• Has major differences of opinion with you;
• If you could be affected financially from the outcome of the application; or
• If you are for some other reason unable to provide an objective review.

If you would be in conflict of interest, or might be perceived to be in conflict of interest, please notify CIHR immediately and the application will be assigned to another reviewer.

All committee members are subject to the same conflict of interest guidelines. CIHR staff and the Chair, if applicable, are responsible for resolving areas of uncertainty.

All committee members must read and agree to abide by the CCIP policy prior to viewing any application information.

Respect the Confidentiality of Applications

Do not forward copies of applications or discuss the confidential information within them with others.

Be Aware of the Reviewer Community's Scoring of the Benchmark Application

Step 1: Rate the Benchmark Application

A benchmark application appears in Annex 1. If you have already rated it, go to Step 2. If not, read then rate it using the six criteria and the scales provided in Annex 2.

Step 2: Compare Your Ratings to the Pattern for the Reviewer Community

View your ratings in relation to those of the relevant research community by completing the table in Annex 3. In cases where your rating falls outside the normal range, revisit your assessment and consider the reasons given by the reviewer community for their ratings.

Please note: Although the Benchmark application (including all elements of Annexes 1-3) is intended to be a generic tool to guide reviews, given the wide variety of applications that are submitted to the CIHR Fellowships program, some elements may not be directly/equally applicable in all research areas/themes/disciplines (for example, expectations regarding publications, etc.). CIHR is currently reviewing this Guide for Reviewers – Fellowships with a view to making further adjustments that will provide more guidance to better reflect the variety of applications received. In the meantime, we ask that reviewers (particularly those focused on health services, health policy and/or population and public health research in the FAH, HTA and HTB committees) take the above into consideration when conducting their reviews and base such reviews on their expertise, experience and understanding of health research and training in these areas.

Reviewing the Assigned Applications

Read the Applications

Read all of your assigned applications before rating any of them. It is important to note that many candidates will likely be conducting research outside of your research specialty. From a non-specialist's perspective, assess the quality of the research proposed by the candidate. However, should you feel that your level of comfort of reviewing an application is unacceptably low, you may identify this specific application as a "conflict" and CIHR staff will re-assign to an alternate member.

As you examine each application, jot down notes to capture your impressions. The Reviewer Worksheet (Annex 4) provides a template that you could use. Note that the worksheet will not be filed with CIHR.

Rate the Applications

Examine each application in detail and rate it against each of the six criteria described in Annex 2.

Reviews for Fellowship applications are submitted to CIHR via ResearchNet. The electronic rating forms are available to reviewers once you log on to ResearchNet.

Issues for CIHR's Attention:

Any concerns in the following areas should be brought to CIHR's attention for follow-up (please use the form described in Annex 6). These issues are not to be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. For detailed regulations concerning these issues, please see the Grants and Awards Guide.

• Eligibility
• Ethics
• Human pluripotent stem cell research
• Budget justification
• Section 56 of the federal Controlled Drugs and Substances Act

Feedback for the Applicant

Prepare brief comments on the application for transmittal to the candidate by CIHR via ResearchNet after the competition:

• Keep it simple;
• Carefully avoid language that might be construed as sarcastic, flippant, arrogant, or inappropriate in any way;
• Cover both strengths and weaknesses, but in a constructive way to help the candidate prepare a stronger application for the future (keep in mind that there can be sensitivity to tone, etc.).

Note: Although comments are not mandatory, they are highly recommended as they help to personalize the assessment and are appreciated by candidates (although caution should be used if you re-use similar comments, e.g., ensure consistent use of he/she, etc.).

Send Review Results to CIHR via ResearchNet

CIHR makes every effort to allow an adequate period of time for reviewers to assess the applications assigned to them. Please respect the deadline provided by CIHR by submitting your reviews via ResearchNet by the date specified in ResearchNet (and in correspondence from CIHR staff responsible for the Fellowship program), as delays in the Peer Review process jeopardize CIHR's ability to release decisions to applicants by the dates specified in the Funding Opportunity. If at any point in the process you determine that you may not be able to submit your reviews on or before the deadline, please contact CIHR staff as soon as possible.

Be Prepared for a Re-Review Request from CIHR

When all initial scores are received, CIHR will calculate an average for each applicant based on both reviewers weighted scores and not on each specific criterion. CIHR will then identify applications which are at risk of an unfair decision because of a wide spread between the two reviewers' ratings (the "acceptable" range is determined based on the level of variation of all scores received). In cases where narrowing the gap between the two reviewers' ratings may affect the funding decision, CIHR will ask both reviewers to reconsider their initial assessment and resubmit scores. Usually this second review will reduce the gap between scores to an acceptable range. If it does not, CIHR will obtain a third review.

Please note: A second review, to narrow the gap between the two ratings, will typically only be applied if a minimum of one of the applicant's scores is above the lowest-scoring application that is to be funded for that particular committee.

Just in case you are asked to do a re-review, keep the applications and your working notes on file until competition results have been announced.

Annex 1: Benchmark Application

Note to Reviewers

The benchmark application presents information on a hypothetical candidate and is presented as though it has been extracted for you from a CIHR application form written by another reviewer. It is intentionally generic; the candidate could be female or male, in any health research area, and training inside or outside Canada. Do not worry about this lack of specificity.

Note: While the candidate in this example has a clinical background, the overall logic of the summary/assessment can be applied to any candidate with appropriate adjustments to expectations based on background, research area/theme/discipline, etc. Reviewers are asked to use their judgment and base their reviews on their expertise, experience and understanding of health research and training in these areas A few guiding notes have been included below.

The information has been organized as follows:

A - Introduction to the Candidate
1. The Candidate's Plans
2. The Proposed Research Project
3. Honours, Awards and Academic Distinctions of the Candidate
4. The Publications and Related Research Achievements of the Candidate
5. Characteristics and Abilities of the Candidate
6. Research Activity, Resources and Mentorship in the Training Environment

A - Introduction to the Candidate

• The candidate holds a health professional degree and will complete a PhD by June 2011.
• The health professional degree was obtained in 2003. After a period of clinical work, the candidate entered a two-year Master's program, September 2005 to June 2007. The candidate then commenced a PhD program at a more research-intensive university.
• The candidate is planning postdoctoral work at a different research institution commencing July 2011.

1. The Candidate's Plans

• The candidate's training plans were logical and easy to follow. There was a lucid description of experiences that have contributed to the development of an interest in health research.
• The candidate was introduced to research as a summer student during the final two years of their professional program. After graduation, exposure to practical challenges in clinical practice led to an interest in researching better ways of diagnosing and treating a particular syndrome. The candidate then entered a Master's program. There they were strongly encouraged by mentors to subsequently prepare for a career as a clinician-researcher by enrolling in a PhD program at a research-intensive university. As a doctoral student, the candidate has found an important research topic, developed some original methodology and published early results.
• It is not entirely clear why the candidate has chosen to work with the Fellowship supervisor, but the fit of expertise seems appropriate.
• After two years of postdoctoral work outlined in the application, the candidate expects to take up a joint research and clinical appointment at a Canadian university and affiliated clinic. Tentative arrangements have been made for these post-training positions. After the postdoctoral period, the candidate will apply for a career award to enable a full-time effort in research complemented by relevant clinical work.

2. The Proposed Research Project

• The candidate's research project was appropriately presented in general language. When technical terms were introduced they were, with two exceptions, adequately explained. The principal research hypothesis was reasonably well articulated.
• The project appeared to fit well with the candidate's background. It could not be called a "stretch" project - one that would take the candidate into unfamiliar territory in which new skills and research strategies would be required. It appeared to be quite closely related to the research work for the PhD.
• Results will likely have an impact on future research in the area. There is also a high likelihood that results will be relevant to clinical practice.
• The project would probably be considered moderately innovative by others working in the area. The work is doable in the time frame as long as resources are available.
• There seems to be a good fit between the candidate's project and the supervisor's research program. However, the candidate's project would not be considered as a critical element of the supervisor's work (Note: the relevance of the relationship may vary depending on research area/theme/discipline).
• The overall training program for the candidate seems well balanced. In addition to working on the postdoctoral project, the candidate will be afforded opportunities to improve skills in supervision, reviewing and writing grant applications, teaching and administration.

3. Honours, Awards and Academic Distinctions of the Candidate

• As an undergraduate, the candidate received an award as runner-up in a competition for best summer student research presentation.
• Master's study was supported by an award from a provincial agency. The candidate is currently holding an award from a research charity for doctoral work.
• The candidate was part of a university rowing team that won a silver medal in the national competition in 2006. In 2007, the candidate was vice president external affairs of the university graduate students' association.

4. The Publications and Related Research Achievements of the Candidate

Journal Articles

• There were 11 papers in the candidate's list of publications: eight published articles, two in press and one submitted. For reference, the expected productivity for a soon-to-be PhD in this particular research area is six or seven publications.

#	Year	Position of candidate's name in the list of authors	Contribution of candidate to the publication	Impact of journal*	Comment
* Field normalized journal impact
1	2003	Second of three authors	25%	Medium	From summer student work
2	2004	First of three authors	65%	Medium	From summer student work
3	2005	First of two authors	60%	Low	Case study from clinical work
4	2007	Third of four authors	20%	Medium	From Master's work
5	2007	Second of three authors	40%	High	Some overlap with article 4
6	2008	Second of five authors	20%	Medium	From Master's
7	2009	Sole author	100%	Low	Review article. From PhD work
8	2009	First of two authors	85%	Medium	From PhD
9	2010	First of three authors	75%	High	In press
10	2010	Second of five authors	30%	Medium	In press
11		First of two authors	75%	High	Submitted

Note: Different disciplines and environments offer different opportunity for publication, and expectations regarding the number of publications should be adjusted accordingly. In addition, journal impact factors vary from one discipline to another and do not necessarily indicate the quality of individual articles, and format should also be considered (article vs. book, etc.).

Related Research Achievements

• The candidate lists nine conference presentations. For reference, the expected number of presentations for someone with the experience of the candidate in this area is seven or eight. The candidate was specifically invited to present at a conference hosted by their university. Three of the nine presentations were at international conferences.
• The titles of the conference presentations suggest some overlap in content. It is reasonable to conclude that the nine presentations reflect four research themes.
• The candidate and PhD supervisor have applied for a patent on the research instrument that they jointly developed.

Note: As with publications, the opportunities and expectations with regards to presentations and conferences differ by discipline and can also be influenced by other factors.

5. Characteristics and Abilities of the Candidate

• The candidate's characteristics and abilities are described by three sponsors. You may rate each specific criteria listed below and then calculate an average score of the three submitted sponsors' assessment forms.
• It is most likely that sponsors’ assessments will be positive. To help in performing  an accurate review of these assessments, the following should be considered:
  • Do the detailed comments provided as part of the assessments support the scores as outlined on the first page of the assessment?
  • How long as the sponsor known the candidate?
  • What is the relationship of the sponsor to the candidate (i.e. are the assessments from three people under whom the candidate has trained and who have had a good opportunity to assess the candidate's potential for research)?

Characteristic and abilities of the candidate
Critical thinking Independence Perseverance Originality Organizational skills Interest in discovery Research ability Leadership	Sponsors' Assessments	4.5 - 4.9 outstanding 4.0 - 4.4 excellent 3.5 - 3.9 very good 3.0 - 3.4 good 2.0 - 2.9 average 1.0 - 1.9 below average 0.0 – 0.9 not acceptable	Assess the rating given for each characteristic as well as the narratives for each of the three sponsor assessment forms and provide a score based on your overall impression.

6. Research Activity, Resources and Mentorship in the Training Environment

• The postdoctoral research supervisor has been a principal investigator since 1987. The supervisor holds a research chair and received a national award for exceptional research achievement in 2006.
• The supervisor has authored or co-authored an average of four papers per year over the last five years. For reference, the usual productivity for researchers in this area is estimated at 3.8 papers. The supervisor is engaged in both national and international collaborations.
  Note: See note above regarding publication productivity and the impact of environment/discipline, which should also be considered for the supervisor.
• The supervisor is the principal investigator for a national agency grant and a grant from a health charity. Co-investigator status is indicated for one grant from an agency outside the country.
• The candidate will be provided with an office (shared with one other person), other necessary research space, a computer, a shared printer and an allowance for travel to meetings. It was not clear how much grant funding would be devoted to the candidate's project.
• The supervisor is currently mentoring three people: a health professional in a Master's program, a PhD student and a Master's student. They will be collaborating with a visiting scientist.
• A review of the authorship of the publications listed by the supervisor suggests that trainees are given an opportunity to lead or participate in the production of papers.
  Note: Opportunity to lead/participate and expectations regarding the candidate's role in relation to the supervisor may differ by research environment/discipline.
• The following table indicates the current activity of former trainees who moved on over the last five years.

Trainee	Period	Current position
Postdoctoral fellow	2004-2007	Public servant (Ministry of Health)
Master's student	2004-2006	Postdoctoral fellow
Postdoctoral fellow	2006-2008	Assistant professor
PhD student	2005-2009	Self-employed
Master's student	2005-2008	PhD student
PhD student	2007-2010	Volunteer in an international aid program

Annex 2: Criteria and Rating Scales

The Annex contains:

A. Overview of the Six Selection Criteria
1. Evaluating the Candidate's Plans
2. Evaluating the Proposed Research Project
3. Evaluating the Honours, Awards and Academic Distinctions of the Candidate
4. Evaluating the Publications and Related Research Achievement of the Candidate
5. Evaluating the Characteristics and Abilities of the Candidate
6. Evaluating the Research Activity, Resources and Mentorship in the Training Environment

A - Overview of the Six Selection Criteria

Fellowship selection criteria and weights are based on studies of the predictors of post-training research activity. They have been fine-tuned through two surveys of Fellowship reviewers.

The raw scores that you submit for each criterion on the 0.0 to 4.9 scale will be weighted automatically by CIHR in the calculation of an overall score.

The Six Criteria and their Weights in the Overall Score:

Criteria	Weight for Post-PhD Candidates1	Weight for Health Professional Candidates without PhD2	Overall Weight
1 Includes health professionals who hold a PhD degree. 2 Includes health professionals either pursuing a degree or research only.
Candidate Data
Candidate's Plans (Training Expectations)	10%	10%	60%
Proposed Research Project	10%	10%
Honours, Awards and Academic Distinction	5%	15%
Publications and Related Research Achievements	35%	25%
Assessments by Sponsors
Characteristics and Abilities	20%	20%	20%
Fellowship Training Environment
Research Activity, Resources and Mentorship	20%	20%	20%
	100%	100%	100%

1. Evaluating the Candidate's Plans

10% weight for all candidates

Working Definition

A description of the applicant's career intentions and proposal for achieving them.

What to Look For

Clarity and logic in the explanation of the candidate's plans for a research career and the relevance of the proposed training.

Rating Range	Description
4.5 - 4.9	Outstanding Faultless depiction of research career intentions and relevance of the proposed training. Ideal career path.
4.0 - 4.4	Excellent Clear, convincing depiction of research career intentions and relevance of the proposed training. Highly appropriate career path.
3.5 - 3.9	Very good Very good depiction of research career intentions and relevance of the proposed training. Logical career path.
3.0 - 3.4	Good Reasonable depiction of research career intentions and relevance of the proposed training. Suitable career path.
2.0 - 2.9	Average
1.0 - 1.9	Below Average
0.0 - 0.9	Not Acceptable

2. Evaluating the Proposed Research Project

10% weight for all candidates

Working Definition

A carefully planned, systematic study aimed at clearly answering a question in health research.

What to Look For

The ideal project is one that is best for the candidate given their education, experience and interests. It is the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training.

Bear in mind that it is not the project per se that is being assessed. It is the project as an integral part of the candidate's development as a researcher.

Rating Range	Description
4.5 - 4.9	Outstanding Extraordinary optimization of: challenge to the candidate, scientific importance and feasibility of completion during the Fellowship period. An ideal project that is faultlessly outlined.
4.0 - 4.4	Excellent Excellent optimization of: challenge, importance and feasibility. A highly suitable project that was superbly outlined.
3.5 - 3.9	Very Good Strong optimization of: challenge, importance, and feasibility. A very suitable project that was very clearly outlined.
3.0 - 3.4	Good Good optimization of challenge, scientific importance and feasibility. A suitable project that was well outlined.
2.0 - 2.9	Average
1.0 - 1.9	Below Average
0.0 - 0.9	Not Acceptable

3. Evaluating the Honours, Awards and Academic Distinction of the Candidate

5% Weight for Post-PhD applicants
15% for Health Professional without PhD applicants

Working Definition

Official recognition or prizes signifying special qualities of the recipient. Includes accomplishments in terms of formal education and scholarships.

What to Look For

When assessing this variable and other achievements of the candidate, it is essential to take into consideration the career path that they have followed to date. Assess the number, importance and breadth of the candidate's special distinctions relative to their education, training and work experience. Note relevance to research and whether the recognition is regional, national or international. Note the length of time required to complete academic programs and any indications of special academic distinction.

Rating Range	Description
4.5 - 4.9	Outstanding All aspects of the candidate's distinctions (their number, importance and breadth) indicate recognition of a very rarely encountered level of talent.
4.0 - 4.4	Excellent Several aspects of the candidate's distinctions (their number, importance or breadth) indicate recognition of superb talent.
3.5 - 3.9	Very Good At least one aspect of the candidate's distinctions (their number, importance or breadth) indicates recognition of talent.
3.0 - 3.4	Good The candidate's distinctions indicate an above-average performance.
2.0 - 2.9	Average
1.0 - 1.9	Below average
0.0 - 0.9	Not acceptable

4. Evaluating the Publications and Related Research Achievements of the Candidate

35% Weight for Post-PhD applicants
25% for Health Professional without PhD applicants

Working Definition

Articles, chapters or books published (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights.

What to Look For

Evidence of achievements in research relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., time spent raising children).

For publications, observe the number of co-authors and the position of the candidate's name in the authors list (note that the importance of this position can vary depending on the discipline, etc.). Note the candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (article, chapter, book, etc.).

Try to get a sense of the entire body of work and its likely impact. Note the publication dates and relate them to the candidate's education and training. Consider the list of abstracts as an indication of conference presentation activities. Note the candidate's other professional activities. Consider any patents or copyrights to which the candidate contributed.

Rating Range	Description
4.5 - 4.9	Outstanding All aspects of the candidate's publications and related research achievements (number, likely impact and breadth) indicate an extraordinarily productive and creative individual in their field of research/discipline.
4.0 - 4.4	Excellent Several aspects of the candidate's publications and related research achievements (number, likely impact or breadth) indicate excellent productivity and creativity in their field of research/discipline.
3.5 - 3.9	Very Good At least one aspect of the candidate's publications and related achievements (number, likely impact or breadth) indicate very good productivity or in their field of research/discipline.
3.0 - 3.4	Good There is evidence of greater than expected involvement in publication and related research activities in their field of research/discipline.
2.0 - 2.9	Average
1.0 - 1.9	Below average
0.0 - 0.9	Not acceptable

5. Evaluating the Characteristics and Abilities of the Candidate

20% weight for all candidates

Working Definition

A perspective on the candidate provided by persons who are familiar with their characteristics and abilities.

What to Look For

Evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with career research achievement. Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not. Read the supporting text carefully, taking note of the extent to which they justify the scores.

Look particularly for indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent.

Look also for indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.

If the candidate has had an opportunity to conduct research, look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.

Rating Range	Description
4.5 - 4.9	Outstanding An outstandingly critical, original and independent thinker. Exceptionally focused, energetic and creative. An ideal role model for others.
4.0 - 4.4	Excellent A highly critical, original and independent thinker. Very focused, energetic and creative. Excellent potential for future research leadership.
3.5 - 3.9	Very Good Clearly a critical, original and independent thinker. Definitely focused, energetic and creative. Very good potential for success as an independent researcher.
3.0 - 3.4	Good Appears to be a critical, original and independent thinker. Seems to be focused, energetic and creative. Above average potential for a productive career in research.
2.0 - 2.9	Average
1.0 - 1.9	Below Average
0.0 - 0.9	Not Acceptable

6. Evaluating the Research Activity, Resources and Mentorship in the Training Environment

20% weight for all candidates

Working Definition

Elements of the research environment that will contribute directly or indirectly to the quality of the candidate's research training experience.

What to Look For

Review information on the research experience, qualifications, honours and awards of the Fellowship supervisor. Examine the supervisor's publication record to get a sense of productivity, impact and collaboration (as noted above, please consider the different disciplines and their impacts on these).

Determine the research environment, including space, facilities, and personnel support available. Review the information on grants currently held, noting the extent to which the supervisor was either listed as a principal or co-applicant for the funds. Get a sense of the resources available and the overall level of activity.

Review the supervisor's training record. Note for each person listed: the level of training, length of time with the supervisor, degree received (if applicable) and current position. Your assessment should take into consideration the career stage and discipline of the supervisor. Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.

Rating Range	Description
4.5 - 4.9	Outstanding A vibrant, world-class research environment. Outstanding availability of research resources. Superb mentorship.
4.0 - 4.4	Excellent A highly active research environment. Excellent availability of research resources. First-rate mentorship.
3.5 - 3.9	Very Good A very active research environment. Very good availability of research resources. Strong mentorship.
3.0 - 3.4	Good An active research environment. Sufficient research resources available. Appropriate mentorship.
2.0 - 2.9	Average
1.0 - 1.9	Below Average
0.0 - 0.9	Not Acceptable

Annex 3: Calibrating Scores

Position your scores relative to those of the reviewer community

For each criterion circle the range of scores that includes your rating for the benchmark application.

Criteria	Range of Ratings of the Benchmark Application
	Low	Normal Ranges (a large majority of reviewers rated the applications in these ranges)	High
Candidate's Plans (Training Expectations)	< 3.9	3.9 - 4.3	> 4.3
Proposed Research Project	< 3.6	3.6 - 4.1	> 4.1
Honours, Awards and Academic Distinctions	< 3.5	3.5 - 4.1	> 4.1
Publications and Related Research Achievements	< 4.0	4.0 - 4.5	> 4.5
Characteristics and Abilities	< 4.0	4.0 - 4.3	> 4.3
Research Activity, Resources and Mentorship in the Training Environment	< 3.7	3.7 - 4.3	> 4.3

Understanding Differences

For criteria on which your rating was Low or High relative to most other reviewers, you should reread the benchmark application and then review the rating scale.

If after reviewing your rating you still see no reason for reconsidering your initial rating into the normal range, consider the reasons why other reviewers did not give a higher or lower rating to the benchmark application.

Criteria	Reviewers' Reasons for Not Giving a Higher Rating
Candidate's Plans (Training Expectations)	Lack of solid justification for choosing the supervisor. No clear indication of what the candidate expects to gain that will enhance their training. No clear idea of how this training will contribute to achieving their future goals.
Proposed Research Project	The candidate is continuing along the same lines of research as previously investigated - challenges not significant enough. Unlikely to gain new skills. Project appears only moderately innovative. A little unclear as to how much time will be spent teaching - may be involved in too many non-research activities.
Honours, Awards and Academic Distinctions	Research awards were local or charity foundation-based and therefore not as competitive as, say, national or international awards. Not many awards or distinctions. Other awards such as oral or poster presentation awards from international conferences would have been an asset.
Publications and Related Research Achievements	The candidate has ten published and one submitted paper, they are the first author on five (one is a review article and none are from their Masters) - the impact could be higher (in this particular scenario). Only one high impact article as first author. Quality of publications is generally low to medium with only two published in high end journals (plus one submitted, but this could end up in a lower end journal). One is a review article and another is a case report. On an annual basis, productivity seems good but is not in my opinion exceptional. Note: Interruptions in candidate's studies/career should be considered as well as their research area/discipline/background.
Characteristics and Abilities	Although sponsors comments were positive, no one indicated a very high enthusiasm for the candidate. Candidate not described as a future leader with exceptional qualities; not viewed as a future leader. Ratings from sponsors were a bit inflated, based on the written comments.
Research Activity, Resources and Mentorship in the Training Environment	Productivity of supervisor(s) - good but not remarkable (in this scenario). Funding held is acceptable, but not remarkable. One concern is the lack of additional personnel at the same level as or higher than the candidate (e.g., less people who could assist in the training). It is unclear how much infrastructure and funds will be directly available for the candidate's proposed training. The number of publications per year and number of trainees in the environment are strong but not excellent. Note: Publications records and funding held may vary depending on the proposed research environment/theme/area (e.g., biomedical vs. Other types of research, including that which is more qualitative).

Criteria	Reviewers' Reasons for Not Giving a Lower Rating
Candidate's Plans (Training Expectations)	Research plans and goals are in place - the candidate plans to pursue an academic research career. Clear and logical plans; early interest in research that has been fostered and nurtured. Good description of the way that past training has influenced their current decisions - solid game plan.
Proposed Research Project	The project was well presented and feasible and would significantly contribute to its field. Excellent fit with past training and research interests and potential impact on their field. Can be accomplished in time allotted. The research hypothesis and summary of the project were well articulated. Opportunity to acquire new skills needed to become an independent investigator.
Honours, Awards and Academic Distinctions	Already holds two competitive academic awards. The number of awards was very good to excellent. The achievements and involvement in student leadership activities were excellent - accomplished at the same time as research and studies. Made the Dean's list one year. Runner-up for research prize. Achievements indicate a well-balanced and capable candidate.
Publications and Related Research Achievements	Publication record shows overall good productivity relative to others in a similar situation. The impact factors of more recent papers are higher, suggesting more innovative work being done as the candidate progresses. The filing of a patent application is a positive achievement. Consistent effort and productivity. More presentations than average and several at international venues. Note: Expectations regarding number of publications, presentations and conferences produced by candidates may differ depending on research area/discipline, etc.
Characteristics and Abilities	All sponsor assessment forms indicate that the candidate is strong and has good potential. Sponsors highlight the candidate's work ethic; development suggests ability of candidate to conduct research independently. A particular sponsor puts in top five; leadership qualities - influenced the thinking of others. The examples provided by sponsors to back up their ratings were convincing. Ability to be involved in extracurricular activities while performing studies/research shows a high degree of organization.
Research Activity, Resources and Mentorship in the Training Environment	Supervisor has been recognized by peers and holds a national research award. Supervisor has a decent record: mentorship, publication frequency. Mentor has more than 20 years experience as a principal investigator. Supervisor has international recognition. Environment looks solid with adequate resources.

Annex 4: Reviewer Worksheet

This template is strictly for your working notes and will not be filed with CIHR.

Name of Applicant
Application Number
Background Information on the Candidate Refer to the CV Module of the Candidate	Degrees held or in progress: Time for completion of degree programs: Research experience:
Candidate's Plans Refer to the "Training Expectations" section of ResearchNet task entitled "Application Details Attachments"	Link between proposed and prior training: Career goals:
Proposed Research Project Refer to the following tasks on ResearchNet: Lay Abstract Application Details Attachments (section entitled "Research Proposal Summary")	Project: Expected duration of training: Suitability of project for applicant's career:
Honours, Awards and Academic Distinction Refer to the CV Module of the Candidate
Publications and Related Research Achievements Refer to the CV Module of the Candidate Note: Be aware of the variances between disciplines.	Papers: Presentations: Other: Candidate's role:
Characteristics and Abilities Refer to sponsors' assessments	Sponsor: Relationship to candidate: Notes:
	Sponsor: Relationship to candidate: Notes:
	Sponsor: Relationship to candidate: Notes:
Research Activity in the Training Environment Refer to the CV Module(s) of the proposed supervisor(s) Note:Be aware of the variances between disciplines.	Name of proposed supervisor: Qualifications: Publication activity: Impact: Collaborators:
Resources Available in the Training Environment Refer to the following tasks on ResearchNet: Space, Facilities and Personnel Support CV Module(s) of the proposed supervisor(s)	Space, facilities and personnel support available: Research funding available to the supervisor: Other key resources:
Mentorship Record in the Training Environment Refer to the CV Module(s) of the proposed supervisor(s)	Trainees in the last five years - level and outcome:
General Notes on the Application	Overall impression:

Annex 5: Guide to Locating Information

Criteria	Section (s) Where to Find Information
Candidate's Plans	Training Expectations Degrees or Qualifications Sought Activity other than Research Reason for Selecting a Foreign Training Environment (if applicable)
Proposed Research Project	Project (abstract) Project (Title) Summary of Research Project
Honours, Awards and Academic Distinction	The "Distinctions" section should list any honours and awards
Publications and Related Research Achievements	The candidate's CV should list the following: Publications table Number and type of publications Invited presentations List of publications Other professional activities Patents and copyrights
Characteristics and Abilities	Sponsors' assessments
Research Activity, Resources and Mentorship in the Training Environment	Space, facilities and personnel support available The CV of the supervisor (s) of the candidate should contain the following: Information on the research activity of the proposed supervisor: publications; other professional activities; patents and copyrights Number of current trainees Funds currently held Supervisory experience

Annex 6: Issues to be Reported for CIHR's Attention

In order to report any ethical issues, et cetera directly to CIHR, please use the PDF form entitled "Issues for CIHR attention Form". Once completed, you may either return it to us via fax (number is indicated on the form) or by an email attachment to the following address: Fellowships@cihr-irsc.gc.ca [ PDF (15.97 KB) ]. It can also be found under the "Support section" of your Peer Review task in your ResearchNet account.