Peer Review Manual - Foundation

Notice

Last updated: November 28, 2016

Due to the low number of applications in the New/Early Career Investigators cohort, it was determined that those applications could not be adequately assessed as a separate cohort, as was originally planned. It is imperative to achieving quality peer review that reviewers have enough applications, which is 8-12, to appropriately rank them against each other and establish a meaningful ranking list. Having reviewers with too few applications to rank against each other could disadvantage some early career applicants. The adjustment to the review process will ensure a fairer process where every application is ranked against a sufficient number of other applications.

In order to ensure early career investigators are still being treated fairly, CIHR commits to: 1) awarding a minimum of 15% of grants to early career investigators as planned; and, 2) ensuring that approximately 50% of the applications from the New/Early Career Investigators cohort move from Stage 1 to Stage 2. Also, regardless of career stage, the proportion of female applicants moving forward from Stage 1 to Stage 2 will equal the proportion of female applicants to the competition, if necessary. (For example, if 30% of the applications submitted at Stage 1 were from female program leaders, then approximately 30% of the applications moving to Stage 2 should also be from female program leaders.)

Contents


Please note that in the context of this document, “the applicant” may refer to an individual Program Leader or multiple Program co-Leaders.

Program Overview

The Foundation Grant Program is designed to contribute to a sustainable foundation of health research leaders by providing long-term support for the pursuit of innovative and high-impact programs of research. Programs of research are expected to include integrated, thematically-linked research, knowledge translation, and mentoring/training components.

Foundation grants are designed to support research leaders at any career stage to build and conduct programs of health research across CIHR's mandate. Eligible applicants will include New/Early Career, Mid Career, and Senior Investigators all of whom are independent researchers with a demonstrable track record of excellence and impact in their field of study.

While the application form and adjudication criteria are the same for all applicants, the criteria must be applied in the context of the applicant's career stage.

The Foundation Grant Program is supported by a three-stage competition and review process that focuses reviewer attention on specific structured review criteria at Stage 1 and at Stage 2.

Peer Review Roles

Reviewers (Stage 1 and Stage 2)

The majority of participants in the review process are reviewers who conduct the evaluation of the applications in the competition.  Reviewers are responsible for reading and assessing the applications based on the evaluation criteria, and participating in online discussion for applications where there are discrepancies between reviewers.  Once the discussion period has passed, reviewers confirm the ranking of their applications and submit their final reviews to CIHR.

Principal tasks:

  • Complete Conflict of Interest and Ability to review tasks;
  • Evaluate/rate applications;
  • Participate in online discussion of applications as required;
  • Rank applications;
  • Submit final reviews;
  • Provide feedback to CIHR regarding the review process.

Success of the peer review system is critically dependent upon the willingness and ability of all reviewers to exercise rigorous scientific judgment, to be fair and reasonable and to understand and take into account in a balanced way the particular context of each application. The opinions expressed in the reviews are those of the reviewers alone and are provided to the applicant without prior editing by CIHR staff.  In circumstances where the Competition Chair or CIHR staff deems that the review expresses comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way or contains factually incorrect information regarding CIHR's policies, CIHR reserves the right to remove such information from the review.  A constructive, quality review, which helps the applicant by pointing out strengths and weaknesses that could be improved upon in a future submission will help to convince an applicant that you provided a fair assessment of the proposal.

Competition Chairs (Stage 1 and Stage 2)

As part of the Foundation Grant Program, the Competition Chairs are responsible for overseeing and supporting the Stage 1 and Stage 2 remote review process by:

  • Assisting with reviewer recruitment and identifying reviewers to fill expertise gaps;
  • Validating the assignment of applications to reviewers;
  • Overseeing and moderating the online discussions;
  • Identifying reviews that require revision and communicate with CIHR staff ;
  • Providing feedback to CIHR regarding the review process.

Competition Chairs are able to:

  • Flag applications for discussion in ResearchNet. This may be used when there are discrepancies in the reviewer's assessment of an application. At Stage 2, Competition Chairs will also be expected to identify applications with discrepant budget recommendations and guide the online discussion to reach consensus on budget recommendations where possible.  If the application is flagged, the reviewers will receive an email that the application must be discussed.
  • Send private messages directly to reviewers, outside the online discussion, using the private messaging functionality, as appropriate.  For example, regarding  reviews that contain insufficient/inappropriate content.

One of the most important roles of the Competition Chair is to monitor the quality of reviews. The Stage 1 reviews will be provided to the applicants. The Stage 2 reviews will also be provided to the applicants as well as the Final Assessment Stage committee members, who will rely on these reviews to make funding recommendations. As such, the following information should be included in the Stage 1 and Stage 2 reviews:

  • Context: reviewers are asked to provide context in their assessment of each adjudication criterion as the Final Assessment Stage reviewers will not have read the applications.
  • Sufficient information: reviewers are asked to provide relevant and substantial information to inform decision making and recommendation for funding.
  • Justification of rating: reviewers are asked to provide sufficient information in the body of their review that would explain the choice of rating.

In monitoring the online discussions to ensure that all comments are appropriate, the Competition Chair will also look for the inclusion of:

  • Information identifying the reviewer;
  • Use of outside information;
  • Mismatch between the comments and the rating being provided.

Time requirements will vary depending on:

  • Total applications assigned to a Competition Chair;
  • Number of applications that require discussion.

Reviewers (Final Assessment Stage - FAS)

Competition Chairs who participated in Stage 2 will be asked to participate as members in the multidisciplinary face-to-face committee meeting. This multidisciplinary committee covers the full spectrum of health research. At this stage, respecting conflicts of interest, the reviewers discuss "grey zone" applications, with particular emphasis on applications with variances in independent reviewer rankings. This final stage therefore builds on Stage 2 where applications will have been thoroughly and expertly assessed. The Final Assessment Stage committee's recommendations will be summarized and submitted to CIHR for the final funding decision. The members will be provided with the Stage 2 reviews. The FAS Reviewer's duties include:

  • Complete Conflict of Interest and Ability to review tasks;
  • Assess Stage 2 reviews;
  • Bin applications;
  • Prepare for the FAS meeting;
  • Attend the FAS meeting;
  • Provide feedback to CIHR regarding the review process.

For each assigned application, members will have access to the following:

  • Written reviews of each of the Stage 2 reviewers;
  • Individual reviewer ratings and rakings from Stage 2;
  • Stage 2 Consolidated Ranking; and
  • Full application.

Chair and Scientific Officer (Final Assessment Stage)

The Final Assessment Stage committee also includes a committee Chair and a Scientific Officer (SO).

The Chair ensures that the peer review committee functions smoothly, effectively and objectively, in accordance with CIHR's policies and procedures.  The Chair's duties include:

  • acting as a moderator to establish a positive, constructive, fair-minded environment in which research proposals are discussed;
  • fulfilling an oversight role ;
  • working with the SO to summarize the discussion of each application before the voting, when required.

The Scientific Officer's duties include:

  • taking notes of the discussion of each application, which will be sent to applicants with their review documents;
  • ensuring that issues of ethics and other concerns that arise during discussion are flagged for the attention of CIHR are recorded for each application;
  • supporting the Chair in his/her role during the peer review committee meeting.

The Chair and the Scientific Officer do not rate applications or vote during the committee meeting.

Equity in the Peer Review Process

Peer review by nature is a subjective process. Bias may manifest in several ways and could be based on a school of thought, fundamental versus applied or translational research, areas of research or approaches (including emerging ones), size or reputation of a participating institution, age, language, personal factors or gender of the applicant. To sensitize reviewer to unconscious biases they may hold, they will be cautioned against any judgment of an application based on such factors, and asked to constantly guard against the possibility of implicit bias influencing the decision-making process.

Addressing gender equity and reducing gender bias

CIHR is actively engaged in increasing gender equity in their review processes to address some of the gender inequities that we have observed from recent funding competitions.

The following are some tips for reviewers:

  1. Before engaging in peer review, reflect on the susceptibility of all humans to bias in judgment. The following resources can help:
  2. Allow sufficient time and avoid multi-tasking when reviewing applications to allow for self-correction of bias related tendencies.
  3. Review the assessment criteria before evaluating applications.
  4. Question whether your evaluation would change if the investigator were of a different gender?
  5. Guard against over-reliance on one piece of information or “first-impression”.
  6. When writing or presenting your review, please:
    • Avoid using stereotypical or interpersonal attributes when describing character and skills, (e.g.,  words like nice, hardworking, conscientious, dependable, diligent, kind, agreeable, sympathetic, compassionate, selfless, giving, caring, warm, nurturing, maternal, etc.);
    • Instead, focus on research skills/achievements using words that describe the candidates research excellence (e.g., thought provoking, innovative, novel, thorough, detailed, impactful);
    • Consider using 'stand-out' adjectives for both men and women, where appropriate (e.g., superb, excellent, outstanding, confident, successful, ambitious, knowledgeable, intellectual etc.);
    • Use the nominee's formal title and last name instead of their first name. Avoid attributing the contribution of an applicant's work to the order of authors, as not all disciplines follow a single convention;
    • Consider whether your comment unintentionally includes 'doubt raisers' (negative language, hedges, unexplained comments, faint praise and irrelevancies (e.g., 'might make an excellent leader' versus 'is an established leader').

How this manual is organized

This manual is divided into three sections which correspond to the three stages of the adjudication process as illustrated in Figure 1.

Figure 1. Overview of the Foundation Scheme Adjudication Process

This figure illustrates the three-stage adjudication process for the Foundation.

  • Stage 1 assesses the Caliber of the Applicant(s) and Vision and Program Direction. Applicants submit their Stage 1 application, which is matched to up to five expert reviewers. Reviewers complete the Stage 1 remote review and submit their results to CIHR. Successful applicants are invited to submit a Stage 2 application.
  • Stage 2 assesses the Quality of the Program and Quality of the Expertise, Experience and Resources. Reviewers complete the Stage 2 remote review and submit their results to CIHR.
  • The Final Assessment Stage involves a face-to-face multidisciplinary committee meeting. This committee will make recommendations on which applications to fund.

Section 1 outlines the adjudication process for Stage 1 of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: 1) preliminary review of the applications, 2) online discussion, and 3) submission of the final rank list to CIHR.

Section 2 outlines the adjudication process for Stage 2 of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: 1) preliminary review of the applications, 2) budget assessment 3) online discussion, and 4) submission of the final rank list to CIHR.

Section 3 outlines the adjudication process for the final stage of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: 1) pre-meeting activities, and 2) a face-to-face committee meeting.

Some reviewers may be invited to participate in both Stage 1 and Stage 2 if required. The Final Assessment Stage face-to-face committee will be composed of the Stage 2 Competition chairs.

In addition to reviewing the material in this manual, it is essential that peer reviewers read and become familiar with the Foundation Grant: 2016 competition. All CIHR peer reviewers are also expected to:

For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.

Section 1 – Adjudication Process for Stage 1

Stage 1 focuses on the Caliber of the Applicant and the Vision and Program Direction.

1.1 Preliminary Reviews of Applications

Each application will be assigned to up to five reviewers, based on optimal matching between the application content and reviewer expertise. Each reviewer will receive between 10 and 16 applications. Reviewers will access their assigned applications through ResearchNet and complete their reviews remotely. It is expected that it will take approximately 1-3 hours to complete each review.

The application will be presented in a structured format to align with the adjudication criteria, outlined in this manual. Reviewers must consult the Stage 1 Adjudication Criteria Descriptors and Interpretation Guidelines for guidance on how to apply the adjudication criteria. Reviewers will also be asked to review the applicant CV(s). Through their CV(s), applicants will highlight their recognitions, funding history, activities and contributions that best demonstrate their leadership, significant contributions and productivity in the context of their research field(s).

Reviewers will be asked to provide a rating and to justify the rating by stating the strengths and weaknesses for each sub-criterion outlined in the table below. The fields to outline strengths and weaknesses for each application in ResearchNet are mandatory fields. It is important for reviewers to clearly articulate the strengths and weaknesses as they will be used for two purposes: 1) to provide applicants with feedback and 2) to provide the other reviewers assigned to the application with a justification for the ratings and rankings given to the application.

In some cases, there may be multiple Program Leaders for a single Foundation grant. These applicants will need to demonstrate synergy and a history of co-leading programs of research. In these cases, the assessment of each adjudication criterion should consider both the individual and joint research contributions of the Program co-Leaders.

Reviewers must take into consideration the career stage, research field and institution setting of all applicants when assessing each criterion. New/Early Career Investigators may show evidence of leadership, contributions and productivity in different ways than Mid Career/Senior Investigators. The evidence should be notable compared to peers in similar fields and career stages.

To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale acknowledging that many of the ratings are likely to fall into this range. Reviewers are encouraged to use the full breadth of the scale and should use the increased granularity within the top descriptors to express differences within these categories. To facilitate this, the following scale including descriptors and definitions is provided:

Descriptor Abbreviation Definition

Outstanding

O++

For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.

O+

O

Excellent

E++

For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.

E+

E

Good

G

For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.

Fair

F

For this sub-criterion, the application broadly addresses all relevant aspects. Major revisions are required.

Poor

P

For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

As each review is completed, ResearchNet will automatically show a ranked list of the individual reviewer's set of assigned applications based on the ratings (highest to lowest). Reviewers may make adjustments to their ratings until they are comfortable with the rankings, however, there will be no requirement to break ties at this stage of the adjudication process. Ultimately it is the application's ranking that will determine its standing within the competition and will be used to make funding decisions.

The reviewers will then be asked to submit their preliminary reviews to CIHR in order to proceed to the next step.

1.2 Stage 1 Adjudication Criteria Descriptors and Interpretation Guidelines

Stage 1 focuses on the caliber of the applicant in the context of the qualities deemed essential to meet the objectives of the Foundation grant. These qualities are: leadership, ability to deliver significant contributions, and productivity. Stage 1 also assesses the ability to define and articulate a clear and compelling vision and program direction. Note that reviewers will be selected based on their expertise in the applicant's field of research.

In some cases, there may be multiple Program Leaders for a single Foundation grant. These applicants will need to demonstrate synergy and a history of co-leading programs of research. In these cases, the assessment of each adjudication criterion should consider both the individual and joint research contributions of the Program co-Leaders.

In the next sections, each of the sub-criteria relating to the Caliber of the Applicant and the Vision and Program Direction will be described in more detail. Reviewers will be reminded of what applicants were asked to provide in the application for the sub-criterion as well as the specific review questions that have been defined for each sub-criterion. A set of interpretation guidelines and considerations have also been summarized for each sub-criterion. These are intended to provide guidance to reviewers as they assess each section of the application.

Stage 1 - Criterion 1: Caliber of the Applicant

Please note that for each sub-criterion under Caliber of the Applicant, reviewers must consider:

  • The applicant's recent experiences and contributions, taking into account accomplishments over the applicant's entire career;
  • The nature, breadth and depth of the applicant's experiences and contributions, in the context of the applicant's career stage;
  • The nature, breadth and depth of an applicant's experiences and contributions, in the context of the applicant's research field;
  • Personal circumstances that result in legitimate career interruptions, such as extended leaves of absence or decreased productivity for health, family or other reasons. Note that applicants are encouraged to explain any personal circumstances to ensure a fair assessment of their achievements. Furthermore, the application may include a PDF document to supplement the publication information included in the applicant’s CCV.

Sub-criteria to be assessed under this criterion include:

1.2.1. Leadership (25%)

This sub-criterion is intended to assess the extent to which the applicant has influenced and inspired others in their field of research, as well as the applicant’s ability to effectively direct a program of research.

  • What applicants have been asked to provide
    The applicant will be asked to highlight their leadership experience. This should include a description that demonstrates leadership in the context of their career stage and field(s) of research, and should also include relevant and recent examples of major projects or research programs that they have led where effective leadership was evident. The applicant will be asked to point to relevant, recent examples in their CV as appropriate.
  • Items for reviewers to consider
    • Is the applicant recognized in their field, demonstrating a history of holding influential roles, inspiring others, mobilizing communities and advancing the direction of a field?
    • Has the applicant demonstrated the ability to successfully establish, resource, and direct programs of research, which should include: securing the required resources; ensuring effective collaboration, and/or incorporating knowledge translation strategies?
  • Interpretation guidelines
    • The applicant should be recognized and respected in their research field. This can be demonstrated through descriptions of the sphere of influence the applicant has achieved to date in their research field, their Institution, within their research community and any influence they have had in society at large. The applicant should also have experience in building and/or mobilizing networks and/or communities.
    • The applicant should demonstrate the ability to effectively direct a program of research. A Program Leader should demonstrate the ability to:
      • Address compelling research questions;
      • Develop and direct major projects and/or programs of research, which may include national or international networks (where appropriate);
      • Resource a program of research - including securing funding for research projects, attracting and retaining research personnel, acquiring appropriate infrastructure, etc.;
      • Build and manage collaborations; and/or,
      • Incorporate knowledge translation strategies and activities into a program of research, including the dissemination, application, and/or uptake of research findings.
  • Considerations
    For New/Early Career Investigators, the potential for leadership should be considered.

1.2.2. Significance of Contributions (25%)

This sub-criterion is intended to assess the extent to which the applicant's contributions to date have had an impact (a tangible benefit, or positive influence) on health, health care, health systems and/or health research. These contributions could include advancements in different research and health-related areas, as well as contributions to maintaining a sustainable foundation of health researchers. The nature and quantity of these advancements may depend on the applicant's field of research and career stage.

  • What applicants have been asked to provide
    The applicant will be asked to highlight the impact of their contributions on health and health research. The applicant will be asked to highlight and describe the tangible benefits or positive influences of their contributions on health, health care, health systems and/or health research. The applicant will be asked to point to relevant, recent examples in their CV(s), as appropriate.
  • Items for reviewers to consider
    • Has the applicant significantly advanced knowledge and/or its translation into improved health care, health systems, and/or health outcomes?
    • Has the applicant engaged, trained, and/or launched the careers paths of promising individuals in research and/or other health-related non-academic fields?
  • Interpretation guidelines
    • The applicant should demonstrate the impact they have had on advancing research knowledge and/or improving research practice in their field. Additionally, the applicant should also demonstrate how they have contributed to the advancement of health care, health systems, health outcomes, and economic prosperity (as applicable). When assessing contributions, consider the following:
      • Contributions to the advancement of knowledge (health or other fields) include advances or improvements to the current thinking in the field of research that have been sustained.
      • Contributions to research (health or other fields) include broad-scale or transformative changes to current research practices and approaches.
      • Contributions to health care include improvements in diagnosis, treatment, and prevention of disease, conditions, disability, injury, and other physical and cognitive impairments.
      • Contributions to health systems include improvements in the organization of people, institutions, health information, health policies, and the human and financial resources to deliver health care services.
      • Contributions to health outcomes include improvements to the health and/or quality of life of individuals or populations.
      • Contributions to economic prosperity include the development of economic opportunities stemming from the application or commercialization of health research.
    • Applicants should demonstrate a history of contributions to the advancement of research capacity.
      • Contributions to a sustainable foundation for health research and knowledge translation (health or other fields) include supporting the mentoring/training, career development, and recruitment and/or retention of highly qualified personnel in academic and non-academic health-related fields.
      • Contributions to a sustainable foundation of health research from new/early career investigators may be different from those of established researchers and might focus more on efforts to build capacity (e.g., ability to attract students/trainees/emerging scholars, developing and implementing innovative mentoring/training opportunities, etc.).

1.2.3. Productivity (25%)

This sub-criterion is intended to assess both the quality and quantity of research outputs generated by the applicant.

  • What applicants have been asked to provide
    The applicant will be asked to highlight both their career and recent productivity, focusing on both the quality and quantity of research outputs. The applicant will be asked to point to relevant, recent examples in their CV(s), as appropriate.
  • Items for reviewers to consider
    • Has the applicant demonstrated an outstanding level (quantity) of research outputs in their field based on prior work?
    • Has the applicant’s previous work generated high-quality research outputs?
  • Interpretation guidelines
    • The applicant should demonstrate an exceptionally high level of research outputs compared to investigators in a similar field and at a similar career stage.
    • The applicant’s previous work should have produced research contributions that are recognized to be of high-quality within their research field.

Stage 1 - Criterion 2: Vision and Program Direction

1.2.4 Vision and Program Direction (25%)

This sub-criterion is intended to assess the applicant’s ability to define and articulate a clear and compelling high level vision of their program of research. The assessment of the quality of the vision should consider the following characteristics:

  • What applicants have been asked to provide

    The applicant will be asked to articulate a compelling high level vision and direction for their research program. This will require the applicant to highlight the goal, overall objective(s), expected outputs/ contribution(s), and the significance of the proposed program if the objectives are met.

    Foundation grants are meant to be flexible in order to allow the Program Leader(s) an opportunity to innovate, or explore new lines of inquiry. The information requested in this section is not intended to include details of each thematically-linked project within the program of research.

    If successful at Stage 1, applicants will be asked to expand on their program of research at Stage 2.

  • Items for reviewers to consider
    • Are the vision, goal, overall objective(s), and potential contributions of the proposed research program well-defined and well-articulated in the context of a logical career progression for the Program Leader(s)?
    • Is the vision forward-looking, creative, and appropriately ambitious?
    • Does the vision aim to significantly advance knowledge and/or its translation to improved health care, health systems, and/or health outcomes?
  • Interpretation guidelines
    • The vision should outline the program direction envisioned for the duration of the grant in a coherent and clear fashion, highlighting the key synergistic components of the program of research.
    • A forward-looking vision is future-oriented, and describes the proposed contribution(s) of the program of research (i.e., what the program intends to achieve in the short- and long-term). A creative vision describes the program's potential to foster innovative approaches, and/or innovative outputs, outcomes, and impacts. An appropriately ambitious vision is believable and realistic (e.g., sets an appropriate scope for the program of research). It considers key elements required for the success of the program, and sets the direction for program planning through a clearly articulated research program goal and supporting objectives.
    • The vision of the research program should aim to achieve significant advancement in health-related knowledge, health research, health care, health systems, and/or health outcomes. This could be defined by the importance of the program (e.g., based on an assessment of the declared benefits, and their relation to any current issues, gaps and opportunities), as well as the significance of short-term and long-term research outcomes.

1.3 Online Discussion

Once the preliminary reviews are submitted, the other reviewer's reviews will be made available. Reviewers will be able to discuss each of their assigned applications with the other reviewers who were assigned the same application. An asynchronous online discussion tool has been developed to allow reviewers to communicate with each other to share expert perspectives and to discuss any discrepancies.

The objective of the discussion will be to discuss and understand discrepancies in reviews. There will be no requirement to reach consensus. Each reviewer's name and their preliminary review (ratings, written comments and ranking) will be visible for each application. If there are limited discrepancies in reviews, it is possible that a discussion may not be required.

The discussion period will last 72 hours and reviewers will be made aware of the planned discussion period for the competition well in advance of the deadlines. Reviewers may log in at any time during the discussion period and will not be expected to participate in online discussions simultaneously.

All reviewers assigned to an application will be able to participate in the discussion as soon as all of the reviews for that application have been submitted. The discussion for all applications will open on the established deadline and it is therefore important to respect the established deadlines. Reviewers who have not submitted their preliminary reviews will not be able to participate in the discussion and this will be flagged in their online status.

A Competition Chair will also be assigned to the application to moderate the discussion. The Competition Chair will be tasked with ensuring that discussions take place if warranted, and may prompt reviewers to discuss. CIHR staff may also initiate a discussion, or provide input where necessary, to ensure that CIHR policies and procedures are followed.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings and/or rankings.

It should be noted:

  1. Every reviewer will have the ability to initiate a discussion thread. Every comment will be shared with every reviewer assigned to that application. Once comments are posted, reviewers will not be able to delete or edit them.
  2. Reviewers will be able to post comments for the attention of CIHR staff; however they will be visible by the other reviewers.
  3. A notification email will be sent daily to advise reviewers of new posts.
  4. As applications are discussed, reviewers may feel the need to update their preliminary ratings and reviews. This can happen at any time both during and after the discussion period. These modifications will be dynamically updated and visible to other reviewers.
  5. A transcript of the online discussions will not be given to the applicants. However, discussion thread content will be retained by CIHR as part of the application record and this record is subject to the Government of Canada’s Access to Information Policy and Privacy Acts.

1.4 Submit the Final Rank List to CIHR

Once the asynchronous discussions are complete, reviewers will be asked to finalize their reviews and their individual ranked list. At this point, reviewers must break any ties in their individual ranked list. Once ties are eliminated, reviewers may modify the rank order of their assigned applications by moving the applications up or down the rank list. The original rank order will remain visible as a reference point. Once satisfied with the rank order of applications that they have been assigned, the reviewers will be asked to submit the final rankings to CIHR. Note that reviewers will not see the final ranked lists of other reviewers. At this point, the adjudication process for Stage 1 is complete.

Once all rankings have been received, CIHR will calculate a consolidated ranking for each application and compile a list of all applications in the competition ranked from highest to lowest. Each application will have up to five reviews, a ranking from each reviewer, a consolidated ranking and a standard deviation. CIHR will determine which applicants will be invited to complete a Stage 2 application based on the number of applications, consolidated ranking, standard deviation and funds available for the competition.

An overview of the Stage 1 adjudication process is outlined in Figure 2.

Figure 2. Stage 1 Overview

Preliminary Reviews of Assigned Applications

Reviewers will be asked to provide a rating for each adjudication criterion and to justify the rating by providing comments on the strengths and weaknesses. To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale to allow reviewers to indicate finer differences between high quality applications.

The adjudication criteria are:

Criterion 1: Caliber of the Applicant

  • Leadership (25%)
  • Significance of Contributions (25%)
  • Productivity (25%)

Criterion 2: Vision and Program Direction (25%)

Online Discussion

Once the preliminary reviews have been submitted, the other reviewer's preliminary reviews (ratings and comments) will be made available to reviewers who have been assigned the same application. Reviewers will be able to read the preliminary reviews and discuss each of their assigned applications with the other reviewers who were assigned the same application. This is the reviewers' opportunity to share their expert perspectives, and to discuss any rating discrepancies in a environment.

The Competition Chair or CIHR staff may initiate discussions when there are large discrepancies between reviewer ratings.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings and/or rankings.

Submit the Final Ranked List to CIHR

A reviewer's set of applications will be ranked in order from highest-rated to lowest-rated based on the ratings given to the application by the reviewer.

At this stage, some applications may be tied (i.e., have the same rank). The reviewer will be required to break these ties by moving one application ahead of the other.

Once the reviewer is satisfied with the ranked list of applications, he/she will submit the ranked list to CIHR. Once all ranked applications have been received, CIHR will compile a list of all applications in the competition ranked from highest to lowest.

The number of applications, consolidated ranking, and the standard deviation of each application, as well as the funds available for the competition will be used to identify which applications will be invited to complete a Stage 2 application.

Section 2 – Adjudication Process for Stage 2

Stage 2 focuses on the Quality of the Program and Quality of the Expertise, Experience and Resources and is distinct from the Stage 1 application.

2.1 Preliminary Reviews of Applications

Each application will be assigned to up to five reviewers, based on optimal matching between the application content and reviewer expertise. Matching and assignments will be done on the basis of reviewer declarations of their ability to review and with the assistance and oversight of Competition Chairs. Reviewers will access their assigned applications through ResearchNet and complete their reviews remotely. Reviewers at Stage 2 will not have access to the materials and reviews from Stage 1.

The application will be presented in a structured format to align with the adjudication criteria, outlined in this manual. Reviewers must consult the Stage 2 Adjudication Criteria Descriptors and Interpretation Guidelines for guidance on how to apply the adjudication criteria.

Reviewers will be asked to provide a rating and to justify the rating by stating the strengths and weaknesses for each sub-criterion outlined in the table below. The fields to outline strengths and weaknesses for each application in ResearchNet are mandatory fields. It is important for reviewers to clearly articulate the strengths and weaknesses as they will be used for three purposes: 1) to provide the other reviewers assigned to the application with a justification for the ratings and rankings given to the application, 2) to provide reviewers in the final stage with a justification for the ratings and rankings given to the application, and 3) to provide applicants with feedback.

Reviewers must take into consideration the career stage, research field and institution setting of all applicants when assessing each criterion.

To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale acknowledging that many of the ratings are likely to fall into this range. Reviewers are encouraged to use the full breadth of the scale and should use the increased granularity within the top descriptors to express differences within these categories. To facilitate this, the following scale including descriptors and definitions is provided:

Descriptor Abbreviation Definition

Outstanding

O++

For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.

O+

O

Excellent

E++

For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.

E+

E

Good

G

For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.

Fair

F

For this sub-criterion, the application broadly addresses all relevant aspects. Major revisions are required.

Poor

P

For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

As each review is completed, ResearchNet will automatically show a ranked list of the individual reviewer's set of assigned applications based on the ratings (highest to lowest). Reviewers may make adjustments to their ratings until they are comfortable with the rankings, however, there will be no requirement to break ties at this stage of the adjudication process.

Reviewers will also be asked to flag applications in ResearchNet that involve any of the issues outlined below. These issues are not to be considered as criteria for adjudication unless they impact the scientific quality of the application. For more information concerning these issues, please refer to the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS-2).

  1. Ethical and other policy requirements: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may flag specific issues, such as the use of human participants, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  2. Human pluripotent stem cell research (PSCC): Applications involving the use of human stem cells which will be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). If the box is not checked by the applicant and the proposed work utilizes HPSC, then it is essential that this is flagged by reviewers.

2.2 Stage 2 Adjudication Criteria Descriptors and Interpretation Guidelines

Stage 2 focuses on the quality of the proposed program of research, including the research concept, research approach, expertise and capacity building, as well as the supporting environment for the proposed research program.

Reviewers are therefore selected on the basis of expertise to review the research concept and approach for each application. The applicants invited to submit an application in Stage 2 of the Foundation competition process have been selected based on their demonstrated past success in leading programs of research. They have proven themselves able to develop and implement detailed methodologies, techniques, etc., to bring research projects to fruition; and, capable of mobilizing and securing the appropriate expertise to deliver significant contributions. Therefore, the level of information provided by the applicants to address their proposed program of research at Stage 2 is expected to be at a high-level, and will not include details of each specific project within the program of research. Foundation grants are meant to be flexible in order to allow Program Leader(s) an opportunity to innovate, or explore new lines of inquiry. Reviewers will be selected based on their declared expertise in the applicant's field of research.

In the next sections, each of the sub-criterion relating to the Quality of the Program and the Quality of the Expertise, Experience and Resources, will be described in more detail. Reviewers will be reminded of what applicants were asked to provide in the application for the sub-criterion as well as the specific review questions that have been defined for each sub-criterion. A set of interpretation guidelines and considerations have also been summarized for each sub-criterion. These are intended to provide guidance to reviewers as they assess each section of the application.

Stage 2 - Criterion 1: Quality of the Program

Sub-criteria to be assessed under this criterion include:

2.2.1 Research Concept (25%)

This sub-criterion is intended to assess the research component of the program, as well as the significance of the anticipated outcomes.

  • What applicants have been asked to provide
    The applicant will be asked to articulate the program of research by outlining the goal, objective(s), and, at a high level, the components of the program of research and potential short-term and long-term impact(s) of the proposed program.
  • Items for reviewers to consider
    • Are the goal and objectives of the proposed program well-defined and well-articulated?
    • Is there conceptual coherence within the program of research?
    • Are the potential short-term and long-term program outputs significant? Are they likely to significantly advance health-related knowledge and/or its translation into improved health care, health systems, and/or health outcomes?
  • Interpretation guidelines
    • The goal and objectives of the program should be well-defined, and appropriate given the nature and scope of the program of research. There should be a sound, conceptual justification for the component parts and should be grounded in the relevant literature and/or previous results (as appropriate).
    • The components of the program of research (including any relevant knowledge translation components) should be aligned with the overall goal and objectives of the program, and demonstrate an integration of concepts among components.
    • The significance and innovation of the proposed program of research should be assessed in relation to other research in the field, and the relative need for the outputs and outcomes of the program of research. Reviewers should also assess the importance of the short-term and long-term anticipated outcomes, as it relates to the likelihood of significantly advancing health-related knowledge, approaches/methodologies, health care, health systems, and/or health outcomes.
  • Other Considerations (if applicable)
  • International Considerations
    Note that CIHR contributes to, and supports, international research projects and international collaborations to address a range of research areas such as global health research issues, and to contribute to the development of health-research capacity both internationally and domestically. For these types of programs, demonstration of significance should focus on showing that the international component(s) is relevant to the goals and objectives of the program of research, and that the collaboration has the potential to improve health outcomes in the broader global community.

2.2.2 Research Approach (25%)

This sub-criterion is intended to assess the quality of the approach of the proposed program of research.

  • What applicants have been asked to provide
    The applicant will be asked to articulate their research approach (i.e., how they will deliver on the objectives of the program of research) and include:
    • Potential challenges to the approach;
    • Mitigation strategies that will be employed to overcome those challenges;
    • How progress and success of their proposed research program will be measured against key achievements.

    The applicant will not be expected to provide extensive details on established methodologies; however, they should appropriately describe novel/innovative approaches.

    Foundation grants are meant to be flexible in order to allow the Program Leader(s) an opportunity to innovate, and explore new lines of inquiry. A detailed research design and project-by-project plan for each thematically-linked project in the program of research is not expected.

  • Items for reviewers to consider
    • Is the research approach appropriate to deliver on the proposed program objectives?
    • Does the approach allow for flexibility in direction as the program evolves?
    • Does the approach include a high-level description of how progress and success will be measured?
    • Does the approach include a plan for identifying potential challenges and applying appropriate mitigation strategies?
  • Interpretation guidelines
    • The research approach encompasses a number of supporting elements. These elements should be well-planned, appropriate, and poised to deliver on the objectives of the program of research. A detailed research design and project-by-project plan for each thematically-linked project in the program of research is not expected.
    • The approach should be:
      • Flexible enough to explore new avenues of research; and,
      • Poised to maximize contributions through appropriate collaboration and knowledge translation.
    • There should be an appropriate plan to measure progress against key milestones.
    • There should be a high-level plan to identify and address any critical challenges or risks related to the key research and any knowledge translation components of the program. An exhaustive list is not required.
  • Considerations
    The appropriateness of knowledge translation strategies will vary by research field. All knowledge translation strategies should be relevant to the context of the proposed program of research.
  • Mandatory Requirements (where appropriate)
    Evidence demonstrates that biological, economic, and social differences between women and men contribute to differences in health risks, health services use, health system interaction, and health outcomes. Therefore, all applicants to CIHR are expected to integrate gender and sex considerations into their research design, where appropriate, in order to maximize the relevance and applicability of health research findings to both men and women.

Stage 2 - Criterion 2: Quality of the Expertise, Experience, and Resources

2.2.3 Expertise (20%)

This sub-criterion is intended to assess the expertise and experience of the Program Leader(s), as well as the proposed Program Expert(s) (see definition below) to collectively deliver on the objectives of the proposed program. It is the responsibility of the Program Leader(s) to ensure that the proposed research program is poised for success.

  • What applicants have been asked to provide
    The applicant will be asked to outline, at a high-level, the collective expertise, experience and resources being assembled, and how it is appropriate to ensure the delivery of the objectives of the proposed research program. This includes:
    • The expertise and experience (disciplinary, professional, or methodological) of the proposed Program Leader(s), as well as Program Expert(s) (e.g., researchers, technicians, knowledge-users, partners, patients and trainees, etc.);
    • The level of engagement (e.g., time commitment and contribution) of Program Leader(s);
    • As applicable, the commitment (cash or in-kind) from interested or engaged knowledge user(s) and/or applicant partners;
    • The coordinated roles of the Program Leader(s) and any Program Expert(s) in the oversight and management of the program of research;
    • A plan to seek out expertise (new Program Expert(s)) based on the anticipated future needs of the program of research, as it is expected that the Program Experts may evolve over the duration of the grant, based on the needs of the proposed program.
  • Items for reviewers to consider
    • Does the applicant have the appropriate expertise and relevant experience to lead and manage the proposed program of research, considering its objectives and scope?
    • Is there an appropriate complement and level of engagement and/or commitment from key Program Expert(s) and (as applicable) applicant partners?
  • Interpretation guidelines
    • The roles and responsibilities the Program Leader(s) should be clearly described, and linked to the objectives of the research program, referencing the CVs of the proposed Program Leader(s) as required. Please note that if the Program Leader(s) had a career interruption, their application may include a PDF document to supplement the roles and responsibilities information included in their CCV.
      • An overview of expertise assembled to undertake the program of research (i.e., Program Expert(s)); as well as a plan to seek out expertise (new Program Expert(s)) based on the anticipated future needs of the program of research should be outlined. Note that Program Expert(s) will not be submitting a CV with the Stage 2 application.
      • The level of engagement (e.g., time commitment and contribution) of the Program Leader(s) should be appropriate based on the roles and responsibilities described. The coordination of the Program Leader(s) and any Program Expert(s) in the oversight and management of the program of research should be clearly described. There should be no gaps in expertise or ability to lead and deliver on the objectives of the proposed program.
    • The Program Leader(s) should demonstrate an ability to plan for the future by securing a time commitment (e.g., participation in the described research program), a resource commitment (e.g., access to equipment) or a financial commitment (cash or in-kind) from identified Program Expert(s) and/or applicant partners.
  • Considerations

    For any knowledge translation approach, the applicant should have the relevant knowledge user(s) identified. In some cases, the appropriate knowledge user(s) is critical to achieving the desired impact of the research program, and, its ultimate success; hence, an integrated knowledge translation approach may be appropriate. Such an approach would require a commitment to a genuine relationship with the relevant knowledge user(s). Knowledge user(s) may be responsible and accountable for the application/uptake of the program outputs.

    A commitment (cash or in-kind) from interested or engaged knowledge user(s) and/or applicant partners is not an absolute requirement, but can be reasonably expected depending on the nature and type of knowledge user. In some cases, the financial commitment may support a critical step towards the application of the program outputs.

2.2.4 Mentorship and Training (20%)

This sub-criterion is intended to assess the quality of the mentorship/training plan, through the demonstrated commitment to/level of engagement in shaping the future of the applicant’s students, trainees, emerging scholars, and New/Early Career Investigators, as well as other individuals in non-academic, health-related fields.

  • What applicants have been asked to provide
    The applicant will be asked to articulate a mentorship and training plan including:
    • An outline of training goals, learning opportunities, and key activities;
    • Appropriate mentoring/training approaches in relation to the proposed program of research and research field;
    • The rationale for the proposed training approach;
    • Potential challenges of the mentorship and training plan and a strategy for identifying and mitigating these challenges;
    • How progress and success will be measured.
  • Items for reviewers to consider
    • Does the research program include a comprehensive mentorship and training plan for building capacity and positioning students, trainees, knowledge users, emerging scholars, and/or New/Early Career Investigators for successful research careers and/or other career paths in non-academic health-related fields?
    • Does the proposed plan demonstrate an appropriate approach for meeting its objectives in relation to the program of research and the research field?
    • Does the plan include a strategy for identifying and mitigating potential challenges?
  • Interpretation guidelines
    • A comprehensive mentoring and training plan includes a combination of formal education, and informal mentoring, and training approaches. The plan should be coherently presented with training goals, learning opportunities, and key activities clearly articulated. Mentoring/training activities should focus on:
      • Skill development (e.g., technical/methodological, oral and written, teaching, grants management, budgeting, research values and ethics, and, if appropriate, lab management);
      • Career development (e.g., preparation for research or non-academic careers).
    • The mentorship and training plan should be appropriate to the field and career stage of the Program Leader(s), as well as the nature and scope of the program of research, and the specific training level(s) (e.g., graduate students, trainees, emerging scholars, New/Early Career Investigators, and/or health-related professionals). The benefits of the training and mentoring activities to the target audience should be clear. Program Leader(s) are expected to show direct involvement in all mentoring/training activities.
    • The risk management strategy for the mentoring/training component of the program should describe any potential challenges to recruiting, retaining, or mentoring/training graduate students, trainees, emerging scholars, New/Early Career Investigators, and/or health-related professionals (where applicable). There should also be an appropriate high-level plan to address any potential challenges related to the mentoring/training component of the program.
  • Considerations
    Activities and collaboration with individuals from non-academic spheres is an important consideration. Non-academic professionals benefit from both interactions with and access to research resources, and training on use of these resources. The reverse is also true.

2.2.5 Quality of Support Environment (10%)

This criterion is intended to assess whether the applicant has the resources necessary in order to successfully deliver on the objectives of the research program in both the short- and long-term.

  • What applicants have been asked to provide
    The applicant will be asked to outline the resources that they currently have in place to ensure the successful delivery of the research program objectives, including:
    • Physical infrastructure (and/or other types of infrastructure such as consortia, professional networks, etc.)
    • Support personnel
    • Equipment
    • Specialized facilities
    • Supplies
  • Items for reviewers to consider
    • Is the described environment(s) appropriate to enable the conduct of the program of research, and to manage and deliver on the objectives and key components of the proposed research program (e.g., research, knowledge translation, mentoring/training) through the provision of, or access to, the required infrastructure?
  • Interpretation guidelines
    • Depending on the nature of the program, access to environments in other organizations may be essential to deliver on the objectives of the program. Reviewers should consider that appropriate support environments may be found within an institution, or may be built through "networked" environments or platforms outside of the host institution.
    Examples of other environments include, but are not limited to: hospitals, long-term care facilities, experimental facilities (e.g., Canadian Light Source, Conseil Européen pour la Recherche Nucléaire (CERN), etc.), schools, First Nations communities, industry laboratories, prisons, foreign jurisdictions/locations, community populations, and specialized databases.

2.3 Budget Assessment

Reviewers will be required to review the requested budget and justification. Applicants will be asked to submit their budget using a modular template with defined budget increments. Reviewers will be provided with the applicants’ baseline funding amount as calculated by CIHR and as provided to applicants in advance of submitting their Stage 2 applications.

Partner contributions, in cash or in-kind, if applicable, should be considered when reviewing support for the proposed program of research. Partner contributions will be outlined in letters of support and in a Partnership budget module through ResearchNet.

Please note that the budget assessment must not be factored into the scientific assessment.

During the online discussion, Stage 2 reviewers will be required to come to a consensus on their budget recommendations. The Competition Chair may communicate with Stage 2 reviewers if there is a need for clarification. In cases where a discrepancy still exists, the budget will be reconciled at the end of the Final Assessment Stage by the Budget subcommittee.

Stage 2 Reviewer Budget Recommendations
The budget recommendation, including any written comments, is intended to assist CIHR in determining the appropriate funding levels for individual programs of research.

  • What applicants have been asked to provide

    The applicants are asked to provide information regarding the budget request to support the core activities of the proposed program of research, including the total amount requested and the amount requested per annum.

    They need to demonstrate that the amount requested is appropriate to support the proposed program of research, given the context of their baseline funding amount. Any increase above the calculated baseline amount will need to be justified by the applicant.

  • Items for reviewers to consider
    • Is the requested budget appropriate in order to support the proposed program of research? Is it realistic and well-justified, in the context of the applicants' baseline funding amount?
    • If the request is significantly higher than the applicant's historical grant levels, is it appropriately justified?
  • Interpretation guidelines
    • Applicants are required to justify that their request is appropriate in the context of their proposed program of research by providing their total financial requirement for the duration of their grant (5 or 7 years) and an amount for each of the identified categories defined by CIHR with a concise justification for their allocation. A precise valuation of each line item within the budget justification is not expected nor will it be provided by the applicants. Additionally, a yearly breakdown of funds will not be provided.
    • Applicants with funding from CIHR must provide a robust justification for requests that are significantly higher than their historical grant levels.
      • Applicants without a CIHR funding history should justify the appropriateness of their request in the context of their past funding history.
      • New/Early Career Investigators should not necessarily be expected to justify their request within their previous funding history (as it may not be robust enough to support their proposed program) and be given some latitude in their request as they ramp up their program.
      • Justifications for CIHR funding to replace other on-going sources of funding (e.g., health charity, provincial funding agency) are not acceptable.
  • Considerations
    Reviewers are asked to consider the proposed budget of the program, and provide a justified recommendation as to whether they:
    • recommend the proposed budget, as described by the applicant(s);  or,
    • recommend a total reduction in the proposed budget of: $ _______.

    Applicants were provided their baseline amount in their application, as calculated by CIHR:

    • For applicants with prior CIHR funding history, this baseline amount was calculated based on identified eligible funding history.
    • For applicants with no prior CIHR funding history, applicants were provided instructions on what to submit in order for their baseline amount to be calculated.

For more information about how the baseline budgets are calculated, please refer to the ”Enter Budget Information” section of the Foundation Grant - Stage 2 Application and to the Budget Baseline Calculation instructions.

2.4 Online Discussion

Once the preliminary reviews are submitted, the other reviewer's reviews will be accessible through ResearchNet. Reviewers will be able to discuss each of their assigned applications with the other reviewers who were assigned the same application. An asynchronous online discussion tool has been developed to allow reviewers to communicate with each other to share expert perspectives and to discuss any discrepancies.

The objective of the discussion will be to discuss and understand discrepancies in reviews and budget recommendations. There will be no requirement to reach consensus for the scientific discussion but if possible, reviewers are asked to reach a consensus regarding recommendations for the proposed budget. Each reviewer's name and their preliminary review (ratings, written comments, budget recommendation and ranking) will be visible for each application. At this point, if any reviewer has suggested a change in the budget, this should also be discussed. If there are limited discrepancies in reviews, it is possible that a discussion may not be required.

The discussion period will last 72 hours and reviewers will be made aware of the scheduled discussion period for the competition well in advance of the deadlines. Reviewers may log in at any time during the discussion period and will not be expected to participate in online discussions simultaneously.

All reviewers assigned to an application will be able to participate in the discussion as soon as all of the reviews for that application have been submitted. The discussion for all applications will open on the scheduled discussion period start date therefore it is important to respect the established deadlines. Reviewers who have not submitted their preliminary reviews will not be able to participate in the discussion and this will be flagged in their online status.

A Competition Chair will also be assigned to the application to moderate the discussion. The Competition Chair will be tasked with ensuring that discussions take place if warranted, and may prompt reviewers to discuss an application or revise or expand on their review. CIHR staff may also initiate a discussion, or provide input where necessary, to ensure that CIHR policies and procedures are followed.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings, and/or rankings.

It should be noted:

  1. Every reviewer will have the ability to initiate a discussion thread. Every comment will be shared with every reviewer assigned to that application. Once comments are posted, reviewers will not be able to delete or edit them.
  2. Reviewers will be able to post comments for the attention of CIHR staff. These comments will be visible to the other reviewers.
  3. A notification email will be sent daily to advise reviewers of new posts.
  4. As applications are discussed, reviewers may feel the need to update their preliminary ratings and reviews. This can happen at any time both during and after the discussion period. These modifications will be dynamically updated and visible to other reviewers.
  5. A transcript of the online discussions will not be given to the applicants. However, discussion thread content will be retained by CIHR as part of the application record and this record is subject to the Government of Canada’s Access to Information Policy and Privacy Acts.

2.5 Submit the Final Rank List to CIHR

Once the online discussions are complete, reviewers will be asked to finalize their reviews and their individual ranked list. At this point, reviewers must break any ties in their individual ranked list. Once ties are eliminated, reviewers may modify the rank order of their assigned applications by moving the applications up or down the rank list. The original rank order will remain visible as a reference point. Once satisfied with the rank order of applications that they have been assigned, the reviewers will be asked to submit the final rankings to CIHR. Note that reviewers will see the final rank lists of other reviewers. At this point, the adjudication process for Stage 2 is complete.

Once all rankings have been received, CIHR will calculate a consolidated ranking for each application and compile a list of all applications in the competition ranked from highest to lowest. Each application will receive up to five reviews, a consolidated ranking and a standard deviation for Stage 2.

CIHR will determine which applicants will be discussed at the Final Assessment Stage based on the number of applications, consolidated ranking, standard deviation and funds available for the competition. Based on the results from Stage 2, CIHR will identify the highest ranked applications that will be considered for funding at this stage. These applications will constitute the "green zone" and do not require further discussion.

A second set of applications will move to the Final Assessment Stage. These applications will constitute the "grey zone". The "grey zone" applications are those applications that were highly ranked and have a large standard deviation due to discrepancies in reviewer ranking.

The bottom-ranked applications will be deemed unsuccessful, and will not be considered for funding. After the Final Assessment Stage, a notice of decision will be provided to all applicants.

An overview of the Stage 2 adjudication process is outlined in Figure 3.

Figure 3. Stage 2 Overview

Preliminary Reviews of Applications

Reviewers will be asked to provide a rating for each adjudication criterion, and then to justify the rating by providing comments on the strengths and weaknesses. To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale to allow reviewers to indicate finer differences between high quality applications.

The adjudication criteria are:

Criterion 1: Quality of the Program

  • Research Concept (25%)
  • Research Approach (25%)

Criterion 2: Quality of the Expertise, Experience, and Resources

  • Expertise (20%)
  • Mentorship and Training (20%)
  • Quality of Support Environment (10%)

Budget Assessment

Reviewers will be required to review the requested budget and the budget justification. Reviewers will be provided with some historical information about the applicant's open grant research funding history with CIHR. Reviewers will be required to determine if the requested budget is appropriate to support the proposed program of research, if it is realistic and well-justified.

Partner contributions, if applicable, should be considered when reviewing support for the proposed program of research.

Please note that the budget assessment must not be factored into the scientific assessment. Reviewers may recommend that the budget remain as requested or recommend an adjusted amount. If a reviewer adjusts the budget, he/she will be required to provide comments to justify their recommendation.

Online Discussion

Once the preliminary reviews have been submitted, the other reviewer's preliminary reviews (ratings and comments) will be accessible through ResearchNet to reviewers who have been assigned the same application. Reviewers will be able to read the preliminary reviews and discuss each of their assigned applications with the other reviewers who were assigned the same application. This is the reviewers' opportunity to share their expert perspectives, and to discuss any rating discrepancies in a environment.

Reviewers will be required to discuss any discrepancies in their budget recommendations. In cases where a discrepancy still exists, the budget will be reconciled at the end of Stage 2 by CIHR staff and the Competition Chair. The Competition Chair may communicate with Stage 2 reviewers if there is a need for clarification.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include editing their comments, changing their ratings, and/or rankings.

Submit the Final Ranked List to CIHR

A reviewer's set of applications will be ranked in order from highest-rated to lowest-rated based on the ratings given to the application by the reviewer.

At this stage, some applications may be tied (i.e., have the same rank). The reviewer will be required to break these ties by moving one application ahead of the other.

Once the reviewer is satisfied with the ranked list of applications, he/she will submit the ranked list to CIHR. Once all ranked applications have been received, CIHR will compile a list of all applications in the competition ranked from highest to lowest.

The number of applications, consolidated ranking, and the standard deviation of the applications, as well as the funds available for the competition will be used to identify which application will be discussed at the Final Assessment Stage.

Section 3 – Adjudication Process for the Final Assessment Stage

In this stage the Competition chairs who participated in Stage 2 will be asked to participate as members in the multidisciplinary committee covering the full spectrum of health research. The committee will include a Chair and Scientific Officer (SO). This committee will be responsible for integrating the results of the Stage 2 reviews and making recommendations on which "grey zone" applications should be funded.

3.1 Pre-Meeting Activities

Prior to the face-to-face meeting, each committee member will be assigned a subset of applications and each application will be assigned to three committee members. For each application, the committee member will have access to information from Stage 2, including the reviews, the consolidated rankings, standard deviations and the full applications.

  • Consolidated Ranking

    The consolidated ranking (or Consolidated % Rank) is the arithmetic average (or mean) of all reviewers' individual rankings of the application (i.e., the sum of the reviewer’s % ranks divided by the number of reviewers), expressed as a percent rank. In other words, the consolidated ranking represents the average of all reviewers’ percent (%) ranks for a given application.

    Applications are attributed a percent (%) rank from each reviewer, based on the ranking of their assigned list of applications. An application that is ranked first out of a list of applications for that reviewer is attributed a percent (%) rank of 100%. Conversely, an application that is ranked last is attributed a percent rank of 0%.

  • Standard Deviation

    The standard deviation indicates the variability among reviewer rankings. Each application receives a reviewer % rank from each reviewer. Depending on the number of reviewers assigned to the application, an application may have 3, 4 or 5 reviewer % ranks that are factored into the Consolidated % Rank.

    To contextualize the average (or Consolidated % Rank), it is important to also provide the standard deviation for this dataset, which is a statistical measure of how close the data points are to the mean. If the standard deviation is calculated at 0, this indicated that all data points are the same (i.e., 100% reviewer consensus). Therefore, the higher standard deviation would indicate that the data points are spread over a wider range of values.

In order to help committee members differentiate between applications, a binning system will be used. Reviewers will be asked to assign applications to a "Yes" bin (to be considered for funding) or a "No" bin (not to be considered for funding). Each reviewer will be allocated a minimum number of applications that may be placed in the "Yes" and "No" bins. This number will depend on the funds available for a given competition. Committee members will be asked to submit their recommendations to CIHR prior to the meeting.

Based on the "yes/no" binning recommendations reviewers made as part of the pre-meeting activities, CIHR will rank all the Final Assessment Stage applications in order from the highest ranked to lowest ranked. The primary sort criteria will be the number of "Yes" votes received, and the secondary sort criteria will be based on the Stage 2 results.

The Chair of the committee will have an opportunity to review the recommendations from the committee members prior to the meeting. CIHR staff will prepare a schedule for the face-to-face committee meeting in consultation with the Chair and SO. An illustration of the process is provided in Figure 4.

Figure 4. Pre-Meeting Activities Overview

Reviewer assigned a subset of applications.

Reviewer reads the Stage 2 reviews of their assigned applications and the Stage 2 application, if required.

Based on the reviews, the Final Assessment Stage reviewer must choose which applications should be considered for funding and which ones should not. The reviewer will receive a minimum number of applications that they may place in the "Yes" Bin (to be considered for funding), and the rest of the applications will be placed in the "No" Bin (not to be considered for funding).

Based on the yes/no binning recommendations reviewers made as part of the pre-meeting activities, CIHR will rank all Final Assessment Stage applications in order from the highest ranked to lowest ranked. The primary sort criteria will be the number of "Yes" votes received, and the secondary sort criteria is based on ranking after Stage 2 results.

3.2 Face-to-Face Committee Meeting

The multidisciplinary committee will be asked to make recommendations on which applications in the "grey zone" should be considered for funding.

At the meeting, committee members will see the grant applications ordered from highest-rated to lowest-rated according to a combination of the pre-meeting yes/no binning recommendations and Stage 2 results. Based on these results, CIHR will place the applications into one of three groups:

  1. Group A (applications that the assigned committee members recommended should be funded)
  2. Group B (applications for discussion at the meeting)
  3. Group C (applications that the assigned committee members recommended should not be funded)

The meeting will start with the committee Chair asking the committee members to validate the groupings of applications. To validate, the committee Chair will ask if any committee member would like to move any of the proposed Group A or Group C applications into Group B so that they may be discussed by the committee. Once the committee is satisfied with the groupings, the Group A and Group C applications will be locked in and will not be discussed any further at the meeting.

The Chair will moderate the discussion of all Group B applications, and the SO will prepare the notes that summarize the key points discussed for each application.

For each application that is discussed, the following process will be used:

  1. Any committee member in conflict with the application will be asked to leave the room.
  2. The Chair will ask the three reviewers who reviewed the application to indicate whether they voted "Yes" or "No" during the pre-meeting activities.
  3. The Chair will ask the assigned member who was the Competition Chair at Stage 2 to start the discussion and then the other members to continue the discussion. The discussion will focus on the key points that led to their yes/no assessments.
  4. Committee members not assigned to the application will be asked to contribute to the conversation by asking questions and adding their opinions/advice/expertise to clarify points of disagreement.
  5. The SO will read the SO notes, which will capture the key elements of the discussion, to the committee for approval.

Once all of the applications in Group B have been discussed, members will be asked to vote on the Group B applications. Members will be provided with a maximum number of "Yes" votes (to be considered for funding) before the discussions begins. This maximum will be based on both the number of applications in Group B and the estimated funding cut-off line.

The committee Chair will go through the Group B applications one at a time, calling for a vote on each application. Using the electronic voting tool, committee members will vote either "Yes" or "No" for any application with which they are not in conflict. The Chair and SO will not be permitted to vote.

Committee members will be expected to adhere to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations. While committee members will not be in the room during the discussion of the applications with which they are in conflict, they will not be asked to leave during the voting process. They will not be permitted to vote on any in-conflict applications.

The committee will conclude its work after all the members have agreed on the applications they recommend for funding. These recommendations will be summarized and submitted to CIHR for the final review of budgets and the funding decision. Applicants will be informed of the results of the competition once CIHR's President has approved the grants to be funded. After this stage all applicants will be sent a Notice of Decision.

An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, as well as a discussion of policy issues that may have arisen in the course of its deliberations. This discussion is important as it provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the adjudication process.

Applicants, research administrators and reviewers will be asked to complete surveys after each stage to provide their feedback. Feedback received through the surveys will help the continuous improvement of the Foundation Grant Program.

ANNEXES

Interpretation guidelines compendium: sample indicators of Foundation Grant adjudication criteria

In the process of working towards interpretation guidelines of Foundation Grant adjudication criteria, CIHR consulted extensively with researchers from across all of CIHR’s research pillars. Working groups from each pillar contributed to the development of sample indicators of each of the following adjudication criteria: Leadership, Significant Contributions, and Productivity.

The list of sample indicators is provided below. The sample indicators are meant to be illustrative, and in no way represent a comprehensive list. Please also note the following:

  1. The list of sample indicators describes a variety of examples from all of CIHR’s pillars. Therefore, not all of the items in the list will be relevant to each applicant.
  2. The list may be updated over time based on feedback from the research community.
  3. Reviewers will be selected based on their expertise in the research field of the applicant, and CIHR is depending on the reviewers’ expertise to determine whether the applicant has included the relevant indicators for each adjudication criterion, while also taking into consideration the applicant's career stage, field of research, and other relevant conceptual factors.

Annex A - Leadership Sample Indicators

  • Advisor in interdisciplinary or inter-organizational collaborations
  • Editorial experience
  • Innovator in existing or emergent areas of research
  • Leader of research endeavours with major impact
  • Leader of provincial, national and international collaborations, societies, associations, etc. within and outside the research community
  • Leader of community-based research and/or knowledge translation initiatives
  • Number and type of awards, honours, distinctions (note that these vary by research field)
  • Number and type of leadership roles/appointments (e.g., key note speaker, committee, advisory group, network, expert witness, appeals counsel, etc.)
  • Number, type and value of previous grants awarded
  • Number and value of salary awards received
  • Peer review committee service

For New/Early Career Investigators, specific indicators to capture early evidence of leadership competency could also include:

  • Evidence of an emerging profile of innovation (e.g., development of new research methods);
  • Nomination for awards, honours or distinctions;
  • Participation in advisory groups in interdisciplinary or inter-organizational collaborations;
  • Key role in significant research endeavours;
  • Significant involvement in key committees, local, national and international collaborations, societies, associations, etc.

Please note that additional examples of Leadership indicators can be found in the Banting Postdoctoral Fellowships application guide under Significance of Leadership Contributions, as well as in the Vanier Canada Graduate Scholarship selection committee guide under Leadership (potential and demonstrated ability).

Annex B - Significant Contribution Sample Indicators

Note that the type of contributions listed may vary by research field. Note that for Stage 1 applications, the applicant has been asked to describe his/her contributions according to the following headings:

Contributions to the Advancement of Knowledge (Health or other fields)

  • Important advances in understanding (e.g., informing controversies, examining assumptions, advancing education theories, improving theoretical understanding in integrative research);  
  • Significant advances in knowledge (e.g., pathogenesis, cell differentiation, biological pathway, medical imaging, evaluation of cognitive functioning, gene-environment interactions, barriers to medication adherence, synthesis);  
  • Significant impact on a current area of research (e.g., paradigm shifts; challenges to current thinking).

And/or

Contributions to Research (Health or other fields)

  • Development of new areas of research
  • Development of theoretical frameworks/models
  • Development of research methods/techniques
  • Benefit of sustained change in research practice (e.g., ethics considerations)
  • Impact of new research approaches (e.g., multi-disciplinary approaches, horizontal (system-wide) research focus, etc.)

And/or

Contributions to the Foundation of Researchers (Health or other fields)

  • Number and types of domestic and international trainees mentored
  • Increased ability to attract students/trainees/emerging scholars
  • Implementation or broader uptake of innovative mentoring/training approaches
  • Career success of mentees/trainees
  • Success of mentees/trainees in working within both academic and non-academic spheres

And/or

Contributions to Health Care

  • Evidence-informed health care practices
  • Evidence-informed health promotion and prevention programs
  • Evidence-informed treatments, therapies, interventions or disease management approaches
  • Evidence-informed health technologies, tools, diagnostics and devices
  • Increased patient satisfaction
  • Reduced burden of illness

And/or

Contributions to the Health System

  • Evidence-informed policies (health or other) that positively impact health
  • Efficiencies from improved resource allocations or practices (e.g., cost savings, reduced wait times)
  • Improvements in health care data, information management and infrastructure
  • Reduced health inequalities in a specific population
  • Effectiveness of new or improved health care services, practices and regulations
  • Improved health professional education policies or education system
  • Cultural, organizational or pedagogical changes that have improved health professional competency

And/or

Contributions to Health Outcomes

  • Improvements to the health status or quality of life of populations
  • Clinical outcomes of health research interventions
  • Incidence and prevalence of chronic and acute illnesses
  • Changes in culture, attitudes and behaviour that improve health
  • Public health awareness and education (documentaries, classroom/course material, and films)
  • De-bunking stereotypes/myths

And/or

Contributions to Economic Opportunities

  • Business opportunities created
  • Creation of jobs from spin-off companies or from expansion of existing company
  • Sales/revenues generated or licensing returns
  • New products, processes or services commercialized
  • License or royalty agreements

Annex C - Productivity Sample Indicators

  • Publications, peer-reviewed or other (e.g., theses, journal articles, books, book chapters, workshop reports, synthesis reports, dissemination report)
  • Collaborations or networks
  • Communication and knowledge translation products or resources
  • Competency frameworks (e.g., health professionals)
  • Conferences and conference proceedings
  • Consultations
  • Databases and evidence repositories
  • Devices (e.g., medical, robotic)
  • Editorial contributions
  • Intellectual property claims
  • Invention disclosures
  • License agreements
  • Media interviews (e.g., television, radio, print)
  • Networking activities (e.g., workshops)
  • Organization of meetings
  • Patents (filed or obtained)
  • Policy briefs
  • Presentations at public forums
  • Prevention or intervention programs
  • Products
  • Professional practices
  • Programs or services
  • Prototypes
  • Public information, resources, and tools
  • Software and hardware
  • Spin-off companies
  • Standards and guidelines
  • Systems
  • Theories, models or frameworks
  • Tools, techniques, instruments, procedures or methods
  • Training approaches/curricula

The competition process begins at Stage 1 with the applicant submitting a registration followed by a structured application. CIHR conducts an eligibility check and matches applications to reviewers. The reviewers identify their conflicts and their ability to review for a subset of applications. CIHR establishes the assignment of applications to reviewers; approval of the assignments is done by the Competition Chairs. The Competition Chairs assist CIHR in filling expertise gaps by recommending additional reviewers. The reviewers conduct the preliminary reviews of their assigned applications; the review process is monitored by the Competition Chairs via the online discussions. The reviewers complete their final reviews and determine the final rank list of their assigned applications. CIHR then establishes the consolidated rank of all the applications and determines the subset of applications that move to Stage 2.

Applicants who move on to Stage 2 are invited to submit a Stage 2 structured application. CIHR will provide a calculated baseline budget amount that will be included in the Stage 2 application. This process triggers the same competition process as outlined above. CIHR once again calculates a consolidated ranking for each application; CIHR then compiles a list of all applications in the competition ranking from highest (“green zone”) to lowest (“red zone”) to determine which of the Stage 2 “grey zone” applications (i.e., those applications that are close to the funding cut-off and that demonstrate a high degree of variability in reviewer assessments) should be discussed at the Final Assessment Stage (FAS or Stage 3). The FAS review committee is composed of the Stage 2 Competition Chairs.

As part of the FAS, prior to the face-to-face meeting, reviewers identify their conflicts and ability to review for a subset of applications. FAS reviewers then have access to information on the applications assigned to them, including Stage 2 reviews, consolidated rankings, standard deviations, and the full applications. FAS reviewers use a binning system to differentiate between applications by assigning them to a “Yes” bin (to be considered for funding) or “No” bin (not to be considered for funding), and submit their recommendations to CIHR prior to the meeting.

At the face-to-face meeting, the multidisciplinary committee identifies, discusses, and votes on the “grey zone” (also known as Group B) applications they recommend for funding. In cases where a discrepancy in the budget still exists, the budget is reconciled by the Foundation Budget Subcommittee. After the Final Assessment Stage, CIHR reviews the recommendations from the FAS and the Foundation Budget Subcommittee to make its final funding decisions; a notice of decision is provided to all applicants.

An illustration of the peer review process is shown in Figure 5.

Figure 5. Foundation grant peer review process

Date modified: