Matching applications to reviewers

An important step in the peer review process is matching the right applications to the right reviewers. For the Project Grant: Fall 2016 competition, CIHR is embarking on a new matching process for the Conflict and Ability to Review step. In this podcast, CIHR Associate Vice-President Jeff Latimer provides an overview of how the matching is done and what reviewers can expect from the process.

Note: This matching process was not used for the 2016 Foundation Grant competition. It is being used for the first time with the Project Grant: Fall 2016 competition.

Transcript

Jeff Latimer: Hi, my name’s Jeff Latimer and I’m the new Associate Vice-President here at CIHR responsible for program operations. Today, I thought it was important to provide some information on current Project grant competition. This week, we’re at a phase called the Conflict and Ability to Review Process, and I want to be clear on that process.

So there are multiple steps in the Conflict and Ability to Review Process. First and foremost, CIHR will conduct its matching exercise. The matching exercise has two sets of data that we use. First, we have the data from the applications themselves wherein applicants have told CIHR information on their application and the best types of reviewers that we should identify. We also have the same identical information on the Reviewer Profile Database wherein each reviewer has flagged those areas of expertise that they are comfortable reviewing in. We use our matching process to identify a series of applications for each reviewer.

The very first step is CIHR will be sending out between 30 and 50 applications to each and every reviewer that has agreed to review within the Project grant competition. We will ask those reviewers to conduct the Conflict and Ability to Review process. First, we’ll ask you to identify any applications you believe you may be in conflict with. Secondly, we’ll ask you to identify those applications where you feel comfortable reviewing on behalf of CIHR.

The review process this time will ask you to declare your ability to review the entire application or certain elements of the applications, such as the area of science, the methodology or the population being studied. Once you complete your Conflict and Ability to Review process, CIHR will analyze the data and develop a set of assignments. We will be assigning four reviewers to each application and 8-12 applications for each reviewer. We will then ask the competition chairs to validate and approve those assignments so that everybody is comfortable with the process.

Once you receive your 8-12 applications and you identify a certain application you changed your mind on or do not believe you now have the expertise to review, we would ask you to not review that application but please contact CIHR so that we can replace it on your behalf.

So I hope that this process is clear to everybody, and you should expect to receive 30 and 50 applications this week to declare your Conflict and Ability to Review.

The matching process – A snapshot

Long description
  1. In their application, applicants identify the expertise required to review their proposal(s).
  2. Through their Reviewer Profile, reviewers identify their area(s) of expertise.
  3. Based on this information, reviewers are matched with approximately 30-50 applications to conduct the Conflict and Ability to Review (CAR) process.
  4. Through the Conflict and Ability to Review (CAR) process, reviewers indicate which applications they can fully review, partially review, or not review at all (due to conflict or not enough expertise).
  5. Once the CAR process is complete, the Project Grant Competition Chairs review and approve the assignment lists.
  6. Reviewers are ultimately assigned 8-12 applications to conduct their full Stage 1 reviews.

Important points to remember

At this time, only reviewers with completed Reviewer Profiles are being considered for the Conflict and Ability to Review (CAR) exercise.

CIHR screened for potential conflicts of interest before the CAR exercise. This step ensures that reviewers are not assigned applications on which they are listed as the Nominated Principal Applicant (NPA) or Principal Applicant (PA). Please note, however, that it is possible for a reviewer to receive an application for which they are listed as a Collaborator or Co-Applicant. As CIHR only has the ability to screen for NPA and PA conflicts, reviewers listed as a Collaborator or Co-Applicant on an application are asked to self-identify their conflict during the CAR exercise.

For the CAR exercise, CIHR will send reviewers the name(s) of the applicant(s), the title of the project, and the abstract. Reviewers will only receive the full applications when the assignments for the Stage 1 review are released. It is possible for a reviewer to read a full application at that step and decide that they do not, in fact, have the appropriate expertise to review that application—or they may realize that they have a conflict that was not identified during the CAR exercise. If this happens, reviewers should contact CIHR immediately so that the application can be removed from that reviewer’s list and re-assigned appropriately.

Date modified: