Postdoctoral Fellow Peer Review Pilot

CIHR is committed to accelerating the research independence and leadership of its postdoctoral fellows (PDFs). To accomplish this goal, CIHR piloted PDF peer reviewer training in the 2016/17 Doctoral Research Award (DRA) competition. The goal of the pilot was to a) give PDFs an opportunity to develop the peer review skills; and b) to assess the feasibility of integrating hands-on peer review experience as a training requirement for all CIHR-supported PDFs.

The first pilot provides preliminary data on using PDFs as reviewers for the doctoral competitions. The follow-up survey indicated that PDFs found the peer review experience would be useful for their future. Detailed results can be accessed through the link below. We are running a second pilot through the 2017/18 DRA competition to validate the results of the first pilot, and to assess the feasibility of integrating peer review experiences for PDFs on a permanent basis.

Results from the Postdoctoral Peer Review Pilot #1


Postdoctoral Peer Review Pilot #1

Background

Banting Postdoctoral awardees were invited to participate as reviewers for the 2016/17 Doctoral Research Award (DRA) competition. The goal of the pilot was to give PDFs an opportunity to develop the peer review skills, and to assess the feasibility of integrating hands-on peer review experience as a training requirement for all CIHR-supported PDFs.

Overall, the results of the pilot indicate that through our peer reviewer training approach, PDFs serve as appropriate reviewers, and that PDFs think that peer review experience is beneficial for their future research careers.

Methods

A total of 47 Banting PDFs were recruited and subsequently trained to act as reviewers. Each application was reviewed by two reviewers as per the DRA competition review process. Each application was assigned a maximum of one PDF reviewer. PDF reviewers were encouraged to reach out to their assigned mentors (with two+ years of prior DRA review experience) and/or CIHR if they had any questions or concerns. A survey was sent to both PDFs and regular reviewers at the end of the pilot to assess the overall experience. Regular reviewers were assigned an average of 9.9 applications and PDF reviewers were assigned an average of 10.5 applications.

Results

CIHR analyzed the competition data to compare the review behaviour of the 'regular reviewers' to that of the 'PDF reviewers.' The analysis was conducted at each of the three review stages:

Table 1 provides an overall comparison of two reviewer groups by review stage:

  Regular-Regular Reviewer Group Regular-PDF Reviewer Group (Pilot)
% of applications approved for funding 12.8% 12.6%
% of applications assigned for re-review 6% 5%
% of applications assigned for 3rd review 1.8% 2.6%

Figure 1 shows the average scores given to applications by both regular reviewers and PDF reviewers.

Figure 1: A comparison of the average weighted scores between regular and PDF reviewers by reviewer group

Figure 1 long description
  Average Weighted Score Standard Deviation
Banting Reviewers (492 Applications) 3.97 0.457
Regular Reviewers (1857 Applications) 4.09 0.384

Figure 2: A comparison of reviewer scores at each stage of the review process (initial, re-review, and 3rd review) within each reviewer group.

Figure 2 long description
Regular-Regular (n=12)
  Average Weighted Score Standard Deviation
Reviewer 1 - Regular (Initial review) 4.64 0.157
Reviewer 1 - Regular (Re-review) 4.60 0.151
Reviewer 2 - Regular (Initial review) 3.86 0.395
Reviewer 2 - Regular (Re-review) 3.89 0.393
Reviewer 3 - 3rd Reviewer Score 4.19 0.162
Regular-PDF (n=13)
  Average Weighted Score Standard Deviation
Reviewer 1 - Regular (Initial review) 4.49 0.335
Reviewer 1 - Regular (Re-review) 4.48 0.277
Reviewer 2 - Regular (Initial review) 4.01 0.429
Reviewer 2 - Regular (Re-review) 4.11 0.316
Reviewer 3 - 3rd Reviewer Score 4.19 0.19

PDF reviewers were asked about their peer review experience through a survey:

"I feel that this peer review experience has helped me with my research career (e.g., grant writing, etc.)"

Figure 3 long description

Percentage response to the opinion question following the peer review pilot project: "Please indicate to what extent you agree with the statement: I feel that this peer review experience has helped me with my research career (e.g., grant writing, etc.)". 100% of the respondents agreed with the statement with more than half (56%) of survey respondents indicating strong agreement.

"Given the opportunity, I would be interested in reviewing for a CIHR competition in the future"

Figure 4 long description

Percentage response to the opinion question following the peer review pilot project: "Please indicate to what extent you agree with the statement: Given the opportunity, I would be interested in reviewing for a CIHR competition in the future". Overall, 96% of survey respondents agreed with the statement, 76% of which agreed strongly.

Date modified: