Evaluation of the Open Operating Grant Program – Long Descriptions
Figure 1-1: Impact of supported papers produced by OOGP-funded researchers vs. OECD health research comparators (2001-2009)
|
2001 |
2002 |
2003 |
2004 |
2005 |
2006 |
2007 |
2008 |
| Supported by OGP Funding |
1.4 |
1.39 |
1.4 |
1.48 |
1.46 |
1.44 |
1.47 |
1.63 |
| Canada |
1.23 |
1.21 |
1.2 |
1.19 |
1.21 |
1.21 |
1.25 |
1.31 |
| Australia |
1.15 |
1.12 |
1.12 |
1.16 |
1.16 |
1.17 |
1.22 |
1.26 |
| France |
0.98 |
1.01 |
1 |
1.02 |
1.06 |
1.06 |
1.14 |
1.16 |
| Germany |
1.07 |
1.07 |
1.08 |
1.09 |
1.12 |
1.11 |
1.18 |
1.23 |
| Netherlands |
1.3 |
1.28 |
1.32 |
1.32 |
1.35 |
1.32 |
1.41 |
1.44 |
| Switzerland |
1.41 |
1.39 |
1.39 |
1.41 |
1.48 |
1.4 |
1.48 |
1.57 |
| United Kingdom |
1.21 |
1.25 |
1.24 |
1.25 |
1.27 |
1.26 |
1.3 |
1.41 |
| United States |
1.33 |
1.33 |
1.32 |
1.3 |
1.3 |
1.3 |
1.34 |
1.36 |
« Back to Figure 1-1
Figure 1-2: Scientific impact of OOGP-supported research papers (ARC)
|
Average Relative Citation |
| 2000-2005 ARC score |
1.44 |
| 2006-2009 ARC score |
1.54 |
| Overall ARC score (2000-2009) |
1.5 |
« Back to Figure 1-2
Figure 1-3: Journal article productivity per year of grant duration by pillar of respondent
| 1st Research Theme/Pillar |
Average number of publications per year of grant |
| Biomedical Research |
2.34 |
| Clinical Research |
2.03 |
| Health Systems & Services Research |
1.31 |
| Soc, Cultural, Enviro, Popln Hlth Research |
2.38 |
| Total |
2.26 |
« Back to Figure 1-3
Figure 1-4: Publication behavior by grant duration - When do supported researchers publish?
| CSY |
CSY+0 |
CSY+1 |
CSY+2 |
CSY+3 |
CSY+4 |
CSY+5 |
CSY+6 |
CSY+7 |
CSY+8 |
CSY+9 |
| All Durations (N=450) |
7.52% |
12.31% |
16.77% |
20.37% |
17.53% |
11.99% |
6.37% |
3.82% |
2.12% |
0.58% |
| 5-year Grants (N=72) |
10.81% |
15.45% |
15.31% |
19.24% |
17.28% |
11.94% |
6.18% |
2.25% |
0.84% |
0.14% |
| 3-year Grant (N=297) |
6.93% |
11.95% |
17.73% |
21.42% |
17.09% |
11.78% |
5.66% |
3.75% |
2.54% |
0.69% |
« Back to Figure 1-4
Figure 1-5: Books/book chapters and reports published as a result of OOGP grants
| 1st Research Theme/Pillar |
Books/Book Chapters Published |
Reports/Technical Reports Published |
| Biomedical Research |
.99 |
.24 |
| Clinical Research |
1.05 |
.01 |
| Health Systems & Services Research |
.08 |
.38 |
| Soc, Cultural, Enviro, Popln Hlth Research |
1.13 |
.55 |
| Total |
.96 |
.25 |
« Back to Figure 1-5
Figure 2-1: Impact of applicants for two years following competition by application status and Canadian papers in health fields by publication year (2000-2009) (ARCs)
|
2000 |
2001 |
2002 |
2003 |
2004 |
2005 |
2006 |
2007 |
2008 |
2009 |
| Successful applications |
1.69 |
1.5 |
1.38 |
1.46 |
1.57 |
1.55 |
1.56 |
1.51 |
1.74 |
1.64 |
| Unsuccessful applications |
1.43 |
1.47 |
1.32 |
1.45 |
1.52 |
1.39 |
1.4 |
1.44 |
1.53 |
1.58 |
| Applications by researchers who are never funded |
1.39 |
1.25 |
1.47 |
1.45 |
1.32 |
1.2 |
1.39 |
1.45 |
1.47 |
1.2 |
| Canada |
1.24 |
1.21 |
1.19 |
1.19 |
1.18 |
1.2 |
1.2 |
1.25 |
1.33 |
1.33 |
« Back to Figure 2-1
Figure 2-2: Average Relative Citations of supported papers of funded researchers by peer review committee percentile ranking and publication year (2001-2009) (ARCs)
|
2001 |
2002 |
2003 |
2004 |
2005 |
2006 |
2007 |
2008 |
2009 |
| Always Top Ranked (n=74) |
0.66 |
2.12 |
1.34 |
1.68 |
1.86 |
2.07 |
1.8 |
2.11 |
2.05 |
| Sometimes Top Ranked (n=470) |
1.65 |
1.47 |
1.57 |
1.55 |
1.54 |
1.61 |
1.64 |
1.75 |
1.7 |
| Never Top Ranked (n=956) |
1.4 |
1.24 |
1.21 |
1.41 |
1.4 |
1.22 |
1.35 |
1.58 |
1.5 |
« Back to Figure 2-2
Figure 2-3: Previously funded status of OOGP renewal applications with FRNs, by Competition (2000-2010)
|
First version |
With more than one version |
| FY 2000-2001 |
48.97% |
51.03% |
| FY 2001-2002 |
54.40% |
45.60% |
| FY 2002-2003 |
55.56% |
44.44% |
| FY 2003-2004 |
66.41% |
33.59% |
| FY 2004-2005 |
66.84% |
33.16% |
| FY 2005-2006 |
68.48% |
31.52% |
| FY 2006-2007 |
75.44% |
24.56% |
| FY 2007-2008 |
76.81% |
23.19% |
| FY 2008-2009 |
76.17% |
23.83% |
| FY 2009-2010 |
76.44% |
23.56% |
« Back to Figure 2-3
Figure 2-4: Concordance between independent and committee stage assessments
|
Bottom 20% at committee stage |
60-80% at committee stage |
40-60% at committee stage |
20-40% at committee stage |
Top 20% at committee stage |
| Was in top 20% at Independent stage |
0% |
1% |
4% |
22% |
75% |
| Was in 20-40% at Independent stage |
1% |
6% |
24% |
52% |
21% |
| Was in 40-60% at Independent stage |
5% |
21% |
49% |
22% |
3% |
| Was in 60-80% at Independent stage |
16% |
55% |
21% |
4% |
1% |
| Was in bottom 20% at Independent stage |
78% |
17% |
2% |
1% |
0% |
« Back to Figure 2-4
Figure 2-5: Origins of top 5% of committee stage applications
|
Top 20% at independent stage |
20-40% at independent stage |
40-60% at independent stage |
60-80% at independent stage |
| Top 5% at committee stage |
94.59% |
4.69% |
0.60% |
0.12% |
« Back to Figure 2-5
Figure 2-6: True Positive applications perform better than the other three comparison groups
|
Average Relative Citation |
| True negative |
1.45 |
| False positive |
1.59 |
| False negative |
1.54 |
| True positive |
1.63 |
« Back to Figure 2-6
Figure 3-1: Percent of funded researchers saying research results have had impacts
| Percent of respondents who said research had… |
Total |
| % |
| Health system/care consumer level impacts |
22.1% |
| Health system/care organization level impacts |
26.0% |
| Health system/care provider level impacts |
34.4% |
| Generated subsequent research by others |
44.2% |
| Generated subsequent research by project team members |
88.2% |
« Back to Figure 3-1
Figure 3-2: Percent saying OOGP research has influenced stakeholders to a considerable or great extent
Community/municipal organizations
|
0.3% |
|
1.7% |
| Federal and provincial reps |
1.7% |
Health system/care manager
|
1.7% |
| Health care professional orgs |
2.3% |
Media
|
2.5% |
| Patients/consumers |
4.5% |
| Health system/care practitioners |
6.2% |
Industry
|
8.7% |
| Formal stakeholders |
26.5% |
| Researchers/academics |
61.4% |
« Back to Figure 3-2
Figure 4-1: Average number of full-time equivalent trainees per year of grant by competition year
| Year of competition |
Competition year 2000 |
Competition year 2001 |
Competition year 2002 |
Competition year 2003 |
Competition year 2004 |
| Total average per year |
1.54 |
1.38 |
2.43 |
1.73 |
3.81 |
« Back to Figure 4-1
Figure 4-2: Proportion of annual expenditure on the OOGP by pillar 2000/01 – 2010/11
| Expenditures (New methodology) |
2000-2001 |
2001-2002 |
2002-2003 |
2003-2004 |
2004-2005 |
2005-2006 |
2006-2007 |
2007-2008 |
2008-2009 |
2009-2010 |
2010-2011 |
| Biomedical |
61% |
74% |
79% |
81% |
82% |
82% |
81% |
80% |
80% |
79% |
78% |
| Clinical |
5% |
7% |
7% |
7% |
7% |
7% |
8% |
8% |
9% |
10% |
10% |
| Health Systems & Services |
1% |
2% |
3% |
3% |
3% |
3% |
3% |
4% |
4% |
4% |
4% |
| Social, Cultural, Environmental or Population Health |
2% |
2% |
4% |
5% |
6% |
6% |
7% |
8% |
7% |
7% |
7% |
| Not Specified |
31% |
14% |
8% |
4% |
2% |
1% |
0% |
0% |
0% |
0% |
0% |
« Back to Figure 4-2
Figure 4-3: Barriers and challenges to Pillar III/IV applicants in the OOGP
There are three broad categories of barriers and challenges faced by Pillar III/IV applicants to the OOGP. These are (a) diverse projects cutting across disciplines and methodologies; (b) higher proportion of applications rated as "unfundable" and lower average peer review scores; and (c) renewal behaviours.
« Back to Figure 4-3
Figure 4-4: Likelihood of an application being deemed non-fundable (odds ratios)
|
Likelihood |
| Theme IV |
2.14 |
| Theme III |
2.15 |
| Theme II |
1.68 |
| Theme I |
1 |
« Back to Figure 4-4
Figure 5-1: Open Operating Grant Program: Number of applications and success rates, 2000-2010
|
Funded |
Fundable, Not Funded |
Not Fundable |
Successe Rate (%) |
| 2000-01 |
801 |
500 |
1,077 |
34% |
| 2001-02 |
833 |
604 |
1,104 |
33% |
| 2002-03 |
935 |
723 |
1,217 |
33% |
| 2003-04 |
880 |
915 |
1,284 |
29% |
| 2004-05 |
911 |
915 |
1,096 |
31% |
| 2005-06 |
957 |
1,167 |
1,241 |
28% |
| 2006-07 |
916 |
1,382 |
1,374 |
25% |
| 2007-08 |
829 |
1,743 |
1,322 |
21% |
| 2008-09 |
816 |
1,568 |
1,241 |
23% |
| 2009-10 |
797 |
1,599 |
1,284 |
22% |
« Back to Figure 5-1
