Canadian Institutes of Health Research Salary/Career Award Programs - Evaluation Report

CIHR Evaluation Unit 2012

Table of Contents

Key Findings, Conclusions and Recommendations

Key findings

1. Performance

Evaluation Question

The extent to which the CIHR Salary Programs achieved their expected outcomes and identify opportunities for improvement to program design/delivery

1.1 Protected time to conduct research
1.2 Researcher productivity
1.3 Research outcomes
1.4 Training and supervision
1.5 Obtaining other sources of funding
1.6 Retention and sustaining a career in health research
1.7 Career trajectory
1.8 Opportunities for improvement in program design/delivery
1.9 Efficiency and economy

Evaluation Question

To determine the degree of efficiency and economy associated with the CIHR salary programs

2. Relevance

2.1 Extent to which the CIHR salary programs address the needs of Canadian health researchers

Evaluation Question

To determine the extent to which the CIHR Salary Programs continue to address the needs of Canadian health researchers

2.2 Alignment with government priorities and the role and responsibilities of delivering salary support to health researchers in Canada

Evaluation Questions

Conclusions

Delivery and program outcomes

Program design

Recommendations

  1. In the context of CIHR's program redesign, consider whether operational efficiencies and equivalent or greater research impacts could be achieved by replacing the new investigator open salary awards with operating grants targeted specifically at this group.
  2. If salary/career award programs are retained at CIHR, the current design issues with protected time should be addressed.
    • One option would be to better specify how protected time for researchers can be measured, monitored and reported to demonstrate compliance with the terms of the award.
    • A second option would be to remove the guidelines on the target amount of time to be spent on research as a condition of holding a salary award. Other approaches that could be more effective at achieving the same objective should be considered, for example, specifying a maximum number of teaching hours, clinical hours or administration.

Management Response

Management agrees with the findings and recommendations of this evaluation. CIHR is committed to investing in world class health research and more specifically to train, retain and sustain a healthy research foundation. New investigators play an important role in creating a sustainable foundation for Canadian health research. CIHR's New Investigator program accounted for 58% of the total financial commitments for salary programs from 2000 to 2010.

This evaluation was timely given the current reforms of CIHR's suite of open programs. The challenge will be to ensure that new investigators continue to be given a fair and equal opportunity for CIHR support. Management intends on using the results of this evaluation to influence these reforms. To address the recommendations noted in the report, the Research Portfolio Management Team along with the Health Research Roadmap Implementation Team will implement the following management actions:

Recommendation Response
(Agree or Disagree)
Management Action Plan Responsibility Timeline
1. In the context of CIHR's program redesign, consider whether operational efficiencies and equivalent or greater research impacts could be achieved by replacing the new investigator open salary awards with operating grants targeted specifically at this group. Agree Specific funding targets for new investigators in the new program funding schemes will be considered as part of CIHR's suite of open programs reform. Associate Vice-President, Research Portfolio Implementation target is Summer 2013
2. If salary/career award programs are retained at CIHR, the current design issues with protected time should be addressed. Agree Associate Vice-President, Research Portfolio Implementation target is Summer 2013
a. One option would be to better specify how protected time for researchers can be measured, monitored and reported to demonstrate compliance with the terms of the award. The criteria for measuring protected time related to strategic salary/career awards will be revised to focus on more applicable metrics such as establishing a maximum of allowable time spent on teaching, clinical or administration duties.
b. A second option would be to remove the guidelines on the target amount of time to be spent on research as a condition of holding a salary award. Other approaches that could be more effective at achieving the same objective should be considered, for example, specifying a maximum number of teaching hours, clinical hours or administration. This would be applicable to strategic salary/career awards only since the bulk of the CIHR Salary/Career Awards will be redesigned as noted in recommendation

1. CIHR Salary and Career Awards Program Profile

1.1. Program Context

The Canadian Institutes of Health Research (CIHR) contributes to Canada's health research capacity by providing a range of salary/career awards that target researchers at different stages of their careers. As is outlined in CIHR's strategic plan, the Health Research Roadmap, a key goal for the agency is to train, retain and sustain outstanding health researchers (CIHR, 2010). This includes a focus on attracting and retaining the best early career researchers, or new investigators, as well as other targeted groups such as clinician scientists.

The CIHR salary support programs operate within a wider Canadian funding landscape of similar awards. This includes large flagship federal funding programs such as the Canada Research Chairs Program1, as well as a wide range of provincial and not-for-profit salary awards.

It is important to consider CIHR's salary/career award programs within their wider context. The organizational scan presented as part of this evaluation shows that several funders have in fact discontinued their salary/career awards. Evaluation findings also show that salary award funding is rarely a zero-sum game; researchers who are unsuccessful in their applications to CIHR are frequently provided with salary support by other provincial or not-for-profit organizations.

1.2. Policy and Program Objectives

Salary/career awards programs are intended to provide investigators who hold research appointments at Canadian universities and institutions with the opportunity to develop and demonstrate their independence in initiating and conducting health research through the provision of a contribution to their salary.

Salary/career awards are offered through both open and strategic competitions. In an open competition, applicants apply to conduct investigator-driven health research in an area of their choice. In strategic competitions, CIHR offers salary/career support to investigators whose research areas are aligned with priorities identified by CIHR and its thirteen institutes.

Program theory is a model of how an intervention, such as a program, contributes to a chain of results and finally to intended or observed outcomes (Funnell & Rogers, 2011). It can be noted that the existing literature on the program theory of salary/career awards is extremely sparse. While some limited literature is available in relation to capacity building in health research more generally, even here there is a need for greater empirical evidence. As Bates et al (2006) note: "Capacity building is a poorly defined and understood concept" (p1).

The key features of the program theory for salary/career awards are shown in Table 1. Many of these are premised on the researcher having 'protected time' to conduct research (instead of teaching or other activities). This protected time is intended to bring about positive outcomes in terms of research production and training, ultimately leading to a strong research community and the retention of excellent researchers in Canada. A logic model for the salary/career awards, along with a detailed description of its components, can be found in Appendix 2 of this report.

Table 1: Salary/Career support awards program theory

Key elements of program theory Description
Protecting time dedicated to research Institutions must commit to enabling researchers to have a minimum of at least 75% protected research time for the duration of their award
Career progression Receiving an award should enable new investigators to launch their careers in health research. The award may also encourage established researchers in other disciplines to reorient their careers toward the field of health research
Research productivity With more time devoted to research, researchers would be expected to produce a greater number of publications and engage in knowledge translation activities
Researcher prestige Success in obtaining a CIHR salary award and the resulting prestige should enable researchers to secure additional funds and participate in additional research projects
Training of Highly Qualified Personnel (HQP) The activities undertaken by CIHR salary award holders should result in the training of research trainees, students and other HQP engaged in research projects
Strong research community Salary support enables institutions to retain talented researchers who in turn train highly qualified personnel. Mentoring and training facilitate new researchers' entry in the field, thereby increasing research capacity (Bryar, 2010). The literature regarding capacity building supports this theory: a critical mass of researchers and an atmosphere conducive to research are required (Segrott at al., 2006)
Retention of excellent researchers in Canada The funds available and the protected time for research offered through these awards may influence awardees in their decision to remain committed to conducting health research in Canada.

1.3. Program Description

As of 2010, the following CIHR salary/career award programs were active:

As of October 2010, CIHR's salary/career award programs commitments have accounted for $354.4 million since 2000.

As shown in Table 2 below, New Investigator awards are by far the largest of the salary/career awards programs. The New Investigators program represents the largest total investment, accounting for approximately 58% of the total salary programs financial commitments for the period 2000-2010. The Investigators program (12% of financial commitments), CIHR Research Chairs, (9% of financial commitments), and Clinician Scientists Phase II (4% of financial commitments) represent the majority of the remaining available funds.

Table 2 also shows that many of these programs have been discontinued over the last ten years, including all of the senior investigator awards. A detailed description of each of the salary/career award programs can be found in the appendices to this report.

Table 2: CIHR Salary/Career Award Programs - lifespan, number of new paid awards and total financial commitments

Program category Program name Program lifespan Total new paid awards since program 00/01 Total financial commitments (new and ongoing) since 00/01*
(in millions)
Total 1258 $354.4
New investigators New Investigators 00/01- Ongoing 760 $204.4
Clinician Scientist Phase 2 00/01- Ongoing 67 $15.1
Senior Fellowship Phase 2 00/01- 06/07 42 $7.5
Career Awards 00/01- 05/06 21 $3.9
Clinical Investigators 2002- Ongoing 12 $2.9
CIHR/NHRDP Salary Support Awards 01/02-02/03 14 $1.6
Partnership Appointment 00/01-03/04 10 $2.2
Investigators Investigators 00/01- Ongoing 147 $44
Mid-Career Awards 2003 - Ongoing 10 $3.5
Research Chairs 00/01- Ongoing 68 $32.2
CADRE- Career reorientation Awards 00/01- 09/10 9 $0.4
CADRE- Chairs 00/01- One competition 13 $9.3
Senior Investigators Distinguished Investigators 01/02- 02/03 9 $3.8
Senior Investigators 00/01- 07/08 57 $20.3
Distinguished Scientist 00/01-01/02 8 $1.9
Other Salary Awards Career Transition Awards 01/02- 08/09 11 $1.4
Source: CIHR Electronic Information System (EIS) database

The relative importance of the salary/career awards within CIHR's overall expenditures has decreased over time. As shown in Figure 1 below, in 2000-2001, salary awards represented 8.6% of CIHR's grants and awards expenditures; in 2009-2010 these programs accounted for only 3.7% of total grants and awards spending. This decrease can be attributed to the elimination of the Senior Investigator and Open Investigator awards in June 2003, and other agency-wide budgetary adjustments.

In keeping with the decline in budget allocation, the number of awards has also been decreasing. The number of new paid awards in 2009/10 is less than half of the 2000/01 total, decreasing from 181 awards in 2000/01 to 82 in 2009/10.

Figure 1: Percentage of annual salary awards expenditures as a proportion of the total CIHR grants and awards budget2 and number of newly paid awards

Bar and line graph showing percentage of annual salary awards expenditures as a proportion of the total CIHR grants and awards budget and number of newly paid awards

Figure 1 long description

Source: CIHR Electronic Information System (EIS) database

2. Evaluation Scope and Design

2.1. Evaluation Objectives and Scope

The evaluation objectives were designed in consultation with CIHR management and in alignment with the TB Directive on the Evaluation Function3. The following objectives were identified as a priority for this evaluation:

Performance

  1. To assess the extent to which the CIHR Salary Programs achieved their expected outcomes and identify opportunities for improvement to program design/delivery;
  2. To determine the degree of efficiency and economy associated with the CIHR Salary Programs.

Relevance

  1. To determine the extent to which the CIHR Salary Programs continue to address the needs of Canadian health researchers;
  2. To determine the degree of alignment of the CIHR Salary Programs with Canadian government priorities;
  3. To determine the role and responsibilities of CIHR in delivering salary support to health researchers in Canada.

A complete list of evaluation issues, questions and their corresponding indicators can be found in the appendices to this report.

2.2. Methodology

Consistent with TBS guidance and recognized best practice in evaluation, a range of methods were employed to triangulate the evaluation findings. To ensure that the evaluation findings and conclusions are robust and credible, the evaluation used multiple methodologies and utilized both quantitative and qualitative evidence.

The following lines of evidence and data sources were used:

  1. CIHR Salary Support Program database analysis and document review – a review of documents and administrative data was conducted to ensure a thorough understanding of the historical development of the program.
  2. Organizational Scan- a scan of Canadian health research funders that offer salary support, to assess the degree of duplication or overlap of CIHR Salary/Career Award Programs.
  3. Literature review- a review of the literature to explore salary award program theory and alternative program design and delivery.
  4. Field observation of the Peer Review process- field observation to assess the degree of consistency between program objectives and the review criteria used by peer reviewers to assess applications.
  5. Quantitative survey online surveys administered between October 28th and November 30th, 2010 ( a period of approximately 5 weeks):
    • Survey of funded salary awardees (N=1007): All researchers who received a CIHR Salary Award between 2000 and 2009 and who kept their award for a period longer than one year were invited to participate in the survey; valid response rate of 53%.
    • Survey of non-funded CIHR salary award applicants (N=777): Researchers who applied to the CIHR salary support programs, who were fundable (rated over 3.5 in the peer review process) but who did not receive an award due to budgetary constraints; valid response rate of 39%
  6. Qualitative interviews with a range of stakeholders – interviews were conducted with the following stakeholder groups:
    • CIHR senior management, Institute and program staff (N=9) – key informants were selected according to roles, responsibilities and familiarity with CIHR salary/career awards;
    • Vice Presidents of Research in Canadian universities and research centres (N=20);
    • Representatives of other Canadian health research funding agencies and NGOs (N=6) – sample based on the results of the organizational scan;
    • Partners (N=4) – partners chosen based on their financial contributions to salary programs.

Further methodological details, the survey questionnaires and interview guides used for data collection and information on the data analyses that were conducted can be found in the evidence binder for this evaluation that accompanies this report.

2.3. Limitations

In keeping with best practices in program evaluation, the limitations of this study are noted below, together with the strategies that were employed to mitigate these.

3. Achievement of expected outcomes

Evaluation Question

1. The extent to which the CIHR Salary Programs achieved their expected outcomes and identify opportunities for improvement to program design/delivery

This section details the extent to which the Salary Support Programs have achieved their expected outcomes. The following areas are assessed:

3.1. Protected time to conduct research

A key objective of the CIHR Salary Awards is to ensure that funded researchers spend 75% of their time (no less than 30 hours per week) conducting research.

A first finding is that while the salary award program documentation states that 75% of researcher time should set aside for research, there is no definition in the program documentation as to exactly what falls within this category. This makes it difficult to accurately assess the extent to which this condition is actually being met. There is also no requirement for award holders to report on the amount of time they spend on research either during the period they hold the award or at the end of this.

To mitigate the issue with definitions of research, the findings in Table 3 below show both a 'broad' and a 'narrow' interpretation of what constitutes research activities. Consultations with stakeholders resulted in two definitions: a broad definition which includes the top three categories in Table 3 (research activities; administrative tasks associated with research; training and supervision of students/research staff) and a narrow interpretation which includes only the first category (research activities).

Regardless of which definition is used, holding an award does not appear to have a substantive impact on how salary funded researchers distribute their time between different types of activities (see Table 3:

Table 3: Protecting time to conduct research

Average percentage of time spent by CIHR funded researchers on specific tasks before and while holding a CIHR Salary Award (n=531)
Task/Activity Before holding award While holding award
Research activities associated with your research (e.g. conducting trials, working in labs, collecting samples, conducting data collection, etc.)* 71% 44% 77% 47%
Administrative tasks associated with your research program (e.g. proposal writing, ethics review of research protocols, financial management, etc.) 14% 14%
Student and research staff training and supervision (e.g., lab meetings, project supervision, mentoring, reviewing theses and papers, etc.)* 13% 16%
Teaching duties (e.g., course preparation, marking and office hours, Curriculum development)* 13% 10%
Institutional administration (committee work, such as advising, theses, tenure, recruitment, etc.) and administrative positions (professor, dean, chair of department, etc.) 7% 6%
Clinical work* 8% 6%

* Statistically significant at p-value =0.05; Statistical tests: non-parametric for paired groups: Wilcoxon Signed Rank test

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

If the broad definition of research is used, funded CIHR salary award holders spend a similar proportion of their time on these activities as non-funded applicants; 77% compared with 74%. A greater difference is observed if the narrow definition is used: funded researchers spend 47% of their time on research compared with 35% for the non-funded group.

The similarities between these groups are perhaps unsurprising if the wider context of Canadian research funding is considered. As is detailed in this report, many researchers who did not receive CIHR salary/career award funding still obtain this form of funding from other organizations. It should also be remembered that salary awards are only one source of support for researchers; many also concurrently hold operating grants to conduct their research.

3.2. Researcher productivity

A key indicator used to measure outcomes of the CIHR salary awards is the research productivity of funded researchers. The hypothesis is that researchers who spend more time conducting research rather than on other activities such as teaching duties will produce more publications (e.g. articles, books or book chapters) and engage in more activities that promote their research such as conferences or presentations.

3.2.1. Refereed journal articles

CIHR salary awardees publish a greater number of refereed journal articles than those in the non-funded salary award applicant comparator group:

In terms of other benchmarks, salary funded researchers' productivity in publishing refereed journal articles is similar to that of Canada Research Chair (CRC) holders: an annual average of 5.9 publications per researcher for both programs4. This comparison should of course be put in context; the CRCP is one of the most prestigious salary awards available to Canadian researchers, particularly for those who are awarded Tier 1 chairs.

A comparative analysis of the number of referred journal articles published by CIHR salary award holders funded by an open competition compared with those receiving a strategic salary award showed no significant differences between the groups:

Statistical analyses of these data were conducted to assess the extent to which the number of papers that researchers publish can be linked to other factors.

First, correlation analysis was used to test associations between the number of publications and the following variables: percentage of time spent on research; dollar amount of awards held; dollar amount of grants held and number of trainees involved in the program. These variables were found to be positively correlated with the number of publications produced.

Multiple regression analysis was then used to test the influence of these variables on the number of publications produced by researchers. Results showed that only the number of trainees had a statistically significant influence; this influence was very limited, explaining only eight percent of the variability in the number publications produced.

An acknowledged limitation of all of these analyses of publication outputs is that while the findings allow us to assess the productivity of researchers, we are unable to assess the scientific impact of their publications, for example through a citation analysis. Bibliometric analyses are frequently used to assess the contributions scholars make in their research publications to advancing valid scholarly knowledge (e.g. Moed, 2005). It is recommended that future evaluations of the salary support programs include this methodology as a line of evidence.

3.2.2. Other research outputs

CIHR funds diverse health research communities in the biomedical, clinical, population health and health systems and services fields and it is important to acknowledge that peer reviewed journal articles are not always the best measure of output for all groups. Some social science researchers may, for example, focus to a greater extent on publishing books or book chapters than on peer reviewed journal articles.

As illustrated in Figure 2 below, CIHR salary awardees produce a greater average number of books/book chapters (2.1 vs. 1.4), reports/technical reports (1.4 vs. 0.8) and Master's thesis/doctoral dissertations (2.3 vs. 1.8) compared with non-funded applicants.

Figure 2: Average research outputs of funded and non-funded salary award applicants for the duration of a salary/career award
Radar chart showing average research outputs of funded and non-funded salary award applicants for the duration of a salary/career award

Figure 2 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.3. Research Outcomes

Funded health research can result in a wide range of outcomes, from knowledge creation through new methods, theories or practices to the development of spin-off companies. From 2011 onwards, CIHR's new end of grant/award Research Reporting System (RRS) will be capturing these outcomes for its funded researchers. The salary support questionnaire administered to both funded and non-funded researchers for this evaluation used these same measures from the RRS to assess the research outcomes of each of these groups.

As is shown in Table 4 below, a greater proportion of CIHR Salary Award recipients reported that their research resulted in these outcomes when compared with those who were not funded:

Table 4: Outcomes resulting from the research program of funded and non-funded salary award applicants while holding (or expected to hold) a CIHR salary award

Outcome type Funded Salary
(n=531)
Non-funded Salary
(n=292)
Open Operating Grants Program (n=596)
Research findings/ Knowledge creation* 99% 96% 94%
New research method 57% 50% 57%
New theory 44% 37% 63%
Replication of research findings* 42% 27% 49%
Adaptation of research findings* 31% 24% N/a
New Practice (Clinical, Tool instruments, Procedure/Technique) 25% 21% 22%
New vaccine/drug 2% 1% 5%
Patents/licenses 12% 13% 12%
Software/database 10% 12% 8%
New or changed policy/program* 19% 10% 5%
Spin off company 2% 3% 4%
Intellectual property claim* 9% 16% 12%
Direct cost savings ( individual, organization, system or population level) 5% 3% 6%

*Statistically significant between funded and non-funded salary support researchers at p-value =0.05; Statistical tests: test Chi-Square.

Sources: Survey of CIHR funded salary/career award researchers and non-funded applicants; CIHR Research Reporting System (RRS) data (OOGP outcomes)

CIHR's Open Operating Grants Program (OOGP) provides a useful benchmark for contextualizing how the outcomes of salary support funded and non-funded salary award applicants compare to other sources of funding. Data from the Research Reporting System for the OOGP shows that many of the research outcomes are in-line with those reported by Salary Support awardees.

There are many caveats here when comparing grants to awards, although in most cases the duration of OOGP grants is the same as that for salary awards (five years). It should also be noted that salary support holders frequently hold a range of other funding in additional to their award, including operating grants from CIHR and other funders.

The difference in the proportions of salary award holders and OOGP grant holdings claiming an outcome relating to a new or changed policy/program may be attributable to differences in the demographic profile of the health researcher in each group. A greater proportion of salary award holders are health systems/services and population health researchers when compared with those in the OOGP grant sample.

An analysis of research outcomes reported by CIHR open salary award holders compared with those who received a strategic award showed few significant differences between these groups (see table in appendices for details).

Table 5 shows the differences in reported outcomes when analyzed by CIHR's four research themes. As would be expected given their research focus, some themes are far more likely to report certain outcomes than others.

Table 5: Outcomes resulting from the research program of funded salary award holders by CIHR research theme

Type of outcome CIHR Theme %
Research findings/ Knowledge creation No statistically significant difference
New research method Biomedical* 64%
Clinical 51%
Health systems/services* 44%
Social/cultural and population health 54%
New theory Biomedical* 56%
Clinical* 33%
Health systems/services* 22%
Social/cultural and population health 43%
Replication of research findings Biomedical 40%
Clinical 54%
Health systems/services 33%
Social/cultural and population health 43%
Adaptation of research findings No statistically significant difference
New Practice (Clinical, Tool instruments, Procedure/Technique) Biomedical* 10%
Clinical* 57%
Health systems/services* 38%
Social/cultural and population health 28%
New vaccine/drug No statistically significant difference
Patents/licenses Biomedical* 17%
Clinical 8%
Health systems/services 6%
Social/cultural and population health* 3%
Software/database No statistically significant difference
New or changed policy/program Biomedical* 4%
Clinical* 23%
Health systems/services* 53%
Social/cultural and population health* 38%
Spin off company No statistically significant difference
Intellectual property claim Biomedical 12%
Clinical 7%
Health systems/services 4%
Social/cultural and population health 3%
Direct cost savings ( individual, organization, system or population level) Biomedical* 3%
Clinical* 10%
Health systems/services* 15%
Social/cultural and population health 3%

* Statistically significant when comparing one theme with the other three themes; Statistical test Chi-Square with Bonferroni correction: p-value=0.05/6=0.008

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.4. Training and supervision

It is expected that salary award holders involve other junior researchers in their research project in order to pass on their knowledge and skills; this increase in research staff is assumed to increase their productivity. During stakeholder interviews, VPs Research confirmed the expectation that the funded researchers should attract and retain trainees for their research programs.

As shown in Table 5 below, there are no statistically significant differences between the number of students trained by CIHR salary awardees and non-funded salary award applicants.

Table 6: Training of national and international students

Number of national and international students who completed their formal training under the supervision of the funded and non-funded respondents while holding (or expecting to hold) a CIHR salary award
Average by researcher
Funded
(n=531)
Non-funded
(n=292)
Undergraduate students (national) 7 5.1
Undergraduate students (international) 0.2 0.2
Master's students (national) 2.6 2.2
Master's students (international) 0.8 0.2
Doctoral students (national) 1.1 1.5
Doctoral students (international) 0.5 0.2
Fellowships (national) 0.6 1.2
Fellowships (international) 0.2 0.2
Post-Doctoral (national) 0.8 1.2
Post-Doctoral (international) 0.5 0.5
Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

When assessing the number of research assistants, trainees and postdoctoral fellows and master's students involved in the research program, CIHR salary awardees tend to have greater involvement compared with non-funded salary award applicants (see Table 6).

Table 7: Students and research staff involved in the research program

Number of students and research staff involved in the research program of CIHR funded and non- funded researchers while holding (or expected to hold) a CIHR salary awards
Average by researcher
Funded
(n=531)
Non-funded
(n=292)
Research assistant(s)* 3.2 2.5
Undergraduate students 5.4 5.1
Research technician(s) 1.3 1
Trainees-Postdoctoral fellows (post-PhD)* 1.7 1.2
Post health professional degree (e.g., MD, BScN, DDS, etc.) 0.9 0.6
Fellows not pursuing a Master's or PhD 0.7 0.6
PhD students 2.5 2.3
Master's students* 3.5 3

* Statistically significant at p-value =0.05; Statistical tests: non-parametric, non-paired groups - Mann-Whitney.

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.5. Obtaining other sources of funding

A key indicator used to assess program outcomes for new investigators in particular is their subsequent capacity for leveraging and obtaining other sources of research funding. The program theory is that holding a CIHR salary award will both allow the researcher to develop a research program and also add to their prestige, enabling them to be successful in other similar competitions.

3.5.1. Operating grants and awards from CIHR

CIHR's Open Operating Grants Program provides a significant amount of funding to Canadian health researchers for investigator-driven open research (around $500m annually). It is also competitive, attracting applicants with higher than average scientific impact scores for their previous publications and with an annual success rate of around 20% (approximately one in five applicants are funded). The agency also offers a range of other operating grants to health researchers including knowledge translation, training, team and catalyst grants.

Figure 3: Percentage of salary award funded and non-funded researchers obtaining CIHR operational funding
Bar graph showing percentage of salary award funded and non-funded researchers obtaining CIHR operational funding

Figure 3 long description

Source: CIHR Electronic Information System (EIS) database

Figure 3 shows the proportion of funded and non-funded salary award applicants who receive CIHR operating grants via the OOGP or other types of grant. The chart shows the triangulation of two quasi-experimental design methods; a pre-post award comparison and a counterfactual comparison between the funded and non-funded groups. As can be concluded from Figure 3:

Figure 3 does not tell the whole story however; while salary award funded researchers are no more likely to receive CIHR operational funds than their non-funded counterparts, they do receive a greater number of grants and a larger total amount of funding:

It cannot be determined with certainty whether the greater number of grants and higher average funding received by funded salary award holders can be attributed to the impact of the program. This could for example reflect a selection bias of the program intake; the peer review committees select the most able applicants who then go on to receive more grants and operating dollars.

3.5.2. Operating grants and awards from other organizations

Outside of CIHR, the majority of CIHR salary awardees (90%) and non-funded salary award applicants (86%) received operating grants from other organizations.

As shown in Figure 4 below, aside from CIHR itself, provincial organizations are the most frequent providers of other operating funding, representing 25% of the total number of grants received from other organizations by both funded and non-funded salary award applicants. Foundations are also a key player in this landscape; 21% of salary awardees and 19% of non-funded salary award applicants received research grants from this source.

Figure 4: Research grants received by funded and non-funded researchers from organizations other than CIHR
Bar graph showing research grants received by funded and non-funded researchers from organizations other than CIHR

Figure 4 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.5.3. Salary awards received from organizations other than CIHR

As noted earlier, CIHR provides salary/career award funding as part of a wider landscape of other organizations offering these awards. This has an important implication when considering the future of the salary award programs; if CIHR were no longer offering some salary awards, what might be the implication for Canadian health researchers?

A greater proportion of non-funded salary award applicants (55%) received salary funding from organizations other than CIHR, compared with CIHR salary awardees (40%). This is perhaps to be expected given that those not funded by CIHR awards have greater financial drivers to seek this financial support elsewhere. It should also be noted that researchers holding CIHR Salary Awards cannot apply to other sources of federal funding for salary awards.

Figure 5 shows that aside from CIHR, provincial organizations are the main source of salary award funding in Canada, representing 62% of the other awards received by the funded researchers and 68% of the other awards received by the non-funded salary award applicants (see Figure 5).

Figure 5: Salary awards received by funded and non-funded researchers from organizations other than CIHR
Bar graph showing salary awards received by funded and non-funded researchers from organizations other than CIHR

Figure 5 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

The average total salary award amount (per researcher) received from provincial organizations by non-funded researchers is higher ($320,539) than the amount received by CIHR funded researchers from provincial organizations ($226,328).

It can be concluded from this data that many non-funded CIHR salary applicants are able to find salary support funding from other organizations and that they receive a significant amount of such funding. As is later detailed in the organization scan however, several funders are no longer offering salary support awards. Retrospective data on past funding may not therefore be indicative of the salary awards that might be available to Canadian health researchers in future.

3.6. Retention and sustaining a career in health research

One of the goals of the CIHR salary career awards is to contribute to attracting and retaining researchers in the health research field and in Canada. As is noted in CIHR's Health Research Roadmap, this has particular relevance to attracting and retaining the best early career researchers, including the many new investigators funded with salary awards.

3.6.1. Retention of health researchers

Table 7 shows that a large majority of researchers remained at the same institution or university they were at when they applied for a CIHR salary award. There is no evidence to suggest that either the funded or non-funded researchers are leaving Canada in large numbers; fewer than one in twenty funded or non-funded researchers were pursuing research outside Canada.

Table 8: Current location of funded and non-funded researchers
Funded researchers (N=531) Non-funded researchers (N=292)
Current Location % %
The same institution or university as when I applied for this CIHR Salary Award 86% 83%
A Canadian research institution or university 13% 14%
A research institution or university in the United States 1% 1%
A research institution or university in another country 0% 2%
I no longer conduct research. - 1%
Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

Only 25% of CIHR salary award funded researchers declared that it is likely (to a moderate or great extent) that they would have left Canada if not for the CIHR salary awards. Thirty-five percent of funded researchers stated that having received a CIHR salary awards influenced (to a moderate or great extent) their decision to stay in Canada after their last award's term.

In terms of the perspective of University VPs and other key informants on the role of the CIHR salary awards in the retention of researchers:

Some key informants gave examples of provincial salary programs which seem to increase retention in those provinces. For example, FRSQ new investigators can benefit from an income tax exemption for five years once they get an appointment. This may allow researchers to establish a research program and they become less likely to move abroad to conduct their research in future. Also, FRSQ award recipients cannot undertake a portion of their research in another country.

3.6.2. Sustaining a career in health research

The majority of the CIHR salary awardees (70%) declared that receiving the salary awards influenced their decision to continue their career in health research (to a moderate or great extent). Key informants who were interviewed also stressed the importance of salary support in allowing researchers to sustain a career in health research and build up a research program.

However, set against these findings, it can be noted that only 1% of non-funded salary award applicants were no longer pursuing a career in health research when surveyed. The conclusion can be drawn here that while the salary awards may influence a decision, they are not an important factor in this regard. Only a tiny minority of non-funded CIHR salary award applicants abandon their health research careers.

Forty-five percent of non-funded CIHR salary award applicants did not receive any salary award during this period, either from CIHR or other funders. It is therefore likely that operational funding had an impact on sustaining the careers of this group, given that 86% of non-funded salary award applicants did receive operational grants after being unsuccessful in a salary award competition.

3.7. Career trajectory

A significant element of the program theory of salary awards for new investigators in particular is that receiving salary support at an early stage of a research career can have benefits in the longer term. Indeed, key informants articulated the view that the awards should increase the likelihood of reaching tenure at a university or sustaining employment and a research program until such a position became available.

As one key informant summarized:

"They have a greater ability to progress through the ranks faster. They get promoted faster and they get tenure faster. They get more grants and more capacity to do more research faster."

This qualitative assessment can be compared with career trajectory data on both the funded and non-funded applicant groups. Findings show a significant difference in the early stage career progression between salary awardees and non-funded salary award applicants (see Figure 6):

Only 28% of the funded salary awardees remained assistant professors, compared with 48% of the non-funded researchers. No statistically significant differences were found for the other categories. It should be noted that most of the researchers in an assistant professor position at the time they applied for funding had applied for or were holding a New Investigator award among both the funded (97%) and non-funded researchers (91%).

Figure 6: Career progression of CIHR salary awarded funded and non-funded researchers

Flow chart depicting career progression of CIHR salary awarded funded and non-funded researchers

Figure 6 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.8. Collaboration and participation in committees/networks

Holding a CIHR salary award appears to have a positive effect on the level of collaboration with other researchers for a large proportion of the salary awardees. As shown in Figure 7:

Confounding factors must be acknowledged here; it is possible that differences in career stage between applying for the award and completing the survey, or length of time in research has a significant impact on collaboration.

Figure 7: Salary awardee collaboration/participation in networks/committees

Bar graph depicting salary awardee collaboration/participation in networks/committees

Figure 7 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.9. Opportunities for improvement in program design/delivery

From the perspective of researchers and University VPs research, the value of the award (typically $60K annually) emerged as a significant issue. As is shown in the findings below, the salary support provided does not generally cover the costs of a researcher salary. However, it should be remembered that the award is intended to be a contribution to the salary cost rather than to cover the total amount. This issue is more significant for clinician scientists than for other groups given their higher salaries and the financial opportunity cost of conducting research.

Calls for an increase in the size of awards by those receiving them should also be put in context. As was highlighted in the interviews with senior management at CIHR, increasing the amount of award funding for this program would either result in a corresponding decrease in the number of awards provided or reducing the budget available for other grants and awards, given finite resources at the agency. From a management perspective, a key design question is whether delivering salary award funding, either through open or strategic competitions, is the most effective mechanism of achieving the agency's overall objectives.

3.9.1. Amount of the awards

Figure 8 shows researcher opinions on the extent to which the CIHR salary /career award covered their salaries for the period of the award:

Figure 8: Extent to which CIHR salary awards cover researchers' salaries for the period of the award
Pie charts showing extent to which CIHR salary awards cover clinician scientists' salaries for the period of the award

Figure 8 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

Many key informant stakeholders expressed concern regarding the current amount offered by the CIHR salary awards in that:

It can be noted, however, that despite these perceptions, application pressure has remained high for many of the salary awards. There is also no evidence to suggest that the awards are not attracting high calibre applicants.

In terms of the clinician scientist awards, the survey findings are supported by the perceptions of the key informants interviewed:

By contrast with the findings for other salary awards, the low application pressure and high success rate for clinician scientists compared with new investigators gives support to this conclusion.

Key informants suggested lowering the proportion of dedicated research time required by clinician scientists and increasing the amount of the awards for clinicians to increase the attractiveness of CIHR salary awards for this group of researchers.

Another significant reason given for relatively low application pressure for clinician scientist awards is the importance placed upon maintaining clinical skills. Key informants suggest that if the value of awards were increased, this would increase the incentive to apply for many clinician researchers.

Key informants were divided as to what should be done if additional funds were available for salary awards. Some key informants argued that funding should be directed at more generous awards to bring them in-line with the costs of research. Other key informants suggested that additional funds should increase the number of awards granted.

It was recognized by CIHR senior management in their interviews that CIHR salary/career award amounts currently fall below research costs, but that this is a reflection of a wider flat funding environment across the board. It was noted that this type of funding must be delivered in the context of partnerships between institutions and CIHR; institutions must also provide financial commitments to provide sustainable salary funding.

3.9.2. Open or strategic funding

Interviews with CIHR senior management highlighted a general view that strategic salary support funding would be most likely to make a significant contribution to capacity development. This was particularly the case for funding clinician scientists, where several of those interviewed felt that a shortage of these researchers made it essential to continue to fund this group.

It was also recognized that clinician scientists have specific needs based on the strong alternatives they have to conducting a research career, and therefore an imperative to provide adequate stipends to support this group. Senior managers felt that consideration should be given to expanding the type of clinicians eligible to apply for the awards.

The importance of supporting new investigators and their entry into research was mentioned by many senior managers. A range of options were discussed in this regard, including whether providing operating funding programs specifically for new investigators would achieve similar objectives to the salary support programs. Some concerns were raised that if this option were taken, it would be important to ensure protected time for research. This was raised as a particular consideration for the social sciences, where the teaching load was felt to be higher.

It can however be noted that the survey findings show salary awardees to be spending a similar amount of time of research before and during holding an award. Moving towards providing operating funds for the new investigator group may not therefore have significant implications in this regard.

3.9.3. Duration of the awards

Opinion was divided among VPs research and other key informants around the optimal duration for the awards:

3.9.4. Holding awards concurrently with operational funds

As previously stated, some key informants point out that it is ineffective to support researchers if they are not otherwise supported in their research through operating funds. Researchers require research infrastructure, equipment, and trainees, as well as salary support. CIHR salary /career awards could therefore be combined with operational funding and research infrastructure funding to ensure full coverage.

Recent CIHR salary/career award program alterations initiated at CIHR in 2011 have responded to this need; salary award holders are now required to hold concurrent operational funding in order to qualify for a salary award.

3.9.5. Quotas for awards

Some key informants suggest that CIHR could award salary support quotas to various institutions who would administer the funding. This was viewed as decreasing the administrative burden for CIHR. This is also the approach taken for the Canada Research Chairs Program (CRCP).

However, other key informants stressed that the savings reclaimed through this strategy would be minimal. Furthermore, VPs Research, partners and CIHR management generally agreed that implementing a quota system could compromise the reputation of the peer review committees and have a negative impact on the prestige associated with the awards.

3.9.6. Peer Review Committees

Some key informants suggested that the peer review process could be updated to promote greater cost-effectiveness. CIHR management agrees that the peer review process could be streamlined in a variety of ways. Several key informants suggested a range of alternatives such as: utilizing online forums instead of face-to-face meetings and the consolidation of a variety of peer review committees into one (CFI, SSHRC, NSERC, CIHR).

4. Efficiency and Economy

Evaluation Question

2. To determine the degree of efficiency and economy associated with the CIHR salary programs

As with all CIHR-administered grants or awards, a key element of efficient and economical program delivery is the management of the competition process. This includes the process for researchers to apply for an award, the work of the peer review committees in assessing the applications and making a decision, and post-award management.

4.1. Satisfaction with program delivery

Figure 9 on the following page shows high levels of satisfaction among applicants with the application and decision-making processes. These results can be benchmarked against a recent survey of all CIHR researchers conducted for the agency's 10th Year International Review.

The findings are comparable; for example, in this survey, 19% of respondents note dissatisfaction with the clarity of application instructions and 21% of researchers and applicants report dissatisfaction on the timeliness of posting results.

While satisfaction levels with program delivery are high among applicants, a range of areas were identified where improvements could be made. It can be noted that many of these relate to only a minority of the salary award applicants surveyed:

Figure 9: Satisfaction with the application and decision-making process (funded and non-funded researchers)

Bar graph showing satisfaction with the application and decision-making process (funded and non-funded researchers)

Figure 9 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

Key informants felt that CIHR's peer review process is highly respected by the research community, and, as such, selection by a peer review committee is typically a mark of prestige and quality for researchers and research institutions. CIHR's peer review process was identified throughout the interviews with university VPs research and other stakeholders as reinforcing and improving the reputation of salary awardees.

When partner funding organizations were invited to share their opinion on the application and decision process, they reported that delays in competition processes can cause significant challenges for industry to carry on with research projects that depend on timely funding.

Additionally, some university VPs Research perceive that the competitions for salary awards sometimes favour larger universities with greater research capacity and are less favourable to health research in the humanities and social sciences. Some key informants suggested a quota system to mitigate a perceived bias; however other key informants recognized that although CIHR's salary awards are very competitive, this adds to the prestige of the awards.

4.2. Consistency between program objectives and criteria used by peer reviewers

A field observation was conducted to assess the degree of consistency between the salary support program objectives and the review criteria used by peer reviewers to assess applications. A high degree of consistency signifies that the program is being delivered as intended and that the applications selected reflect the program objectives.

Each application is reviewed according to three main criteria:

There are also specific sub-criteria within each of these main criteria that are intended to guide the peer review committees' discussions. Taken together these criteria and sub-criteria reflect the objectives of the salary/career awards.

In February 2010, a sample of four peer review sessions were observed by two evaluators for the following peer review committees:

In all the four sample peer review committee sessions that were observed, all of the main criteria and at least 71% of the sub-criteria were discussed for each applicant. From this, it can be concluded that the peer review committees assessed applications in accordance with program objectives.

The observation form used by the evaluators to assess the consistency between the peer review committees' discussions and the peer review criteria can be found in the evidence binder.

4.3. Partnerships

CIHR partners with a variety of organizations to deliver salary awards; provincial and national health funding agencies, charitable organizations, industry, hospitals and universities and research institutes. Since 1999, CIHR has leveraged a total of $68,581,141 in partner financial contributions ranging in size from $7,102 to $6,645,767.

Figure 10 demonstrates a decline in partnership contributions to salary support awards from 1999 to 2010 and in particular from 2002-2003 onwards. It is noted that partnership contributions fell in accordance with the decrease in CIHR corporate funding dedicated towards Salary Awards over this time period.

Figure 10: Partner financial commitments for CIHR salary awards (open and strategic) (1999-2010)

Bar graph showing partner financial commitments for CIHR salary awards (open and strategic) (1999-2010)

Figure 10 long description

Source: Electronic Information Systems (EIS) Partnerships database

A few of the key informants interviewed suggested that CIHR could do more to develop partnerships and leverage funding in this program space. According to some partner organizations, it may become progressively difficult for CIHR to partner with industry on salary awards due to a paradigm shift in the priorities of these organizations. Some felt that industry would be increasingly focusing its financial support on strategic areas of research rather than on researchers.

Some partner organizations also expressed concern in regards to CIHR's recognition of partner contributions. According to these key informants, CIHR does not always do enough to publically recognize the contribution of their partners. This said, most partners who were interviewed reported overall satisfaction with their relationship with CIHR.

Interviews with CIHR management included suggestions that the agency should continue to examine how sustainable partnerships are facilitated, including introducing more effective communications strategies.

5. Addressing the Needs of Canadian Health Researchers

Evaluation Question

3. To determine the extent to which the CIHR Salary Programs continue to address the needs of Canadian health researchers

5.1. Health researchers' need for salary support and the extent to which CIHR meets this need

As shown in Figure 11 four key factors were identified by the stakeholders involved in this evaluation (researchers, university VPs, partner organizations and CIHR management) as being important to researchers applying for CIHR Salary Awards. Each of these elements is described in further detail below.

Figure 11: Health researcher needs for salary/career awards

Graphical depiction of the four key factors identified by the stakeholders involved in this evaluation (researchers, university VPs, partner organizations and CIHR management) as being important to researchers applying for CIHR Salary Awards

1. Ability to develop or maintain a research program

Surveyed researchers identified that the most important factor for them in applying for salary support was the ability to develop or maintain a research program. Other stakeholders also acknowledge that salary support is necessary in order to allow researchers to dedicate the majority of their time (a minimum of 75%) to their research while reducing their time spent on other activities (e.g. teaching). Several interviewees from CIHR's Senior Management stressed that this 'protected' research time is particularly important for New Investigators, who are often burdened with activities, such as teaching, that can detract from their research time.

2. Ability to obtain other research funds

Both funded and non-funded researchers perceive the ability to obtain other research funds as one of the most important factors in salary support funding. Interviews with stakeholders and CIHR Senior Management highlighted that researchers require solid research infrastructure and operational funding in order to be successful, and that salary support is less effective if researchers do not have sufficient access to operational funds. Thus, in order for salary award funding to be effective in advancing research careers, it must be coupled with operational funding.

3. Recognition - improving profile and reputation as a researcher

Researchers require 'recognition' in order to increase their research profile and obtain funding from a variety of sources. CIHR's peer review process was identified throughout the stakeholder interviews as reinforcing and improving researchers' reputations; as the peer review committees are well respected in the research community, being selected as a salary award recipient greatly enhances the perceived quality of the candidate and their research.

4. Career advancement - improving prospects for promotion

CIHR Salary Awards were described by some survey respondents and interviewees (other stakeholders and CIHR senior management) as being essential to the career progression of researchers. Many key informants pointed out that some universities that will not hire health researchers if they do not qualify for a salary support award. Other universities support researchers for some time—two to five years—on a contractual basis, to allow them to build up their curriculum, after which, if they do not have a salary award of some kind, the institution will stop supporting them.

The differing perspectives of universities and research funders should be considered here. A number of research institutions claimed that they would not have sufficient financial resources to continue supporting some of their researchers without salary awards. However, in other stakeholder interviews, the role of health research funders in providing salary support of this type was questioned. Some interviewees felt that research institutions should be in a position to support their researchers without relying on salary awards as an ongoing funding stream that is sometimes built into the budgets and financial planning of an institution.

Figure 12 presents the importance given by salary awardees to each of these factors and the extent to which CIHR salary/career awards met the needs represented by these factors.

Figure 12: Health researcher needs for salary/career awards
Bar graph showing health researcher needs for salary/career awards

Figure 12 long description

Note: The mean corresponds to the average value of the responses.
Importance: 1=Not important at all, 2=Not important, 3=Somewhat important, 4=Important, and 5=Very important
Needs met: 1=Not at all, 2=To a small extent, 3=To some extent, 4=To a moderate extent and 5=To a great extent
There is a strong correlation between the two ratings.

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

As would be expected, the specific needs of different groups of salary award holders vary somewhat according to their career stage and whether open or strategic funding is received.

5.1.1. New Investigators

Key informants stress that although helpful at all career stages, salary support is somewhat more critical at the beginning of a researcher's career. While senior researchers typically secure funding more easily than New Investigators based on their reputation and experience, New Investigators depend on funding from programs such as the CIHR Salary Awards to "break into the system."

Supporting this, surveyed New Investigators indicated that CIHR salary awards met their needs in term of their "career advancement and improvement of their prospects for promotion" to a greater extent than Mid- Career Investigators.

5.1.2. Clinician Scientists

As noted earlier in this report, the Clinician Scientist Phase II program, offered through CIHR's suite of salary awards, is of particular interest to stakeholders as it is intended to address the unique needs of clinician scientists. Throughout the key informant interviews, stakeholders described the needs of clinician scientists as differing from those of other health researchers in two predominant ways:

  1. The protected time that they receive as a result of obtaining an award, and;
  2. The amount of the salary award.

The nature of clinician scientists' work requires regular practice in order for them to maintain their clinical skills, making it especially challenging for them to devote 75% of their time to research and only 25% to clinical duties. Also, clinical environments such as hospitals can put significant pressure on these scientists to spend as many hours as possible on clinical duties - particularly those attached to teaching hospitals.

Second, the salary of clinicians is generally far higher than that of other researchers'. This diminishes the relative impact of salary support awards; the awards may cover the bulk of other researchers' salaries, but represent a small portion of the salary of clinician-researchers. It also diminishes the incentive to apply for such awards.

"You have to buy that time and the award does not cover it. You certainly don't get 75% of a clinician scientist's time [with] a CIHR new investigator award."

5.2. Universities and research centres' needs for salary support and the extent to which CIHR addressed them

Key informants in universities and research centres identified a range of needs for salary support. At an overall level, they acknowledge there is 'never enough' support for research and researchers; universities and research centres are always looking to expand their research programs and infrastructure in order to remain competitive. Nevertheless, they feel that CIHR salary awards address most of the needs of universities and research centres, albeit to a varying extent.

5.2.1. Support for operations and infrastructure

While universities and research centres are able to put a portion of their public funding toward supporting their research operations and infrastructure, they must typically solicit additional funds to put towards these ends. As a general rule, a well-supported research capacity and research infrastructure enables institutions to remain competitive, to attract prominent researchers, and to leverage additional funds, among other benefits.

5.2.2. Attract, hire, and retain researchers

Universities and research centres compete at the national and international level to attract and retain researchers. To this end, CIHR salary awards help to reduce institutions' financial burden, allowing them to reallocate or reinvest some of their resources toward improving infrastructure and increasing research budgets and capacity, which contributes to the attraction and retention of new and prominent researchers.

Key informants pointed out that some universities only hire or offer tenured positions to researchers who receive external salary support. Consequently, the more salary awards offered, the more researchers hired by universities. In other cases, universities hire researchers without external salary support on the condition that they apply for a salary award during their employment.

5.3. Availability of CIHR salary awards

An analysis of CIHR administrative data for the period 2000/2001-2008/2009 for the application process of the currently offered salary awards suggests that there is a continued demand for salary support among health researchers in Canada.

Both Open and Strategic New Investigator awards attract significant levels of applications and have a success rate of approximately 20%.5 It is important to note that these two programs also comprise the bulk of the salary support applications, representing 93% of total applications. The success rates for the other five programs range between 45% for CIHR Research Chairs and 76% for Clinical Scientists Phase II.

Figure 13: Application pressure for CIHR salary/career award programs

Stacked bar and line graph showing application pressure for CIHR salary/career award programs

Figure 13 long description

Source: CIHR Electronic Information System (EIS) database

The availability of CIHR salary awards was also discussed though interviews with key stakeholders:

Some key informants felt that although the responsibility to support social science health research had been transferred from the Social Sciences and Humanities Research Council (SSHRC) to CIHR, CIHR had not made the necessary adjustments to reflect this change in its suite of salary awards.

Some interviewees from CIHR's Senior Management responded to this perception by emphasizing the importance of ensuring that each of CIHR's themes are represented on salary award Peer Review Committees, and suggested tailoring the eligibility criteria for salary awards to be more inclusive of social science researchers.

6. Alignment with Government Priorities, Roles and Responsibilities

Evaluation Question

4. To determine the degree of alignment of the CIHR Salary Programs with Canadian government priorities

CIHR Strategic Plan – Health Research Roadmap

Key informants argue that it is the responsibility of the federal government to support research to increase Canada's competitiveness, enhance the productivity of Canadians, and ultimately increase Canadians' standard of living. The Government's Science & Technology Strategy sets out a comprehensive, multi-year science and technology agenda.

CIHR's salary/career award programs support the following sections of this strategy:

Knowledge Advantage

Canada must build upon our research and engineering strengths, generate new ideas and innovations, and achieve excellence by global standards.

People Advantage

Canada must grow its base of knowledge workers by developing, attracting, and retaining the highly skilled people we need to thrive in the modern global economy.

CIHR's strategic plan, the Health Research Roadmap, sets out four strategic directions to achieve CIHR's long-term goals. These strategic directions enable CIHR to carry out its full mandate in all its complexity, show leadership within the wider health research community, and demonstrate accountability and results to Canadians.

By attracting and retaining the best early career researchers through the New Investigator program, salary awards align directly with the first strategic direction of the Roadmap, Invest in World-class Research.

7. Roles and Responsibilities in Delivering Salary Support

Evaluation Question

5. To determine the role and responsibilities of CIHR in delivering salary support to health researchers in Canada.

CIHR is widely considered by stakeholders to be the main source of salary support for health researchers in Canada. However, key informants identified a range of other sources of salary support which are available to health researchers in Canada. This includes delivery of salary support programs to support health research by provincial and not-for-profit agencies.

7.1. Coordination role for CIHR

Key informants state that there is an opportunity for CIHR to do more in terms of coordinating, positioning, and aligning its activities with other organizations that support health research in Canada. Interviews with CIHR management supported conducting an environmental scan to ensure that the salary support landscape is fully understood. There may also be subsequent opportunities to discuss complementarity between programs at suitable forums attended by agencies providing salary support programs.

Most key informants believe that the existence of other programs is not a sign of duplication or overlap with CIHR salary programs but is often complementary:

7.2. Environmental scan of salary support programs

Table 8 on the next page gives details of 14 provincial and not-for-profit organizations offering salary support programs in Canada.

To summarize the overall characteristics of these organizations:

It should be noted that three of these fourteen organizations do not have active competitions planned for 2011-12; two of these organizations have cancelled or suspended salary award programming. Interviews with key informants support this data, in that these suggested a trend away from salary awards and towards other approaches to funding.

Table 9: Environmental scan of salary/career support awards provided by Canadian funders

Organization Funding Source Career Stage Annual stipend or other funding Award Term Number of Awards per Year Current status
Alberta Innovates Provincial Junior, Mid-career & Senior $100, 000- $160,000 7 years 2006: 45 awards Re-organization underway, no current programs
Fonds de recherche en santé du Québec Provincial Junior & Senior (0 years and 4 years) $18,000 - $100,000 plus benefits 4 - 7 years 2010: 410 awards Yes
Ontario Ministry of Health and Long-Term Care (OMHLTC) Provincial Junior (0-3 years) $70,000 per annum plus benefits 5 years 2005: 10 awardees, 2009: 6 awards Discontinued in 2011
Michael Smith Foundation for Health Research (MSFHR) Provincial Junior & Mid-career $80,000 - $100,000 5-6 years 2006: 44 awards Not accepting further applications
Arthritis Society National not for profit Mid-career $70,000 plus benefits Up to 4 years A limited number of Senior Distinguished Investigators According to the Arthritis Society's new Strategic Plan for 2011-2016
Canadian Cystic Fibrosis Foundation National not for profit Junior $60,000 1-3 years 2010-11: none awarded Only in odd-numbered years
Canadian Diabetes Foundation National not for profit Junior $50,000 - $55,000 + start up funding 5 years 2010: 16 awards Yes
Heart and Stroke Foundation National not for profit Junior & Mid-career $60,000-$81,000 plus benefits 5 years 2009: 11 awards A limited number of New Investigators will be supported by the Foundation at any given time and new awards will be made on the basis of vacancies occurring each year.
Kidney Foundation of Canada National not for profit Junior $60,000 -70,000 3 years 2010: 4 awards Yes
Multiple Sclerosis Society of Canada National not for profit Junior $50,000 3 years Information unavailable Yes
National Cancer Institute of Canada National not for profit Junior & Senior $50,000-$100,000 3-5 years Information unavailable Suspended in 2011
National Neurofibromatosis Foundation National not for profit Junior $35,000 - $45,000 plus benefits 2 years 2010:6 awards Yes
The Alberta Prion Research Institute Provincial not for profit All $50,000/year 3 years 2009: 5 awards Yes
The Ontario HIV Treatment Network Provincial not for profit Junior & Mid-career $21,000-$75,000 1-5 years 2009: 3 awards Yes
CIHR Federal New Investigator, Mid-career $60,000 3-5 years 2009-10: 82 Yes

Bibliography

Bates, I., Akoto, A.Y., Ansong, D., Karikari, P., Bedu-Addo, G., Critchley, J., Agbenyega, T. and A. Nsiah-Asare (2006).Evaluating Health Research Capacity Building: An Evidence-Based Tool. PLoS Med 3 (8), e299. DOI: 10.1371/journal.pmed.0030299

Bryar R.M. (2003). Focus. Practitioner research: an approach to developing research capacity in primary care. Nursing Times Research 8 (2): 101–114.

CIHR Mission – Retrieved on October 2009 from:
http://www.cihr-irsc.gc.ca/e/7263.html

CIHR Career Awards: Consultation Report – April 2006 - Retrieved October 2009 from:
http://www.cihr-irsc.gc.ca/e/31458.html

Research Net – New Investigator Salary Award Request for Applications (RFA): 2008-2009 - Retrieved October 2009 from:
http://www.researchnetrecherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=519&view=seedlist&org=CIHR
&type=AND&resultCount=25&sort=program&all=1&masterList=true&language=E

CIHR - Understand Peer Review – Retrieved October 2009 from:
http://www.cihr-irsc.gc.ca/e/37790.html.

2008-2009 CIHR Grants and Awards Guide – Retrieved October 2009

Charles F. Reynolds III, M.D., Paul A. Pilkonis, Ph.D. , David J. Kupfer, M.D., Leslie Dunn, M.S., Harold A. Pincus, M.D., (2007). Training Future Generations of Mental Health Researchers: Devising Strategies for Tough Times. Academic Psychiatry. 31(2): 152-159.

Funnell, S. C. & Rogers, P. J. (2011). Purposeful Program Theory. San Francisco, CA: John Wiley and Sons.

Moed, H. F. (2005). Citation Analysis in Research Evaluation. Dordrecht, Netherlands: Springer.

Segrott, J., M. McIvor and B. Green (2006). Challenges and strategies in developing nursing research capacity: A review of the literature. International Journal of Nursing Studies, 43(5): 637-651.

Task Force on Career Support: Final Report – Retrieved October 2009 from:
http://www.cihr-irsc.gc.ca/e/25943.html

Treasury Board of Canada – Policy on Evaluation - Retrieved October 2009 from:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024&section=text#cha1

Treasury Board of Canada –Directive on Evaluation Function – Retrieved October 2009 from:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681

Appendix 1: Health researchers needs for salary/career awards by CIHR theme and career stage

Importance given to each of the factors by funded researchers and the extent to which the CIHR salary/career awards met the needs represented by these factors:

Biomedical theme (N=264)

Line graph showing biomedical pillar health researchers' needs for salary/career awards

Biomedical theme (N=264) long description

Clinical theme (N=98)

Line graph showing clinical pillar health researchers' needs for salary/career awards

Clinical theme (N=98) long description

Health system and services theme (N= 55)

Line graph showing health systems and services pillar health researchers' needs for salary/career awards

Health system and services theme (N= 55) long description

Social, cultural and population health theme (N= 77)

Line graph showing social, cultural and population health pillar researchers' needs for salary/career awards

Social, cultural and population health theme (N= 77) long description

New Investigators (N=426)

Line graph showing new investigator health researchers needs for salary/career awards

New Investigators (N=426) long description

Investigators (N=84)

Line graph showing investigator health researchers needs for salary/career awards

Investigators (N=84) long description

Clinician Scientists (N=20)

Line graph showing clinician scientist health researchers needs for salary/career awards

Clinician Scientists (N=20) long description

Appendix 2: Protected time to conduct research by CIHR theme and career stage

Time spent by CIHR funded researchers on different tasks:

Average percentage of time spent by CIHR funded researchers on different tasks

Task/Activity Biomedical (N= 264) Clinical
N= 98)
Health system services
(N= 55)
Social/cultural and population health
(N= 77)
Research activities associated with your research 45% 49% 53%** 49%
Administrative tasks associated with your research program 15% 13% 13% 15%
Student and research staff training and supervision 20%* 11% 11% 12%
Teaching duties 10% 7% 9% 13%*
Institutional administration 7% 5% 7% 8%
Clinical work 3% 15%* 7% 3%

* Statistically significant when comparing one theme with the other three themes;

** Had a p-value= 0.062; Statistical tests: non-parametric, non-paired groups Kruskal Wallis test with Bonferroni correction: 0.05/6=0.008

Average percentage of time spent by CIHR funded researchers on different tasks

Task/Activity New Investigators
(N= 426)
Investigators
(N= 84)
Clinician Scientists (N=20)
Research activities associated with your research 46%* 53%* 47%
Administrative tasks associated with your research program 15% 13% 14%
Student and research staff training and supervision 16% 13% 11%
Teaching duties 10% 8% 6%
Institutional administration 7% 6% 4%
Clinical work 6% 6% 18%

* Statistically significant at p-value =0.05; Statistical tests: non-parametric, non-paired groups - Mann-Whitney for New Investigators and Investigators only.

Appendix 3: CIHR salary/career awards holders research productivity outputs by CIHR theme and career stage

Average research outputs for the entire duration of the salary/career award

Type of output Biomedical Clinical Health systems/
services
Social/cultural and population health
Refereed journal articles published 15.21* 28.94 31.98 23.81
Conferences, symposium presentations and posters produced 23.71* 41.03 36.16 39.05
Books/Book chapters published 1.52* 3.18 1.93* 3.26
Reports/Technical reports published 0.35* 1.71* 3.84 2.55
Master's thesis/Doctoral dissertation published 2.64* 1.92 1.76 2.12
Multi-media content (including non-refereed articles) produced 1.32* 2.28 1.93 2.58
Prizes/Professional awards received 1.37* 2.25 1.65 1.40

* Statistically significant when comparing one theme with the other three themes; Statistical tests: non-parametric, non-paired groups Kruskal Wallis test with Bonferroni correction: p-value= 0.05/6=0.008

Average research outputs for the entire duration of the salary/career award

Type of output New Investigators Investigators
Refereed journal articles published* 19.8 26.1
Conferences, symposium presentations and posters produced* 30.5 37.6
Books/Book chapters published* 1.8 3.3
Reports/Technical reports published* 1.1 3.1
Master's thesis/Doctoral dissertation published 2.1 3.0
Multi-media content (including non-refereed articles) produced 1.6 2.4
Prizes/Professional awards received 1.6 1.8

* Statistically significant at p-value= 0.05; Statistical tests: non-parametric, non-paired groups - Mann-Whitney

Appendix 4: CIHR salary/career awards holders research productivity for open and strategic funding

Research outputs of the CIHR salary/career awards holders

Average per researcher for funded researchers Open competitions (N=312) Strategic competitions
(N=219)
Refereed journal articles published 21.6 20.9
Conferences, symposium presentations and posters produced 30.8 32.9
Books/Book chapters published 2.1 2.2
Reports/Technical reports published* 1.0 2.0
Master's thesis/Doctoral dissertation published* 2.6 1.9
Multi-media content (including non-refereed articles) produced 1.4 2.1
Prizes/Professional awards received 1.7 1.5

* Statistically significant at p-value= 0.05; Statistical tests: non-parametric, non-paired groups - Mann-Whitney

Research outcomes of the CIHR salary/career awards holders

Outcome type Open competitions
(N=312)
Strategic competitions
(N=219)
Research findings/ Knowledge creation 99% 99%
New research method* 61% 51%
New theory 45% 43%
Replication of research findings 44% 39%
Adaptation of research findings 32% 31%
New Practice (Clinical, Tool instruments, Procedure/Technique)* 22% 30%
New vaccine/drug* 1% 4%
Patents/licenses 13% 10%
Software/database 11% 10%
New or changed policy/program 17% 21%
Spin off company 3% 2%
Intellectual property claim 9% 8%
Direct cost savings ( individual, organization, system or population level) 6% 5%

* Statistically significant at p-value= 0.05; Statistical tests: non-parametric, non-paired groups - Mann-Whitney

Appendix 5: Students and research staff involved in the research program by CIHR theme and career stage

Number of students and research staff involved in the research program of CIHR funded researchers while holding a CIHR salary awards

Average by researcher for the entire award period
Biomedical Clinical Health systems/
services
Social/cultural and population health
Research assistant(s) 1.57* 5.02 5.7 6.16
Undergraduate students 6.8* 6.86* 4.23 5
Research technician(s) 1.57* 2.14* 1.31 1.26
Trainees-Postdoctoral fellows
(post-PhD)
2.45* 1.24 2.08 1.41
Post health professional degree
(e. g., MD, BScN, DDS, etc.)
0.64* 2.59 1.59 1.1
Fellows not pursuing a Master's or PhD 0.89* 2.37 1 2.84
PhD students 2.96 2.21* 2.82 2.9
Master's students 3.34* 3.71 4.5 4.41

* Statistically significant when comparing one theme with the other three themes; Statistical tests: non-parametric, non-paired groups Kruskal Wallis test with Bonferroni correction: p-value= 0.05/6=0.008

Number of students and research staff involved in the research program of CIHR funded researchers while holding a CIHR salary awards

Average by researcher for the entire award period
New Investigators Investigators
Research assistant(s)* 3.4 4.7
Undergraduate students 6.5 4.8
Research technician(s) 1.6 2.1
Trainees-Postdoctoral fellows (post-PhD)* 1.9 2.4
Post health professional degree (e.g., MD, BScN, DDS, etc.) 1.3 1.3
Fellows not pursuing a Master's or PhD 1.47 0.6
PhD students* 2.63 3.3
Master's students 3.7 3.6

* Statistically significant at p-value =0.05; Statistical tests: non-parametric, non-paired groups - Mann-Whitney


Appendix 6: Extent to which the CIHR salary/career award covers researchers' salaries for the period of the award

No differences were found across CIHR themes.

New Investigators (N=426)

Pie chart depicting extent to which the CIHR salary/career award covers researchers' salaries for the period of the award for new investigators

New Investigators (N=426) long description

Investigators (N=84)

Pie chart depicting extent to which the CIHR salary/career award covers researchers' salaries for the period of the award for investigators

Investigators (N=84) long description

Footnotes

  1. Operated by CIHR in partnership with the Natural Sciences and Engineering Research Council (NSERC) and the Social Science and Humanities Research Council (SSHRC).
  2. These figures do not include CIHR expenditures for Canada Research Chairs (CRC) program.
  3. Treasury Board – Directive on Evaluation Function 2009: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681
  4. http://www.innovation.ca/docs/accountability/2011/ten_year_evaluation_e.pdf
  5. Success in this context was defined as being funded by CIHR. The success rate was calculated by dividing the total number of applications funded in a program over the period examined in this evaluation, by the total sum of applications received in program competitions and multiplying the result by 100 in order to obtain a percentage.
Date modified: