Strategic Training Initiative In Health Research (STIHR) 2001-2006: Final Evaluation Report

Table of Contents


Abstract

Background

The Strategic Training Initiative in Health Research (STIHR) was launched by CIHR in the spring of 2001 to build capacity within Canada's health research community through the training and development of researchers, and through fostering the development and ongoing support of their careers.

The Initiative was evaluated in 2007 to provide evidence on program performance, to facilitate corporate decision-making around program renewal and funding decisions, and to fulfill reporting requirements to central agencies.

In addition to document analysis and administrative file reviews, the evaluation employed a web survey with STIHR principal investigators; telephone surveys with STIHR trainees and for comparison, recipients of CIHR direct individual awards1 and interviews with CIHR management, scientific directors of CIHR Institutes, STIHR partners and university staff.

Key Findings

An initial evaluability assessment showed that evaluation would be difficult due to a lack of clarity in the STIHR's objectives. That aside, overall, respondents from all stakeholder groups believed that the STIHR was important and relevant to the CIHR mandate, the health research community, the federal government and other external stakeholders noting that it provided good value added.

The stakeholders considered the STIHR to be comparable with programs like the Canada Graduate Scholarships, the Doctoral Research Awards and CIHR fellowship awards in terms of trainees' disciplinary background; early scholarly productivity; and perceptions of the training environment and trainee experience. Additionally, the STIHR appeared to be increasing the number of research training opportunities for health systems and services and clinical trainees.

However, the implementation of the partnerships aspect of the STIHR was perceived to be only partially successful because beyond the funding contributions, there was no further collaboration or interaction with individual STIHR Programs for most of the partners despite a desire for such ongoing collaboration.

Recommendations

Based on the key findings, it was recommended to the CIHR to review and revise the level of funding for the STIHR. The need to clarify the STIHR's objectives and to revise and strengthen performance measurement and data collection tools at both the CIHR and individual STIHR program levels was also highlighted along with improving communication among all key stakeholders.

Executive Summary

Background

The Strategic Training Initiative in Health Research (STIHR) was launched by CIHR in the spring of 2001 and was designed to:

  1. Increase the capacity of the Canadian health research community, including areas where it can be demonstrated that there is a need to develop capacity.
  2. Enable recruitment and retention of highly qualified individuals from Canada and abroad to undertake health research training in Canada.
  3. Support the development of innovative, effective, interdisciplinary and internationally competitive training programs.
  4. Engage new mentors and educators in the development and evolution of training strategies.
  5. Encourage programs that:
    1. embrace diverse research disciplines and methodological approaches to resolve major health issues and scientific challenges;
    2. integrate training and discussion on the ethical conduct of research and related ethical issues;
    3. develop and measure the individual's communication, teamwork, and leadership skills, i.e. grant writing, peer review; and
    4. incorporate effective research strategies that translate knowledge into practice.2

Important note to reader:

For ease of reading, this document will employ the following terms related to the STIHR throughout.

While "transdisciplinary"3 training is an overarching goal of the STIHR Initiative, most respondents in the evaluation appeared to use the term interchangeably with "interdisciplinary". Therefore, we use the term "interdisciplinary" throughout this report to maintain consistency including when it is used in CIHR documents relating to the STIHR Initiative.

Objectives of the Evaluation

The objectives of this evaluation are:

The evaluation focuses on the following issues:

Issue 1: Relevance of the STIHR Initiative to CIHR, Federal government and the health research community

Issue 2: Success of the STIHR Initiative in producing its desired outcomes

Issue 3: Success of partnerships

Issue 4: Success of the STIHR Initiative in providing integrated training environments to groups of trainees

Issue 5: Implications of the STIHR Initiative's selection processes for the potential success of funded STIHR programs

Evaluation Methodology

The following methods were employed to obtain the data:

  1. Web-survey with principal investigators (n = 56).
  2. Telephone surveys with trainees from individual STIHR programs (n = 396), and as comparison groups, recipients of: CIHR Canada Graduate Scholarships (CGS) awards (n = 62), Doctoral Research Awards (DRA) (n = 83) and CIHR Fellowships (FE) (n = 132).
  3. In-depth interviews with CIHR management (n = 4), scientific directors (n = 13), STIHR Initiative partners (n = 13) and university staff (n = 3).4
  4. Document analyses and administrative file reviews of: annual Progress Reports, Peer Reviews of Midterm Reports, Program documentation including archived Requests For Applications (RFA), the STIHR Results-based Management and Accountability Framework (RMAF) and various program files; and data from the CIHR's Electronic Information System (EIS).

Comparability of STIHR and CGS/DRA/FE Groups

The data analysis included a comparison of STIHR trainees' responses to the telephone survey with those of recipients of Canada Graduate Scholarship awards, Doctoral Research awards and CIHR Fellowships (CGS/DRA/FE group) however, the two groups were not entirely comparable or mutually exclusive and this constrained our ability to draw clear distinctions between them.5 Preliminary analysis indicated that STIHR trainees were less likely to be training at the post-doctoral level and more likely to be training at the master's level compared to the CGS/DRA/FE trainees. Anecdotally, it is known that many beginning trainees start out with STIHR program funding, but then "cross-over" to personal/direct funding awards from CIHR and other funding agencies once their thesis/research program is better developed.

The problems of lack of mutual exclusivity and differences in training levels were further confounded by the occurrence during the STIHR Initiative's initial funding period, of the launch of the CGS Program in 2003, beyond CIHR's power to control or predict and the subsequent "compensatory" reductions in CIHR doctoral research awards and Fellowships in that period which made it difficult to use a simple before-after design to retrospectively assess the performance of the STIHR Initiative. It is against this background that the findings of this evaluation should be viewed.


Findings and Conclusions

An evaluation assessment of the STIHR Initiative's objectives was conducted as a preliminary step in the evaluation process and the objectives were found to be unclear thus making evaluation difficult. For instance, the first objective referred to increasing capacity in areas of demonstrated need but this was more related to strategic capacity building and was difficult to operationalize. Also, whereas Objective 3 called for "support" for interdisciplinary training programs and seemed to couch interdisciplinary as a mandatory requirement, Objective 5 called for interdisciplinarity to be "encouraged." As used in this context, "encourage" is rather vague and since it is not possible to measure performance without targets or measurable indicators of success, it created problems with evaluability.

Recommendation 1: The original objectives and rationale of the STIHR Initiative were unclear and should therefore be revised in consultation with key STIHR stakeholder groups.

Recommendation 2: Strengthen STIHR performance measurement and data collection that focuses on outcomes (as outlined in the STIHR RMAF) both at the STIHR Initiative and Program levels.

Recommendation 3: The regular reporting of performance data (e.g., annual progress reports) should be revised so that the process of data collection is more streamlined, accurate and less burdensome. This should include a revision of the data collection tool(s) and the process that assesses performance of the individual STIHR Programs.

The findings, conclusions and recommendations on evaluation issues are presented below:

Evaluation Issue 1: Relevance of STIHR Initiative to CIHR, Federal government and the health research community

Key Findings:

Conclusion: The STIHR Initiative's interdisciplinary focus is appropriate and relevant to the CIHR mandate and stakeholders consider the STIHR Initiative to be important and provides good value added.

Recommendation 4: Review and revise the level of funding for the STIHR Initiative.

Evaluation Issue 2: Success of the STIHR Initiative in producing its desired outcomes

Key Findings:

Conclusion: In the past five years, there are some indications that the individual STIHR Programs are reaching towards attaining some short-term and some intermediate-term outcomes. In particular, the STIHR Initiative appears to be increasing the number of research training opportunities for health systems and services and clinical trainees.

However, there is little evidence at this point to support the claim that these early outcomes are entirely attributable to the influence of the STIHR Initiative.

Evaluation Issue 3: Success of Partnerships

Key Findings:

Conclusion: The implementation of STIHR Initiative Partnerships has only been partially successful because beyond the funding contributions, there was no further collaboration or interaction with individual STIHR Programs for most of the Partners despite a desire for such ongoing collaboration.

Most of the individual STIHR Programs were successful in involving other partners in their programs beside the STIHR Initiative Partners. The most common benefits of including additional partners were an increase in funds, improvement of training opportunities for students, and an increase in the opportunities for Knowledge Translation.

Recommendation 5: Improve communication between STIHR Initiative partners and other program stakeholders including CIHR Research Capacity Development, CIHR Institutes, and STIHR Principal Investigators. This should include more pertinent and regular reporting to partners, a clarification of the role of partners and other STIHR stakeholders, inclusion of partners in key program decision-making bodies, and formal mechanisms to facilitate partner networking with STIHR stakeholders.

Evaluation Issue 4: Success of the STIHR Initiative in providing integrated training environments to groups of trainees

Key Findings:

Conclusion:
Individual STIHR programs are considered by trainees to be as good training environments as any other training environments experienced by CGS/DRA/FE trainees.

STIHR trainees are more likely to be interacting with a larger pool of experienced scientists and appear more satisfied with their supervision experience.

Evaluation Issue 5: Implications of STIHR Initiative's selection processes for the potential success of funded STIHR Programs

Key Findings:

Conclusion: The STIHR Initiative appears to be attracting the same caliber of trainees as comparable programs and the trainee experience in STIHR Programs does not vary in a measurable way from the trainee experience when funded through CGS, DRA and Fellowship programs.

After the first two years of implementation, individual STIHR Programs were relatively stable.

Management Response

Recommendation 1: The original objectives and rationale of the STIHR Initiative were unclear and should therefore be revised in consultation with key STIHR stakeholder groups.
Response (Agree or Disagree): Agree.
Action Plan: Working group to work on objectives for next STIHR launch. Included Evaluation team on working group.
Responsibility: RCD
Timeline: September 2007 - December 2007
Actions Taken: STIHR objectives were modified in time for the re-launch of the program in January 2008 - completed.

Recommendation 2: Strengthen STIHR performance measurement and data collection that focuses on outcomes (as outlined in the STIHR RMAF) both at the Initiative and Program levels.
Response (Agree or Disagree): Agree.
Action Plan: Modify performance measurement procedures and templates (progress and mid-term reports). Link performance measurements directly to the modified objectives.
Responsibility: RCD and Evaluation
Timeline: April 2009
Actions Taken: Discussions are being initiated with the ResearchNet team and the Evaluation team to ensure that the focus on outcomes is emphasized in the ongoing plan to integrate STIHR reporting with the CIHR Research Reporting System that is currently being developed.

Recommendation 3: The regular reporting of performance data (e.g. annual progress reports) should be revised so that the process of data collection is more streamlined, accurate and less burdensome. This should include a revision of the data collection tool(s) and the process that assesses performance of the individual STIHR Programs.
Response (Agree or Disagree): Agree.
Action Plan: Modify Progress Report procedures. Modify Progress Report templates. Link performance measurements directly to modified objectives.
Responsibility: RCD and Evaluation
Timeline: November 2008 - November 2009
Other Comments: We will pilot any changes with two new STIHR programs funded in an ad-hoc competition. Funding began in 2007, so the first progress report will be due November 2008. Annual progress reports should no longer be peer reviewed by Institutes, they should simply be a mechanism for data collection and information sharing.
Actions Taken: Discussions are being initiated with the ResearchNet team and web services, as well as evaluation team to ensure that the questions being asked in the progress reports are relevant and align with objectives. The goal is to integrate STIHR reporting with the CIHR Research Reporting System (currently under development) as much as possible.

Recommendation 4: Review and revise the level of funding for the STIHR Initiative.
Response (Agree or Disagree): Agree.
Action Plan: Prepare documentation for senior management to make decisions on level of funding that should be committed in re-launch of program.
Responsibility: RCD
Timeline: December 2007
Actions Taken: RKTC approved re-launch for January 2008 with significant reductions in funding.

Recommendation 5: Improve communication between STIHR Initiative partners and other program stakeholders including CIHR Research Capacity Development (RCD), CIHR Institutes, and STIHR Principal Investigators. This should include more pertinent and regular reporting to STIHR partners, a clarification of the role of partners and other STIHR stakeholders, inclusion of partners in key program decision-making bodies, and formal mechanisms to facilitate partner networking with STIHR stakeholders.
Response (Agree or Disagree): Agree.
Action Plan: Involve partners in the creation of Funding Opportunities. The role of partners in the STIHR Initiative will be made clearer.
Responsibility: RCD and Partnerships and Citizen Engagement
Other Comments: Partners wish to better understand the program before they decide on whether to participate. It should be noted that of the partners that are not participating in the re-launch of the program, the issue is not necessarily a lack of CIHR communication. Some have created a similar program on their own and others do not have the funds to commit at this time, particularly since the STIHR grants are such a long commitment (six years).
Actions Taken: We have included partners, institutes, etc. in most of the decision making of the STIHR program to date, including all changes to the relaunch of the program. Partners were contacted and briefed on the relaunch as part of the process of asking them to participate.


  1. CIHR direct awards include Canada Graduate Scholarships, Doctoral Research Awards and CIHR Fellowships.
  2. CIHR Strategic Training Program Grants 2002 Competition, Instructions for the Full Application, p. 1.
  3. Transdisciplinarity is defined here as "the integration and transformation of fields of knowledge from multiple perspectives in order to define, address and resolve complex real-world problems." See STIHR Performance Measurement and Evaluation Framework.
  4. A purposive sampling strategy was used and looked to interview a maximum of five university staff and six from CIHR top management.
  5. Preliminary data analysis indicated considerable overlap with other CIHR awards among STIHR trainees: 31% of all trainee respondents have received Fellowship awards, 25% DRAs and 17% CGS. Overall, 66% have received funding through another CIHR program. Thus when the overlap is excluded from the analysis, the remaining n is too low to allow for any meaningful discussion of differences in statistically significant terms. This impacted our ability to do statistical tests on differences since doing so would attribute more importance to any statistical or factual difference than the data could support.
  6. This could be due to the previously discussed problems of lack of mutual exclusivity and comparability of the two groups which impacted our ability to make any clear distinctions between them.
  7. This could be due to the previously discussed problems of lack of mutual exclusivity and comparability of the two groups which impacted our ability to make any clear distinctions between them.
Date modified: