Table of Contents

• Abstract
  • Background
  • Key Findings
  • Recommendations
• Executive Summary
  • Background
• Objectives of the Evaluation
• Evaluation Methodology
  • Comparability of STIHR and CGS/DRA/FE Groups
• Findings and Conclusions
  • Issue 1: Relevance of the STIHR Initiative to CIHR, Federal government and the health research community
  • Issue 2: Success of the STIHR Initiative in producing its desired outcomes
  • Issue 3: Success of partnerships
  • Issue 4: Success of the STIHR Initiative in providing integrated training environments to groups of trainees
  • Issue 5: Implications of the STIHR Initiative's selection processes for the potential success of funded STIHR programs
• Management Response
  • Recommendation 1
  • Recommendation 2
  • Recommendation 3
  • Recommendation 4
  • Recommendation 5

Abstract

Background

The Strategic Training Initiative in Health Research (STIHR) was launched by CIHR in the spring of 2001 to build capacity within Canada's health research community through the training and development of researchers, and through fostering the development and ongoing support of their careers.

The Initiative was evaluated in 2007 to provide evidence on program performance, to facilitate corporate decision-making around program renewal and funding decisions, and to fulfill reporting requirements to central agencies.

In addition to document analysis and administrative file reviews, the evaluation employed a web survey with STIHR principal investigators; telephone surveys with STIHR trainees and for comparison, recipients of CIHR direct individual awards¹ and interviews with CIHR management, scientific directors of CIHR Institutes, STIHR partners and university staff.

Key Findings

An initial evaluability assessment showed that evaluation would be difficult due to a lack of clarity in the STIHR's objectives. That aside, overall, respondents from all stakeholder groups believed that the STIHR was important and relevant to the CIHR mandate, the health research community, the federal government and other external stakeholders noting that it provided good value added.

The stakeholders considered the STIHR to be comparable with programs like the Canada Graduate Scholarships, the Doctoral Research Awards and CIHR fellowship awards in terms of trainees' disciplinary background; early scholarly productivity; and perceptions of the training environment and trainee experience. Additionally, the STIHR appeared to be increasing the number of research training opportunities for health systems and services and clinical trainees.

However, the implementation of the partnerships aspect of the STIHR was perceived to be only partially successful because beyond the funding contributions, there was no further collaboration or interaction with individual STIHR Programs for most of the partners despite a desire for such ongoing collaboration.

Recommendations

Based on the key findings, it was recommended to the CIHR to review and revise the level of funding for the STIHR. The need to clarify the STIHR's objectives and to revise and strengthen performance measurement and data collection tools at both the CIHR and individual STIHR program levels was also highlighted along with improving communication among all key stakeholders.

Executive Summary

Background

The Strategic Training Initiative in Health Research (STIHR) was launched by CIHR in the spring of 2001 and was designed to:

1. Increase the capacity of the Canadian health research community, including areas where it can be demonstrated that there is a need to develop capacity.
2. Enable recruitment and retention of highly qualified individuals from Canada and abroad to undertake health research training in Canada.
3. Support the development of innovative, effective, interdisciplinary and internationally competitive training programs.
4. Engage new mentors and educators in the development and evolution of training strategies.
5. Encourage programs that:
   1. embrace diverse research disciplines and methodological approaches to resolve major health issues and scientific challenges;
   2. integrate training and discussion on the ethical conduct of research and related ethical issues;
   3. develop and measure the individual's communication, teamwork, and leadership skills, i.e. grant writing, peer review; and
   4. incorporate effective research strategies that translate knowledge into practice.²

Important note to reader:

For ease of reading, this document will employ the following terms related to the STIHR throughout.

• Strategic Training Initiative in Health Research: "STIHR Initiative"
• STIHR-funded programs: "STIHR Program(s)" or "individual STIHR Programs"
• Other, non-STIHR awards programs may be referred to by name or as "program(s)"

While "transdisciplinary"³ training is an overarching goal of the STIHR Initiative, most respondents in the evaluation appeared to use the term interchangeably with "interdisciplinary". Therefore, we use the term "interdisciplinary" throughout this report to maintain consistency including when it is used in CIHR documents relating to the STIHR Initiative.

Objectives of the Evaluation

The objectives of this evaluation are:

• to provide CIHR with evidence on program performance facilitating: corporate decision-making around program renewal and funding decisions and to fulfill reporting requirements to central agencies; and
• to provide all the STIHR Initiative's stakeholders with information or feedback that could be used to improve the program upon renewal.

The evaluation focuses on the following issues:

Issue 1: Relevance of the STIHR Initiative to CIHR, Federal government and the health research community

Issue 2: Success of the STIHR Initiative in producing its desired outcomes

Issue 3: Success of partnerships

Issue 4: Success of the STIHR Initiative in providing integrated training environments to groups of trainees

Issue 5: Implications of the STIHR Initiative's selection processes for the potential success of funded STIHR programs

Evaluation Methodology

The following methods were employed to obtain the data:

1. Web-survey with principal investigators (n = 56).
2. Telephone surveys with trainees from individual STIHR programs (n = 396), and as comparison groups, recipients of: CIHR Canada Graduate Scholarships (CGS) awards (n = 62), Doctoral Research Awards (DRA) (n = 83) and CIHR Fellowships (FE) (n = 132).
3. In-depth interviews with CIHR management (n = 4), scientific directors (n = 13), STIHR Initiative partners (n = 13) and university staff (n = 3).⁴
4. Document analyses and administrative file reviews of: annual Progress Reports, Peer Reviews of Midterm Reports, Program documentation including archived Requests For Applications (RFA), the STIHR Results-based Management and Accountability Framework (RMAF) and various program files; and data from the CIHR's Electronic Information System (EIS).

Comparability of STIHR and CGS/DRA/FE Groups

The data analysis included a comparison of STIHR trainees' responses to the telephone survey with those of recipients of Canada Graduate Scholarship awards, Doctoral Research awards and CIHR Fellowships (CGS/DRA/FE group) however, the two groups were not entirely comparable or mutually exclusive and this constrained our ability to draw clear distinctions between them.⁵ Preliminary analysis indicated that STIHR trainees were less likely to be training at the post-doctoral level and more likely to be training at the master's level compared to the CGS/DRA/FE trainees. Anecdotally, it is known that many beginning trainees start out with STIHR program funding, but then "cross-over" to personal/direct funding awards from CIHR and other funding agencies once their thesis/research program is better developed.

The problems of lack of mutual exclusivity and differences in training levels were further confounded by the occurrence during the STIHR Initiative's initial funding period, of the launch of the CGS Program in 2003, beyond CIHR's power to control or predict and the subsequent "compensatory" reductions in CIHR doctoral research awards and Fellowships in that period which made it difficult to use a simple before-after design to retrospectively assess the performance of the STIHR Initiative. It is against this background that the findings of this evaluation should be viewed.

Findings and Conclusions

An evaluation assessment of the STIHR Initiative's objectives was conducted as a preliminary step in the evaluation process and the objectives were found to be unclear thus making evaluation difficult. For instance, the first objective referred to increasing capacity in areas of demonstrated need but this was more related to strategic capacity building and was difficult to operationalize. Also, whereas Objective 3 called for "support" for interdisciplinary training programs and seemed to couch interdisciplinary as a mandatory requirement, Objective 5 called for interdisciplinarity to be "encouraged." As used in this context, "encourage" is rather vague and since it is not possible to measure performance without targets or measurable indicators of success, it created problems with evaluability.

Recommendation 1: The original objectives and rationale of the STIHR Initiative were unclear and should therefore be revised in consultation with key STIHR stakeholder groups.

Recommendation 2: Strengthen STIHR performance measurement and data collection that focuses on outcomes (as outlined in the STIHR RMAF) both at the STIHR Initiative and Program levels.

Recommendation 3: The regular reporting of performance data (e.g., annual progress reports) should be revised so that the process of data collection is more streamlined, accurate and less burdensome. This should include a revision of the data collection tool(s) and the process that assesses performance of the individual STIHR Programs.

The findings, conclusions and recommendations on evaluation issues are presented below:

Evaluation Issue 1: Relevance of STIHR Initiative to CIHR, Federal government and the health research community

Key Findings:

• CIHR Managers in the stakeholder interviews agreed that the STIHR Initiative is contributing to achieving the CIHR mandate.
• Interdisciplinary skills were reported by respondents as valued, however, the marketability of these skills remains unknown.
• Interdisciplinary research training and capacity building were considered by respondents to be important and relevant components of the STIHR Initiative.
• Respondents rated the ethics component of the STIHR Initiative consistently low in importance and relevance as compared to other components of the program.
• Partners and Scientific Directors rated the STIHR Initiative's objectives as highly relevant to their organizations.
• In general, interviewees felt that the STIHR Initiative's objectives were less relevant to health researchers and the larger health research community than they were to CIHR and Institute objectives.
• Overall, the STIHR Initiative's objectives and the training model were considered by all categories of respondents to be both relevant and important to CIHR and external STIHR stakeholders.

Conclusion: The STIHR Initiative's interdisciplinary focus is appropriate and relevant to the CIHR mandate and stakeholders consider the STIHR Initiative to be important and provides good value added.

Recommendation 4: Review and revise the level of funding for the STIHR Initiative.

Evaluation Issue 2: Success of the STIHR Initiative in producing its desired outcomes

Key Findings:

• Many individual STIHR Programs are exceeding admission targets.
• Individual STIHR Programs are composed of trainees who come from many disciplines, representing approximately the same distribution and variety in terms of disciplines as the CGS/DRA/Fellowship recipient group surveyed.
• The STIHR Initiative is likely increasing the number of research training opportunities for Health Systems and Services as well as Clinical trainees.
• At this early stage, there is little difference between STIHR trainees and CGS/DRA/FE trainees with respect to quantity of publications or the interdisciplinary nature of publications and conference participation.⁶
• Though there is evidence of innovation in encouraging networks and linkages between trainees and researchers, the STIHR Initiative does not have enhanced linkages and networking between students and other trainees/researchers, as compared with CGS/DRA/FE.⁷
• Of the STIHR trainees who have finished their training and are currently working, most say that their job involves some type of interdisciplinary research or setting.

Conclusion: In the past five years, there are some indications that the individual STIHR Programs are reaching towards attaining some short-term and some intermediate-term outcomes. In particular, the STIHR Initiative appears to be increasing the number of research training opportunities for health systems and services and clinical trainees.

However, there is little evidence at this point to support the claim that these early outcomes are entirely attributable to the influence of the STIHR Initiative.

Evaluation Issue 3: Success of Partnerships

Key Findings:

• STIHR Initiative Partners' contributions represent an important component of the STIHR Initiative enabling the funding of individual STIHR Programs that could not have been funded otherwise.
• For most of the STIHR Initiative Partners there was no collaboration or interaction with the individual STIHR Programs beyond providing funding despite a desire for such ongoing collaboration.
• Only four STIHR Initiative Partners (of the total 11 interviewed) have stated that they will continue to support the STIHR Initiative in the subsequent launches; the rest are undecided (five out of 11) or will stop their funding (two out of 11).
• STIHR Initiative Partners perceived a lack of clarity in their role as partners.
• Almost half of the STIHR Initiative Partners reported that individual STIHR Programs influenced their organization by: confirming the benefits in partnering with organizations/programs that share the same or complimentary objectives, improving their own organization's visibility and offering their organization the possibility to further enhance their corporate goals within the research community.
• Most of the individual STIHR Programs reported having other partners involved in, or supporting their activities in addition to the STIHR Initiative Partners. Among these Program partners are other Canadian and international granting bodies, universities, and other public or private sector organizations.

Conclusion: The implementation of STIHR Initiative Partnerships has only been partially successful because beyond the funding contributions, there was no further collaboration or interaction with individual STIHR Programs for most of the Partners despite a desire for such ongoing collaboration.

Most of the individual STIHR Programs were successful in involving other partners in their programs beside the STIHR Initiative Partners. The most common benefits of including additional partners were an increase in funds, improvement of training opportunities for students, and an increase in the opportunities for Knowledge Translation.

Recommendation 5: Improve communication between STIHR Initiative partners and other program stakeholders including CIHR Research Capacity Development, CIHR Institutes, and STIHR Principal Investigators. This should include more pertinent and regular reporting to partners, a clarification of the role of partners and other STIHR stakeholders, inclusion of partners in key program decision-making bodies, and formal mechanisms to facilitate partner networking with STIHR stakeholders.

Evaluation Issue 4: Success of the STIHR Initiative in providing integrated training environments to groups of trainees

Key Findings:

• The experience of STIHR and CGS/DRA/FE trainees concerning mentoring is very similar.
• A higher percentage of STIHR trainees reported having a supervisor who is a different person from their mentor suggesting that as compared to CGS/DRA/FE trainees, STIHR trainees may be interacting with a larger pool of experienced scientists.
• As compared to CGS/DRA/FE trainees, STIHR trainees were more likely to rate their supervisors as excellent or good in providing support (intellectual and emotional) and effective mentoring implying that STIHR trainees may be more satisfied with their supervision experience.
• As compared to CGS/DRA/FE trainees, STIHR trainees were less likely to say that their program provided training "to a great extent" in communication, teamwork and leadership skills, and grant writing and peer review skills, and they were also less likely to rate the training as useful "to a great extent."
• STIHR and CGS/DRA/FE trainees equally believed that their programs provided an interdisciplinary training environment to "a great" or "some" extent.
• Similar proportions of STIHR and CGS/DRA/FE trainees reported that their training program's contribution to the development of their research skills had fully met or exceeded their expectations.

Conclusion:
Individual STIHR programs are considered by trainees to be as good training environments as any other training environments experienced by CGS/DRA/FE trainees.

STIHR trainees are more likely to be interacting with a larger pool of experienced scientists and appear more satisfied with their supervision experience.

Evaluation Issue 5: Implications of STIHR Initiative's selection processes for the potential success of funded STIHR Programs

Key Findings:

• The STIHR Initiative is potentially reaching a greater number of trainees than would have been possible if it were a funding program granting full scholarships to individuals due to its flexibility in accepting both full and part time trainees.
• There are no apparent differences in criteria, including undergraduate GPA and honours received, between the quality or caliber of STIHR trainees versus trainees receiving excellence-based scholarships directly from CIHR.
• There is a considerable amount of co-funding with other CIHR programs among STIHR trainees and many trainees have multiple sources of funding including other CIHR or non-CIHR excellence-based scholarship sources further indicating high quality of trainees.
• The most common way that STIHR trainees have learned about individual STIHR Programs is through a Program Leader or supervisor and secondly through word of mouth.
• Individual STIHR Programs report challenges in recruiting trainees at start-up but these issues are usually resolved within the first two years of program funding and individual Programs are currently operating close to full trainee capacity.
• Due to lack of baseline data it is unknown whether the STIHR Initiative is attracting additional staff members to Institutions but according to University representatives the STIHR Initiative is encouraging mentors to consider research outside their regular area of research and creating research communities that were absent prior to the Initiative.
• The majority of key informants believed that in the peer review process for selecting STIHR Programs, excellence of the mentors' previous research programs is an important criterion and that there are no overriding criteria. Other criteria considered important were mentorship -- training programs should be focused on the relationship between mentors and trainees, the PI's ability to teach/train, providing younger faculty with the opportunity to be a mentor, innovation, assurances of a robust training program and institutional support, and interdisciplinarity.

Conclusion: The STIHR Initiative appears to be attracting the same caliber of trainees as comparable programs and the trainee experience in STIHR Programs does not vary in a measurable way from the trainee experience when funded through CGS, DRA and Fellowship programs.

After the first two years of implementation, individual STIHR Programs were relatively stable.

Management Response

Recommendation 1: The original objectives and rationale of the STIHR Initiative were unclear and should therefore be revised in consultation with key STIHR stakeholder groups.
Response (Agree or Disagree): Agree.
Action Plan: Working group to work on objectives for next STIHR launch. Included Evaluation team on working group.
Responsibility: RCD
Timeline: September 2007 - December 2007
Actions Taken: STIHR objectives were modified in time for the re-launch of the program in January 2008 - completed.

Recommendation 2: Strengthen STIHR performance measurement and data collection that focuses on outcomes (as outlined in the STIHR RMAF) both at the Initiative and Program levels.
Response (Agree or Disagree): Agree.
Action Plan: Modify performance measurement procedures and templates (progress and mid-term reports). Link performance measurements directly to the modified objectives.
Responsibility: RCD and Evaluation
Timeline: April 2009
Actions Taken: Discussions are being initiated with the ResearchNet team and the Evaluation team to ensure that the focus on outcomes is emphasized in the ongoing plan to integrate STIHR reporting with the CIHR Research Reporting System that is currently being developed.

Recommendation 3: The regular reporting of performance data (e.g. annual progress reports) should be revised so that the process of data collection is more streamlined, accurate and less burdensome. This should include a revision of the data collection tool(s) and the process that assesses performance of the individual STIHR Programs.
Response (Agree or Disagree): Agree.
Action Plan: Modify Progress Report procedures. Modify Progress Report templates. Link performance measurements directly to modified objectives.
Responsibility: RCD and Evaluation
Timeline: November 2008 - November 2009
Other Comments: We will pilot any changes with two new STIHR programs funded in an ad-hoc competition. Funding began in 2007, so the first progress report will be due November 2008. Annual progress reports should no longer be peer reviewed by Institutes, they should simply be a mechanism for data collection and information sharing.
Actions Taken: Discussions are being initiated with the ResearchNet team and web services, as well as evaluation team to ensure that the questions being asked in the progress reports are relevant and align with objectives. The goal is to integrate STIHR reporting with the CIHR Research Reporting System (currently under development) as much as possible.

Recommendation 4: Review and revise the level of funding for the STIHR Initiative.
Response (Agree or Disagree): Agree.
Action Plan: Prepare documentation for senior management to make decisions on level of funding that should be committed in re-launch of program.
Responsibility: RCD
Timeline: December 2007
Actions Taken: RKTC approved re-launch for January 2008 with significant reductions in funding.

Recommendation 5: Improve communication between STIHR Initiative partners and other program stakeholders including CIHR Research Capacity Development (RCD), CIHR Institutes, and STIHR Principal Investigators. This should include more pertinent and regular reporting to STIHR partners, a clarification of the role of partners and other STIHR stakeholders, inclusion of partners in key program decision-making bodies, and formal mechanisms to facilitate partner networking with STIHR stakeholders.
Response (Agree or Disagree): Agree.
Action Plan: Involve partners in the creation of Funding Opportunities. The role of partners in the STIHR Initiative will be made clearer.
Responsibility: RCD and Partnerships and Citizen Engagement
Other Comments: Partners wish to better understand the program before they decide on whether to participate. It should be noted that of the partners that are not participating in the re-launch of the program, the issue is not necessarily a lack of CIHR communication. Some have created a similar program on their own and others do not have the funds to commit at this time, particularly since the STIHR grants are such a long commitment (six years).
Actions Taken: We have included partners, institutes, etc. in most of the decision making of the STIHR program to date, including all changes to the relaunch of the program. Partners were contacted and briefed on the relaunch as part of the process of asking them to participate.

1. CIHR direct awards include Canada Graduate Scholarships, Doctoral Research Awards and CIHR Fellowships.
2. CIHR Strategic Training Program Grants 2002 Competition, Instructions for the Full Application, p. 1.
3. Transdisciplinarity is defined here as "the integration and transformation of fields of knowledge from multiple perspectives in order to define, address and resolve complex real-world problems." See STIHR Performance Measurement and Evaluation Framework.
4. A purposive sampling strategy was used and looked to interview a maximum of five university staff and six from CIHR top management.
5. Preliminary data analysis indicated considerable overlap with other CIHR awards among STIHR trainees: 31% of all trainee respondents have received Fellowship awards, 25% DRAs and 17% CGS. Overall, 66% have received funding through another CIHR program. Thus when the overlap is excluded from the analysis, the remaining n is too low to allow for any meaningful discussion of differences in statistically significant terms. This impacted our ability to do statistical tests on differences since doing so would attribute more importance to any statistical or factual difference than the data could support.
6. This could be due to the previously discussed problems of lack of mutual exclusivity and comparability of the two groups which impacted our ability to make any clear distinctions between them.
7. This could be due to the previously discussed problems of lack of mutual exclusivity and comparability of the two groups which impacted our ability to make any clear distinctions between them.