Mid-Term Evaluation of the Institute of Health Services and Policy Research (IHSPR)

Executive Summary

December 2005

1. Executive Summary
    1.1 Overview of CIHR
    1.2 Overview of the Institute of Health Services and Policy Research (IHSPR)
        1.2.1 Mandate
        1.2.2 Strategic Research Priority Themes
    1.3 Evaluation Objectives and Issues
        1.3.1 Methodology
2. Evaluation Results
    2.1 Relevance
    2.2 Effectiveness
    2.3 Delivery
3. Recommendations
4. Management Response

1. Executive Summary

1.1 Overview of CIHR

The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

CIHR was created under The CIHR Act that came into force on June 7, 2000.

Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).

In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:

outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.

These strategic outcomes will be enabled through:

effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.

CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.

CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.

Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.

CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).

1.2 Overview of the Institute of Health Services and Policy Research (IHSPR)

1.2.1 Mandate

The IHSPR was established to support innovative research, capacity-building and knowledge translation initiatives geared towards improving the way health care services are organized, regulated, managed, financed, paid for, used and delivered, in the overarching interest of improving the health and quality of life of all Canadians.

1.2.2 Strategic Research Priority Themes

Research themes identified from the IHSPR 2004 priority-setting process¹, Listening for Direction II, a national consultation on health services and policy issues for 2004-2007, are the following:

Health workforce planning, training, and regulation;
Management of the healthcare workplace;
Timely access to quality care for all;
Managing for quality and safety;
Understanding and responding to public expectations;
Sustainable funding and ethical resource allocation;
Governance and accountability;
Managing and adapting to change;
Linking care across place, time, and settings; and
Linking public health to health services.

1.3 Evaluation Objectives And Issues

The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:

to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.

The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:

Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?

The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.

1.3.1 Methodology

The evaluation consisted of four main lines of evidence:

a review of documents and administrative data relating to the IHSPR;
45 key informant interviews with IHSPR staff and IAB members, researchers and students, stakeholders and partners, as well as a focus group discussion with IAB members and one with stakeholders;
case studies of four IHSPR initiatives; and
a telephone survey of 88 funded and 46 non-funded researchers affiliated with the IHSPR.²

2. Evaluation Results

2.1 Relevance

The evaluation evidence indicates that the majority of key informants across all respondent groups are satisfied with the mandate and strategic priorities that currently guide the IHSPR. The original mandate of the IHSPR, provided by CIHR, was refined during the strategic planning process to better reflect the interests of the community. Some noted, however, that in addition to this formal mandate there is an expectation on IHSPR to grow Theme 3 (health systems and services) for CIHR, and if this is the case perhaps this should be reflected in the mandate statement. The Listening for Direction priority-setting process led to the development of the strategic research priority themes which were perceived by key informants to accurately reflect the views and interests of the Institute's constituency and to dovetail with the priorities of other Institutes. The majority of the Institute's affiliated researchers see a close fit between their research interests and the IHSPR mandate (though this is much less true among non-funded researchers who are aligned with the IHSPR). Most key informants also regard the research domains within the IHSPR's mandate as appropriate. There were no substantial modifications suggested to the strategic priorities of the Institute at this time.

There is broad support among key informants for the IHSPR as an appropriate mechanism to make a difference in its research domain. Moreover, researchers surveyed believe there is a great need for the IHSPR to support research excellence, capacity development and funding of strategic priorities (the latter two ratings being substantially higher compared to CIHR researchers overall referring to their own Institute). This strong endorsement of the need for the IHSPR is significant. In this new research field with a need to grow a community, having an Institute that is able to develop Canadian capacity and support strategic priorities of importance to policy makers is critical.

In the view of a minority of key informants, IHSPR may not have the capacity/resources to achieve its dual role (to advance its own strategic agenda and to champion Theme 3 across CIHR and other Institutes). As well, key informants also emphasized the important, complementary role of other players in this domain (e.g., CHSRF³ and CAHSPR⁴) in addressing research excellence, capacity and, particularly knowledge translation.

Few alternatives to the IHSPR mechanism were identified. Also, the CIHR model of virtual Institutes is generally regarded as appropriate, though some concerns were expressed, for example, about potential disruption with the transition of SDs.

2.2 Effectiveness

Overall, IHSPR has performed well and has made good progress in contributing to CIHR's overall objective and in fulfilling the Institute's mandate and strategic priorities, even though it had to invest a significant share of its strategic funds in the initial years to capacity development.

IHSPR is making progress in creating new knowledge through the provision of strategic funding through RFAs.⁵ Most key informants across all respondent categories agree that the Institute is making a contribution in this area, though some are of the view that it is difficult to draw this conclusion because the work of all funded researchers is not tracked or it is too soon to conclude how much new knowledge will truly be created as a direct result of the IHSPR and its funding.

The majority of key informants agree that knowledge translation has been a priority for the Institute and IHSPR contributions include such activities as providing support and leadership within CIHR and to other Institutes, through RFAs, championing the creation of a new journal in the field (Healthcare Policy), contributions to national policy development commissions among others. Most funded IHSPR researchers surveyed believe that the Institute's activities support KT (more so than CIHR researchers overall, referring to their own Institute) and researchers are highly involved in disseminating their research findings. A minority of interviewees believe that progress on KT has been less than initially hoped due, at least in part, to the Institute's initial focus on capacity building. Other perceived challenges are a lack of clarity in the role/appropriateness of Institutes in the KT area, as well as a lack of clear guidance from CIHR central as to what Institutes are expected to do.

IHSPR has made significant investments in capacity building through numerous initiatives, including CIHR mechanisms (ICE and NET teams, STIHRs⁶ and Training Awards), as well as other activities (CADRE, Summer Institutes, support for CAHSPR). Key informants generally agree that IHSPR has made a strong contribution to capacity development. The number of trainees in this research domain is increasing. Similarly, the majority of funded researchers in the survey feel that IHSPR has contributed, to at least some extent, to the development of people and the research environment. IHSPR stakeholders note that the Institute, on its own, cannot fully address issues of capacity and that more support for health services and policy research is necessary from universities for sustainable increases in capacity to occur.

There is broad consensus among key informants in all respondent categories that IHSPR is contributing to CIHR's transformative vision. The Institute, as champion of Theme 3, in itself represents a significant departure from the former biomedical model of the Medical Research Council. The Institute has also supported interdisciplinary research teams, promoted and included knowledge translation as a key element in the research process, and successfully advocated for changes to CIHR granting mechanisms to ensure their appropriateness for Theme 3 (and 4) researchers. In the survey, funded IHSPR researchers reported that the vast majority of their projects are interdisciplinary and include a KT component. Most of the CIHR funding for research related to the IHSPR's mandate is in Theme 3 and, to a much lesser extent, Themes 1, 2 and 4.

The Institute has also made a significant contribution to the CIHR ethics mandate, for example, by supporting the CIHR ethics office, contributing financially to joint strategic initiatives with some relation to ethics issues and related activities (e.g., workshops, sponsoring of "Data, Data Everywhere" which raised privacy protection issues). There is low awareness of the ethics mandate and the Institute's contribution in this regard, however, among many key informants.

Interviewees generally feel that IHSPR is performing moderately to very well relative to its mandate and strategic priorities. In terms of the Institute's original BEST⁷ strategy, evaluation evidence suggests that most progress has been made in the area of building a community of excellent researchers, while progress in expanding and enhancing research resources is less than originally hoped (given the challenging and time-consuming nature of this work). The Institute has made numerous contributions toward its top priority strategic research themes. Furthermore, the Institute has made a significant contribution to the organizational excellence of CIHR as a whole (e.g., through its provision of expertise and assistance for corporate files such as KT, ethics and evaluation, and participation on numerous CIHR committees and working groups). While key informants note that the IHSPR has played a leadership role in KT, funded IHSPR researchers rate the Institute's contribution to date in this area as lower than that for supporting strategic research and building capacity. The vast majority of these researchers, however, perceive that the overall mandate of the Institute has been achieved to at least some extent. CIHR investments in research related to health services and policy have increased significantly over the past five years, further indicating that IHSPR is influencing research under its mandate.

IHSPR has collaborated extensively and developed numerous partnerships, linkages and alliances with a broad range of stakeholders from numerous sectors. The Institute has worked effectively with other Institutes, partnering for 98% of its RFAs, as well as holding joint meetings and workshops and joint funding of grants submitted to open competitions. The Institute's Performance Report and key informants note extensive and diverse linkages with other organizations, including federal agencies such as CIHI, the Conference of Federal, Provincial and Territorial Deputy Ministers of Health, voluntary organizations such as Safe Kids Canada and professional associations (the former Canadian Health Economics Research Association). A major and very productive partnership has been with the Canadian Health Services Research Foundation (CHSRF). Also, most funded IHSPR researchers indicate that the Institute has facilitated collaboration within their research community.

According to most key informants, IHSPR has demonstrated leadership and has had an influence on research agendas in its community. The leadership of the SD was praised (among stakeholders and researchers in particular). The Listening for Direction process identified strategic research priorities and a plan for coordination, and initiatives such as ICE and NET team grants, strategic priority RFAs (e.g., genetics and health services, health human resources) and quick response RFAs related to SARS and wait times were noted as examples where the Institute has provided leadership. Leadership is also illustrated by the Institute's efforts to support the broader community such as through the formation of CAHSPR and the creation of the Healthcare Policy journal.

2.3 Delivery

IHSPR has used strategic planning mechanisms very effectively. The Institute has undertaken extensive consultations with researchers and stakeholders to seek input and identify strategic and emerging priorities (e.g., Listening for Direction I and II, Strengthening the Foundations Symposium, workshop programs, SD's meetings, IAB consultations). Moreover, almost all researchers surveyed feel that the IHSPR has successfully identified and responded to emerging priorities within its mandate, to at least some extent.

The IAB has been operating effectively and contributing to the achievements of the Institute (e.g., by providing useful input into planning, RFAs, ICE teams, KT). Staff and Board members are highly satisfied with the operation and effectiveness of the IAB, though some concerns are expressed by a minority around mandatory turnover of membership and limited responsibility of members after budgetary commitments have been made.

IHSPR has consulted extensively and effectively with a wide range of stakeholders using both formal and informal methods. Key informants and researchers surveyed feel that the IHSPR has been responsive to input and feedback. In addition, the Institute has utilized a wide range of communications approaches/vehicles. Some key informants feel, however, that the IHSPR's communications could be improved (e.g., more communications with researchers in the community as well as policy makers and decision makers, improved Web site navigation).

Factors that exert some influence on the effectiveness of IHSPR include central CIHR functions and funding levels. Among interviewees who were able to comment, responsiveness of CIHR central to the needs of the Institute is noted to have been weak initially, with particular concerns around knowledge translation, data analysis, communications, and information technology (perceived to be due, at least in part, to staff turnover in these areas).

Most key informants view the level of funding ($1M for the Institute support grant and approximately $6.5M for strategic funding) as inadequate for the Institute to make a difference under its dual mandate. Still some (e.g., some IAB members) feel that the level was adequate for the Institute's first five years (e.g., due to the limited initial capacity in the researcher community), but now there is a need for additional funding. Although mixed views are expressed on the 70:30 split between funding for investigator-initiated and strategic research, the majority regard the balance as either appropriate or would prefer a higher proportion for strategic funding (stakeholders in particular).

3. Recommendations

Following are recommendations that emerge from this mid-term evaluation of IHSPR. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.

Recommendation 1:

The Institute has been found to be doing well and is encouraged to continue the following:

sustain its efforts in supporting research excellence, capacity development and funding of strategic priorities in the field of health services and policy research, in collaboration with other key players such as CHSRF and CAHSPR;
maintain the current structure and operation of the IAB as it is functioning effectively;
maintain planning mechanisms, as they are effective;
sustain its efforts in consulting with researchers and a diversity of stakeholders using a range of formal and informal methods;
maintain its work in collaborating and developing partnerships with a range of organizations and sectors;
sustain its efforts related to knowledge translation;
sustain its efforts related to CIHR's ethics mandate, and that it inform its researcher and stakeholder community about these efforts and any achievements; and
sustain its efforts in contributing to the transformative vision of CIHR through its emphasis on building capacity in Theme 3, interdisciplinary research and knowledge translation.

The following areas are ones in which it is recommended the Institute take action to improve:

Recommendation 2:

There is a substantive degree to which IHSPR has become responsible for the promotion of theme three research not only within its own community, but also across CIHR. This role, however, is not formally acknowledged. IHSPR should undertake to clarify roles and responsibilities with CIHR corporate on the promotion and funding of theme three research and capacity development across CIHR.

Recommendation 3:

IAB Functioning - The Institute should ensure that all new members are given a thorough orientation to minimize the disruption of member rotation.

Recommendation 4:

Capacity Building - The Institute (along with other players such as CHSRF) should endeavour to work with Canadian universities to help ensure that they support students and faculty in health services and policy research so that capacity will be sustained.

Recommendation 5:

Knowledge Translation - It is recommended that IHSPR seek clarification with CIHR as well as other key players (e.g., CHSRF) on the most suitable role in KT it can feasibly take on, considering its available resources.

Recommendation 6:

Communications - IHSPR should sustain its communications activities, using a variety of approaches/vehicles, and if feasible consider some improvements raised by key informants in this evaluation - for instance, making the Web site more user-friendly, and increasing the amount/frequency of communications to the research community (including clinical researchers) and to policy makers and decision makers (e.g., CEOs, Regional Health Authorities).

Recommendation 7:

Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.

4. Management Response

(Recommendation 1 notes that IHSPR "has been found to be doing well and is encouraged to continue" certain activities; IHSPR agrees)

Recommendation	Response	Action Plan
2) Theme 3 Responsibilities -There is a substantive degree to which IHSPR has become responsible for the promotion of Theme three research not only within its own community, but also across CIHR. This role, however, is not formally acknowledged. IHSPR should undertake to clarify roles and responsibilities with CIHR corporate on the promotion and funding of theme three research and capacity development across CIHR.	Agree	1) Theme 3 is part of CIHR's organization-wide mandate, and should be led but not entirely carried by IHSPR. Therefore: IHSPR's IAB Chair will write a letter to Dr. Bernstein proposing the formation of a limited purpose 'Theme 3 Advisory Group' for strengthening Theme 3 support at CIHR. The advisory group will create a development and resourcing plan (including recommendations to GC, the President ,IHSPR and the other institutes). 2) Preliminary discussion with Dr. Bernstein at an IHSPR IAB meeting regarding CIHR's Theme 3 mandate in general, and opportunities for action (including the proposed advisory group). 3) IHSPR's SD will raise the issue of CIHR's need for an enhanced Theme 3 agenda, at RPPC.
3) IAB Functioning - The Institute should ensure that all new members are given a thorough orientation to minimize the disruption of member rotation.	Agree (already actioned)	1) IHSPR has been a vocal advocate for less frequent IAB member rotation. 2) IHSPR was instrumental in developing a CIHR prototype IAB member orientation manual, for adaptation and use by all institutes. 3) For the past two cycles, IHSPR has provided a collection of documents to new IAB members, in addition to the orientation CD/manual created by CIHR corporate. It will continue to do so. 4) IHSPR will incorporate a half-day orientation session for new IAB members into its September IAB meetings. 5) As part of its Policy & Procedure Manual, the institute is in the process of developing a 'tick list' and timeline for IAB member orientation.
4) Capacity Building - The Institute (along with other players such as CHSRF) should endeavour to work with Canadian universities to help ensure that they support students and faculty in health services and policy research so that capacity will be sustained.	Agree	1) IHSPR continues to work collaboratively with CHSRF on developing an increased understanding of issues that hspr faculty and students face at Canadian universities, and the opportunities and efforts underway toward their resolution. 2) IHSPR will continue to provide financial and in-kind support to the Network of Applied Health Services Research Centre Directors, in order to (amongst other purposes) understand and influence university culture as relates to faculty and student support. 3) IHSPR and CHSRFare collaborating on a joint hspr research community development plan that will include a 'funded initiatives profiling plan', targeted at universities. 4) IHSPR will explore the potential value and feasibility of engaging university delegates as partners for addressing challenges related to HSPR faculty and student support within their respective universities. 5) IHSPR will re-engage its IAB on this issue at an upcoming IAB meeting. 6) IAB and current SD will provide a full briefing to the incoming SD.
5) Knowledge Translation - It is recommended that IHSPR seek clarification with CIHR as well as other key players (e.g., CHSRF) on the most suitable role in KT it can feasibly take on, considering its available resources.	Agree	1) IHSPR's IAB Chair recently submitted a letter to Dr. Bernstein encouraging renewed and increased efforts to recruit a VP KT, and to ramp up KT Branch Staff. 2) IHSPR's IAB KT Working Group includes a CHSRF representative. This enables an ongoing exchange of ideas for KT planning and action. 3) IHSPR will invite a CIHR KT Branch rep to join its IAB KT Working Group. 4) IHSPR's SD will continue to use the RPPC table to press for greater coherence to CIHR's organization-wide KT agenda (and will continue to bring KT experts to that forum).
6) Communications - IHSPR should sustain its communications activities, using a variety of approaches/vehicles, and if feasible consider some improvements raised by key informants in this evaluation - for instance, making the Web site more user-friendly, and increasing the amount/frequency of communications to the research community (including clinical researchers) and to policy makers and decision makers (e.g., CEOs, Regional Health Authorities).	Agree	1) All institute communication plans and activities will be undertaken in compliance with corporate policy, and with the participation of corporate staff (e.g. Communications, and KT). 2) In the Spring of 2005, IHSPR added a full-time staff person to lead and coordinate Institute communication activities. 3) Institute Web site developments will continue to be conducted in compliance with corporate policy. (CIHR Corporate continues to upgrade corporate and institute Web site protocols. IHSPR has no independent control over key aspects of CIHR's Web site such as layout, user interface, navigability and search capacity). 4) Web maintenance is now consolidated under one IHSPR staff person. A schedule for ongoing Web site maintenance is being developed. 5) IHSPR will continue to use its listserv and newsletter to provide timely and relevant news related to CIHR funding opportunities, funding decisions and other organizational news to the HSPR community. 6) IHSPR will continue to work with CAHSPR to improve the association's capacity to act as a conduit for HSPR-related news to its community of researchers and decision makers via its listserv. 7) IHSPR's three-times-yearly Newsletter will be evaluated for community satisfaction and effectiveness. 8) IHSPR will continue to produce Research Spotlights as a means of communicating research investments and outcomes to the broader scientific community and decision makers.
7) Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.	Agree	1) IHSPR's IAB Evaluation Working Group will review the CIHR Blueprint, IHSPR Strategic Plan and the Common Indicators to determine whether additional reporting areas are required for IHSPR. 2) IHSPR's public annual reports will incorporate / adapt to the 15 Common Indicators. 3) IHSPR's performance indicators and targets will be incorporated into: Corporate Annual Reports Public Annual Reports Annual Operational Plans Quarterly Operational Reports

¹IHSPR was one of six national partners in Listening for Direction II.
²Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
³The Canadian Health Services Research Foundation (CHSRF)
⁴The Canadian Association for Health Services and Policy Research (CAHSPR), which replaced the former Canadian Health Economics Research Association in late 2003.
⁵A request for application (RFA) is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.
⁶A STIHR (Strategic Training Initiative in Health Research) consists of innovative and interdisciplinary training programs that support the development of new researchers. Each program can receive up to $310,500 a year for up to 6 years and at least 71% of these funds must be used for trainee stipends or travel between training locations.
⁷BEST is the acronym for the strategy intended to Build a community of excellent researchers, Expand and enhance research resources, Support strategic research and Translate knowledge.

Breadcrumb trail