Mid-Term Evaluation of the Institute of Circulatory and Respiratory Health (ICRH)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Circulatory and Respiratory Health (ICRH)
1.2.1 Mandate
1.2.2 Vision and Values
1.2.3 Strategic Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Circulatory and Respiratory Health (ICRH)
The ICRH's aim is to support research into causes, mechanisms, prevention, screening, diagnosis, treatment, support systems, and palliation for conditions associated with the heart, lung, brain (stroke), blood, blood vessels, critical care and sleep.
The ICRH's vision is to achieve international leadership through fostering an environment of extraordinary openness, excitement, energy, commitment and excellence in highly ethical, partnered initiatives focused on research, research training, and research translation for the circulatory and respiratory sciences and for the betterment of the health of Canadians.
Its values are the following: Innovative people; innovative programs of research; interdisciplinary research findings; internationally recognized, ethical research; impact of findings on health; and inclusiveness.
1.2.3 Strategic Research Priorities
Based on the ICRH's May 2003 Strategic Outlook and 2002 Strategic Plan ("Reaching over the Horizon: A strategic plan for circulatory and respiratory health research"), the Institute's strategic priorities, which align with the four strategic directions of CIHR (as described in "Revolution, CIHR: Towards a National Research Agenda"), were the following:
Strategic Research Initiatives (5)
- Interaction of genes and environment in determining susceptibility to circulatory and respiratory diseases;
- Fetal-maternal influences on circulatory and respiratory diseases;
- Novel and integrative approaches to the assessment, care and management of patients with circulatory and respiratory diseases;
- Obesity-associated vascular and respiratory conditions; and
- Self-regeneration, repair and replacement of damaged cells, tissues and organs in circulatory and respiratory disease.
Research Priorities for Further Development (10)
- Building synergy in the blood research community: Setting research priorities in the area of thrombosis;
- Operational and ethical issues associated with banking human biological materials;
- Legal and ethical issues in circulatory and respiratory research;
- Improving our understanding of circulatory and respiratory diseases using tools rooted in the computational sciences and mathematics;
- E-research: Information technology applications in the Canadian circulatory and Infectious causes of circulatory and respiratory diseases;
- The influence of sex and gender on the risk of cardiovascular disease;
- Altering the disease consequences of cystic fibrosis;
- Cardiovascular complications of diabetes mellitus; and
- Global health research.
- Training Initiatives through:
- Strategic training program grants;
- Personnel awards;
- Clinical research and clinician scientists; and
- Institutional establishment grants.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to the ICRH;
- 42 key informant interviews with ICRH staff and IAB members, researchers and students, stakeholders and partners;
- a case study of one ICRH initiative; and
- a telephone survey of 188 funded and 51 non-funded researchers affiliated with the ICRH. 1
2. Evaluation Results
The evaluation evidence indicates that the mandate and priorities of the ICRH are appropriate and relevant. A majority of key informants in all respondent groups regard the Institute's mandate and strategic research priorities as appropriate, though a minority feels the scope of the mandate and several priorities is too broad in light of limited resources. In addition, in the survey three-quarters of funded ICRH researchers and over half of non-funded researchers see a close fit of their research interests with the Institute's mandate. The research domains within the mandate are also viewed as appropriate, though some key informants feel that some domains (e.g., critical and intensive care, sleep) do not fit as well.
ICRH is generally viewed as an appropriate mechanism to make a difference in its research domain of circulatory and respiratory health. Furthermore, researchers surveyed believe there is a great need for the ICRH to support research excellence, capacity development and funding of strategic priorities. No major alternatives to the ICRH mechanism were identified, although some modifications to the structure of CIHR were suggested (e.g., reducing, regrouping or centralizing Institutes). The CIHR model of virtual Institutes is generally regarded as appropriate, with both advantages and disadvantages. For example, it is advantageous that Institutes are located across the country (as opposed to all being in Ottawa) and that this flexible model does not require researchers to move away from their home institution. On the other hand, disadvantages include high travel costs for Institutes like ICRH that are located far from Ottawa as well as the disruption and loss of corporate memory that may accompany the transition of SDs and staff.
ICRH has performed well, making good progress in contributing to CIHR's overall objective and in fulfilling the Institute's mandate and strategic research priorities.
Through its efforts in identifying areas of strategic and emerging research importance (e.g., New Frontiers Program, four annual Partners Forums) as well as through Requests for Applications (RFAs)2 and the funding of research in priority areas, ICRH is making progress in creating new knowledge in the field of circulatory and respiratory health. Most funded ICRH researchers believe that the Institute has contributed to at least some degree in advancing health research to respond to opportunities/priorities and create health knowledge. Most key informants also agree that progress is being made toward knowledge creation.
With respect to knowledge translation (KT), ICRH has funded or implemented a number of KT-related activities (e.g., Partners Forums and workshops addressing KT-related issues, supporting KT through communications products, plans to develop a KT Case Book). Most funded ICRH researchers surveyed believe that the Institute's activities support KT to at least some extent and researchers indicate sharing their research findings, mostly with other researchers and health care professionals. Also, key informants generally agree that knowledge translation is a priority for the ICRH, though many note that it is too early to assess the Institute's contribution in translating knowledge or they do not know. They also stress that KT is a challenging area, for instance, due to a lack of clarity in the definition and guidelines for KT from CIHR central, the reliance on the efforts of partner organizations, and the need to engage policy makers.
ICRH has contributed significantly toward capacity building through a range of initiatives (e.g., Young Investigators Forum, New Emerging Team grants, Institute Establishment Grant, Focus on Stroke awards/fellowships) and through increasing investments in STIHRs3 and Training Awards. Key informants generally agree that ICRH has made a strong contribution to capacity development. Similarly, the majority of funded ICRH researchers in the survey feel that the Institute has contributed, to at least some extent, to the development of people and the research environment.
Key informants agree that ICRH is making a major contribution to the transformative vision of CIHR through its fostering of interdisciplinary research in circulatory/respiratory health and its support for research in strategic priority areas. Moreover, in the survey, funded ICRH researchers report that an average of 62% of their projects are interdisciplinary. CIHR administrative data indicate, however, that funding by CIHR as a whole for research related to the ICRH mandate is primarily in Theme 1 (biomedical) and, to a lesser extent, Theme 2 (clinical) with very small proportions being directed to the other two themes. The percentages of research funded across the four themes by the ICRH itself are much more distributive.
The Institute has also made a modest contribution to the CIHR ethics mandate, for example, by contributing financially to strategic initiatives with some relation to ethics issues, by addressing ethical issues as part of a national discussion on human data sources in circulatory/respiratory sciences, and by having an ethics representative on the IAB. There is, however, low awareness of the ethics mandate and the Institute's contribution in this regard among many key informants.
Key informants generally believe, and documentation confirms, that ICRH is progressing well relative to its mandate and priorities, in particular, by supporting the advancement of knowledge in these strategic areas but also through capacity development, partnerships and fostering of interdisciplinary research. Similarly, the majority of funded ICRH researchers perceive that the mandate and strategic priorities have been achieved to at least some extent. ICRH investments through grants and awards have covered the range of research domains and areas within its mandate well, though sleep research has received notably less investment. In addition, CIHR investments in research related to circulatory and respiratory health have increased steadily over the past five years, further indicating that ICRH is influencing research under its mandate.
Key informant interviews and the document review indicate that ICRH has collaborated extensively and developed numerous partnerships, linkages and alliances with a broad range of stakeholders from numerous sectors. In addition, substantial financial resources have been leveraged (e.g., almost $15.5M contributed by external partners). ICRH has also worked effectively with other CIHR Institutes (e.g., as partners/contributors in joint RFAs and major Institute initiatives), though a minority of IAB members would like to see more cross-Institute collaboration.
Despite the success of partnerships, there are potential barriers and challenges to partnership development, including issues of ownership, visibility and control, trust, limited Institute resources, the complexity and cost of managing large partnerships, and related rules and regulations of CIHR and the federal government. Most interview respondents agree that ICRH has provided the right leadership in managing partnerships across communities. In addition, interviewees believe that there are new opportunities for creative partnering on common international and global health issues, and that there is a continued role for ICRH to play in facilitating large-scale, cross-cutting activities.
ICRH has demonstrated leadership and has had some influence on research agendas in circulatory and respiratory health. Most key informants agree that the Institute has demonstrated leadership and influence through extensive engagement of stakeholders and researchers in the setting of strategic priorities, through RFAs/funding of research in its priority areas and by encouraging collaborative interdisciplinary research, for example, through New Emerging Teams. Most funded ICRH researchers in the survey also indicated that the Institute has influenced the research agenda in this field to at least some extent.
ICRH has consulted extensively as part of the planning process (e.g., through the New Frontiers Program workshops and symposium, annual Partners Forums and ICRH Strategic Planning Consultation) and has used planning mechanisms effectively. Most ICRH staff and Board members feel that strategic and operational planning mechanisms have been effectively used by the ICRH. However, stakeholders and researchers expressed mixed views and suggested some improvements (e.g., inclusion of health care providers and smaller players). In the survey, most researchers indicated that the ICRH has been at least somewhat successful in identifying emerging priorities within its mandate.
The IAB has been operating effectively and contributing to the achievements of the Institute (e.g., strategic planning and key initiatives). Most ICRH staff and IAB members expressed satisfaction with the functioning and effectiveness of the Board, in particular, its composition/membership, discussions and contribution to the research priority-setting process. Some concerns were raised, however, about the degree of engagement of Board members on some topics/projects (possibly due to a lack of focused communications to the Board), and some lack of clarity in roles and responsibilities (e.g., for the Chair and non-scientist members).
ICRH has consulted extensively with a wide range of stakeholders using both formal and informal methods.
The consensus among key informants, supported by documentary evidence, is that the ICRH and its Scientific Director have gone to considerable lengths to consult stakeholders, and most feel that this input and feedback is taken into consideration in Institute plans and activities and that it responds well to the needs of researchers. Almost all stakeholders and most researchers feel that they have had sufficient participation in the process of defining the Institute's priorities, or at least the opportunity to do so. In addition, the Institute has utilized a wide range of communications approaches/vehicles, however, some key informants feel that the communications could be improved (e.g., with better targeting, more resources and a formal communications strategy).
Factors that exert some influence on the effectiveness of the ICRH include central CIHR functions and funding levels. Key informants feel that CIHR central has been generally responsive to the Institute and are satisfied with the human resources and communications functions. Some concerns are raised, however, regarding central functions in the areas of ethics, knowledge translation, evaluation, IT, finance and Governing Council. Most key informants view the level of funding ($1M for the Institute support grant and approximately $6.5M for strategic funding) as inadequate to make a difference in all areas of ICRH's broad mandate. Views are mixed on the appropriateness of the 70:30 split between investigator-initiated and strategic research funding.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of ICRH. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- sustain its efforts in supporting research excellence, capacity development and funding of strategic priorities in the field of circulatory and respiratory health;
- maintain its planning mechanisms, ensuring that its researcher/stakeholder community is adequately consulted as plans are revisited and refined;
- sustain its productive efforts in building capacity, utilizing a range of funding vehicles and initiatives;
- sustain its efforts directed toward the creation of knowledge in the field of circulatory and respiratory health, utilizing its RFAs/funding vehicles and major initiatives; and
- sustain its efforts in contributing to the transformative vision of CIHR through its emphasis on interdisciplinary research in areas of strategic importance and knowledge translation for the ultimate benefit of the health of Canadians.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
IAB Functioning - While the IAB is generally considered to be running well, the Institute should endeavour to review and (if necessary) clarify the roles and responsibilities of the Chair and members (including non-scientists), including responsibilities related to knowledge translation. It should also ensure that Board members are provided with focused communications on all topics in advance of meetings so all members can be fully engaged.
Recommendation 3:
Planning Mechanisms - ICRH should consider periodic refinements to its priorities, based on its previous investments and successes, to ensure that the priority areas of most importance to its communities receive attention.
Recommendation 4:
Consultation and Communication - The Institute should maintain its communications activities, using a variety of approaches/vehicles, and if feasible consider some improvements suggested by key informants in this evaluation - for example, devote more resources to communications, develop a formal communications action plan, and send targeted communications to particular sub-groups such as policy makers, private industry and the general public. Efforts should be made to engage/consult with any stakeholders/sectors that have been less engaged to date (e.g., provincial health policy makers, non-profit sector, researchers in critical and intensive care).
Recommendation 5:
Collaboration and Partnerships - While overall encouraged to sustain its work in collaborating and developing partnerships with a range of organizations and sectors, the Institute should, to the extent feasible, also explore opportunities to develop partnerships on common international and global health issues in its research domain and to take the role of convenor or facilitator of large-scale, cross-cutting initiatives (e.g., in public health).
Recommendation 6:
Knowledge Translation - The ICRH should review its efforts in support of KT. The efforts expended by the ICRH in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the ICRH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 7:
Ethics - The Institute is encouraged to maintain and further develop its efforts related to CIHR's ethics mandate, and that it inform its researcher and stakeholder community about these efforts and any achievements.
Recommendation 8:
Contribution to Transformative Vision - While ICRH has been making progress in terms of the implementation of the transformative vision of CIHR, it should endeavour to broaden its reach to include more research in Themes 3 and 4.
Recommendation 9:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:
| Recommendation | Response | Action Plan |
| 1. IAB Functioning - While the IAB is generally considered to be running well, the Institute should endeavour to review and (if necessary) clarify the roles and responsibilities of the Chair and members (including non-scientists), including responsibilities related to knowledge translation. It should also ensure that Board members are provided with focused communications on all topics in advance of meetings so all members can be fully engaged. | Agree |
|
| 2. Planning Mechanisms - ICRH should consider periodic refinements to its priorities, based on its previous investments and successes, to ensure that the priority areas of most importance to its communities receive attention. | Agree |
|
| 3. Consultation and Communication - The Institute should maintain its communications activities, using a variety of approaches/vehicles, and if feasible consider some improvements suggested by key informants in this evaluation - for example, devote more resources to communications, develop a formal communications action plan, and send targeted communications to particular sub-groups such as policy makers, private industry and the general public. Efforts should be made to engage/consult with any stakeholders/sectors that have been less engaged to date (e.g., provincial health policy makers, non-profit sector, researchers in critical and intensive care). | Agree |
|
| 4. Collaboration and Partnerships - While overall encouraged to sustain its work in collaborating and developing partnerships with a range of organizations and sectors, the Institute should, to the extent feasible, also explore opportunities to develop partnerships on common international and global health issues in its research domain and to take the role of convener or facilitator of large-scale, cross-cutting initiatives (e.g., in public health). | Agree |
|
| 5. Knowledge Translation - The ICRH should review its efforts in support of KT. The efforts expended by the ICRH in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the ICRH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT | Agree |
|
| 6. Ethics - The Institute is encouraged to maintain and further develop its efforts related to CIHR's ethics mandate, and that it inform its researcher and stakeholder community about these efforts and any achievements. | Agree |
|
| 7. Contribution to Transformative Vision - While ICRH has been making progress in terms of the implementation of the transformative vision of CIHR, it should endeavour to broaden its reach to include more research in Themes 3 and 4. | Agree |
|
| 8. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree |
|
1 Note that the survey of researchers was a cross-Institute surv ey conducted by EKOS Research Associates.
2 A Request for Applications (RFA) is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.
3 STIHR (the Strategic Training Initiative in Health Research) consists of innovative and interdisciplinary training programs that support the development of new researchers. Each program can receive up to $310,500 a year for up to 6 years and at least 71% of these funds must be used for trainee stipends or travel between training locations.