Mid-Term Evaluation of the Institute of Aboriginal Peoples’ Health (IAPH)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Aboriginal Peoples' Health (IAPH)
1.2.1 Vision
1.2.2 Mission
1.2.3 Values
1.2.4 Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Aboriginal Peoples' Health (IAPH)
The IAPH will strive to improve the health of First Nations, Inuit and Métis people by supporting innovative research programs based on scientific excellence and Aboriginal community collaboration.
The IAPH will play a lead role in building research capacity in the First Nations, Inuit and Métis communities, and will support partnerships and alliances between Aboriginal communities and non-Aboriginal health research organizations/institutes at the local, regional, national and international levels. The IAPH will support health research that respects Aboriginal cultures, while generating new knowledge to improve the health and well-being of Aboriginal people.
The IAPH is guided by the following set of core values:
- Present health research results to Aboriginal people in a way that is accessible, appropriate and easily understood.
- Maintain ethical standards by adhering consistently to prescribed values and principles.
- Act in an honest, fair and just manner.
- Aim to include Aboriginal people in all health research activities.
- Share new knowledge with all research partners.
- Conduct all activities and business in a transparent manner.
The IAPH is guided by four research priorities and all initiatives undertaken by the Institute support one or more of the following research priorities:
- Forge partnerships and share knowledge;
- Respect Aboriginal cultures;
- Build capacity; and
- Resolve critical health issues.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
1.3.1 MethodologyThe evaluation consisted of four main lines of evidence:
- A review of documents and administrative data relating to the IAPH;
- 41 key informant interviews with IAPH staff and IAB members, researchers and students, stakeholders and partners,
- Case studies of two IAPH initiatives; and
- A telephone survey of 24 funded and 9 non-funded researchers affiliated with the IAPH. 1
2. Evaluation Results
2.1 Relevance
There is clear agreement across interviewees and survey respondents that the mandate and priorities of IAPH are appropriate. The need for a clearer ethics statement expressed by a few interviewees reflects the importance of the relationship between Aboriginal communities and researchers and the need to ensure that the communities are treated in a fair, open and ethical manner. The need for clear ethics guidelines is not unrelated to the history of abuse on the part of researchers conducting research in Aboriginal communities.
The Institute is seen by interviewees as the most appropriate mechanism for making a difference with respect to developing Canadian research capacity, research excellence and knowledge translation in the area of Aboriginal health research. The existence of an Institute focused on Aboriginal health issues is seen as a signal that the government of Canada sees Aboriginal health as an important issue.
The virtual model is seen by interviewees as suitable due to its flexibility and sustainability over the long term. Some challenges with respect to communication were identified as a result of the geographic dispersion of Institute staff and IAB members. However, these challenges would likely exist even if the Institute was not virtual since little can be done about the geographic dispersion of IAB members while still maintaining the quality and diversity of IAB membership.
2.2 Effectiveness
Based on interviews, survey results, the IAPH Performance Report, and case studies conducted for this evaluation, it can be concluded that IAPH has been effective in contributing to CIHR objectives. However, the contribution of IAPH to CIHR objectives is expected to increase as the Institute develops more capacity within the Aboriginal health research community and within Aboriginal communities themselves.
The Institute, based on input from interviewees and the Survey of Funded Researchers, has made a contribution to the creation of new knowledge. However that contribution has been limited to date. The Institute, unlike those linked to more established research communities, started from a situation where very little research in the area of Aboriginal health was being conducted, and so there was little capacity and infrastructure. The Institute is expected to contribute an increasing amount of new knowledge as research capacity and infrastructure are put in place.
Knowledge translation (KT) has, based on information collected, been a priority for IAPH. A significant number of KT activities have been undertaken. The majority of those interviewed feel IAPH has been successful in its knowledge translation efforts thus far, however some noted a strong need for more direction from CIHR central on what is meant by knowledge translation as well as the expectations for Institutes with respect to knowledge translation. Because of its strong focus on community participation in research, IAPH has a two-way view of knowledge translation, unlike other Institutes. Also, because of this link to communities, IAPH and researchers conducting research, have an obligation to conduct knowledge translation with the communities through the knowledge translation component of grants. All research projects funded by IAPH are required to include a knowledge translation element focused on Aboriginal communities.
Even before IAPH was implemented, it was recognized that a significant amount of capacity building would need to take place in order for IAPH to be successful. The IAPH IAB recognized the need to build capacity early on and has made capacity building a priority. The principal mechanism through which capacity building is being undertaken by IAPH is through the eight Aboriginal Capacity Development Research Environments (ACADRE) Centres spread across Canada. The ACADRE Centres serve as an important link between Aboriginal communities, researchers and the IAPH itself. The ACADRE Centres are not only intended to build capacity within the academic research community but also within Aboriginal communities themselves who are seen as researchers as well. Based on data collected through interviews, survey results, case studies as well as other sources, it is reasonable to conclude that IAPH has been successful in increasing capacity within the academic research community. However, the evidence suggests that IAPH has been less successful in addressing capacity issues within Aboriginal communities.
Besides a need for increasing human capacity within the Aboriginal health research community, IAPH was, at implementation, faced with a need for improving the structural capacity. The most important issue within the Aboriginal health research community upon implementation of IAPH was the lack of clear ethical guidelines for conducting research in Aboriginal communities. This was identified as a pressing need by researchers and Aboriginal communities. In response to this, IAPH has actively participated in the Tri-Council ethics reviews. In addition, when the research community and Aboriginal communities expressed concerns over the length of time the Tri-Council review was taking, IAPH initiated its own process to develop guidelines. Both ethics processes are nearing completion and will soon be ready for consultation with Aboriginal communities.
The transformative vision of CIHR involves an integrated, multidisciplinary, strategic research organization intended to improve the health of Canadians. There is strong evidence to suggest that IAPH has made a contribution to the transformative vision of CIHR. IAPH has resulted in the funding of projects related to Aboriginal health that, because of their community focus, would likely not have been funded under the former Medical Research Council (MRC) model. In addition, the activities of the ACADRE Centres have resulted in a change, or transformation, in the relationship between researchers and communities. The IAPH has been a key Institute with respect to expanding the research environment beyond the traditional biomedical and clinical focus of the MRC.
As noted above, the need for clear, relevant ethics guidelines that ensure research is done in a respectful manner was identified as crucial by researchers and Aboriginal communities early after the implementation of IAPH. In direct response to this clearly identified need IAPH undertook activities to develop ethics guidelines. These guidelines are nearing completion and will be presented to Aboriginal communities and organizations for feedback in the coming months. The vast majority of those interviewed feel that IAPH has made a significant contribution to the CIHR ethics mandate through its participation in the Tri-Council ethics process as well as its own internal process undertaken to develop ethics guidelines.
Evidence from interviews and survey results suggest that IAPH has performed well with respect to achieving its vision, mission and research priorities. This is not to say that IAPH has met or fulfilled the vision, mission or mandate, but the evidence indicates that IAPH is moving in the right direction. There is currently little data beyond qualitative and anecdotal evidence to assess whether IAPH has made progress. However, IAPH has developed a logic model and a related evaluation framework which are expected to facilitate the evaluation of progress towards the vision, mission and research priorities in the future.
IAPH has developed a number of linkages, exchanges, partnerships and alliances, both nationally and internationally, with a broad range of organizations including federal departments, Aboriginal organizations, non-government organizations and other CIHR Institutes. IAPH management sees partnerships as important because of the potential for pooling resources, leveraging funding and increasing the credibility of IAPH nationally and internationally. An area where IAPH could improve performance is with partnering and coordinating with other Aboriginal health organizations such as National Aboriginal Health Organization (NAHO) and the Aboriginal Healing Foundation. Although IAPH has developed a working relationship with both these organizations, interviewees felt that a better integration between IAPH and such organizations was needed.
Prior to the implementation of the IAPH, Aboriginal communities were generally reticent of becoming involved in research because of a past history of negative experiences in many communities. Since the implementation of IAPH, interviewees report that Aboriginal communities are much more willing to participate in research. There is agreement among interviewees, that without the leadership of IAPH, much of the progress made with respect to Aboriginal health research would not have occurred.
In addition to directly influencing the research environment, IAPH has also influenced research agendas. IAPH has sought to identify and address, mainly through issuing RFAs, the key health issues faced by Aboriginal people in Canada.
2.3 Delivery
The approach to planning taken by IAPH is largely focused on the IAB. This is seen as appropriate and effective in that the IAB members are carefully chosen to reflect the research and stakeholder communities and so their views are seen as reflective of these communities. Planning mechanisms are ongoing as well as activity/output-specific. Examples of planning mechanisms that are output-specific are the approach used to develop the strategic plan and IAPH Evaluation Framework. Ongoing activities include IAB meetings and teleconferences as well as workshops and conferences that are supported by IAPH. There is general satisfaction with the planning mechanisms used by IAPH although stakeholders interviewed were often not familiar enough with IAPH planning mechanisms to comment. This is not entirely surprising since stakeholders represented organizations that had partnered with IAPH but were not in a position to provide input into IAPH planning.
An important challenge with respect to planning noted by interviewees was the tension between the expectations and needs of Aboriginal communities and the university-focussed research environment within which IAPH must function. The IAPH must balance the need for scientifically sound research practices with the need to ensure that the research conducted reflects the needs and perspectives of Aboriginal communities. This has implications for planning in that IAPH must ensure that Aboriginal community perspectives are reflected in all aspects of IAPH.
The main mechanism through which the IAPH obtains input and feedback from stakeholders and researchers is through the Scientific Director and IAB members who interact with stakeholder and researcher communities on a frequent basis both formally and informally. The eight ACADRE Centres serve as a direct link between IAPH and the Aboriginal communities served by the regional ACADRE Centres. The IAPH is seen to be successful in obtaining feedback and input from its researcher and stakeholder communities and responding to key issues identified.
Based on interview results, interviewees are generally satisfied with the communication strategy of IAPH. The Institute communicates with researchers and stakeholders through a variety of channels. Using ACADRE Centres as the main mechanism through which to communicate with Aboriginal communities is likely the most appropriate mechanism given the geographic dispersion of Aboriginal communities. ACADREs are, because of their geographic proximity to Aboriginal communities they serve, better able to establish a working relationship thus facilitating ongoing communication.
The relationship between IAPH and CIHR central is generally seen as positive. Two specific areas where challenges were identified include knowledge translation and Aboriginal representation on CIHR Governing Council. IAPH would like to see Aboriginal representation on CIHR Governing Council. This would serve to increase the credibility of CIHR within the Aboriginal community as well as ensuring that Aboriginal perspectives are taken into consideration at the CIHR corporate level. There appears to be a need for clearer direction from CIHR corporate with respect to knowledge translation, specifically what is meant by knowledge translation (definition) and the expectations of CIHR corporate with respect to Institute knowledge translation activities. We note that a definition of knowledge translation has been posted on the CIHR Web site for some time, and recently a Knowledge Translation strategy was approved by Governing Council and posted on the Web site.
Funding is generally seen as inadequate, however there is some evidence that the current funding available to IAPH is sufficient given the current capacity in the Aboriginal health research community. Overall, a slight majority of researchers and staff/IAB members interviewed feel the current balance between funding for investigator-initiated research and strategic research is fine. A slight majority of stakeholders feel that the proportion of funding to strategic research should be increased. This reflects the perspectives of stakeholders, most of whom are affiliated with policy organizations and various levels of government. Stakeholders see the link between research and policy in a much more direct manner.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of IAPH. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- maintain its efforts in the areas of research excellence, capacity development, and funding strategic priorities in collaboration with other national Aboriginal organizations;
- maintain its planning mechanisms;
- sustain its current approach to consultation which incorporates input from Aboriginal communities, stakeholders, students and researchers;
- maintain ACADREs for communicating and relationship-building with Aboriginal communities;
- maintain its efforts at knowledge creation while continuing to recognize that there remains a need to build capacity within the research community and Aboriginal communities which will provide a foundation for increased knowledge creation in the future; and
- sustain its efforts in contributing to the transformative vision of CIHR through its emphasis on interdisciplinary research in areas of strategic importance and knowledge translation for the ultimate benefit of the health of Canadians.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
IAB Functioning - While encouraged to maintain the current structure and operation of its IAB, the Scientific Director (SD) should clarify the roles and responsibilities of the IAB in relation to those of the office of the SD. The advisory nature of the board needs to be clearly communicated.
Recommendation 3:
Collaboration and Partnerships - The Institute should endeavour to increase collaboration and partnering with National Aboriginal organizations, particularly where there are common goals and objectives especially to build research capacity within Aboriginal communities so they may more fully and easily participate in health research projects.
Recommendation 4:
Consultation and Communications - The Institute should - with the support of CIHR corporate - assess mechanisms for raising its profile within its community and the general public by drawing attention to its activities and results.
Recommendation 5:
Knowledge Translation - The IAPH should review its efforts in support of KT and work to improve knowledge translation at the level of Aboriginal communities. The efforts expended by the IAPH in the area of KT do not always appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IAPH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 6:
Ethics - While encouraged to persist with its already significant efforts in the area of ethics, given the important need for ethics guidelines in Aboriginal health research, the Institute should endeavour to have the newly developed guidelines implemented as quickly as possible.
Recommendation 7:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:
| Recommendations of IAPH evaluation | IAPH Response | Action Plan |
| 1. IAB Functioning - While encouraged to maintain the current structure and operation of its IAB, the Scientific Director (SD) should clarify the roles and responsibilities of the IAB in relation to those of the office of the SD. The advisory nature of the board needs to be clearly communicated. | Agree |
|
| 2. Collaboration and Partnerships - The Institute should endeavour to increase collaboration and partnering with National Aboriginal organizations, particularly where there are common goals and objectives especially to build research capacity within Aboriginal communities so they may more fully and easily participate in health research projects. | Agree |
|
| 3. Consultation and Communications - The Institute should - with the support of CIHR corporate - assess mechanisms for raising its profile within its community and the general public by drawing attention to its activities and results. | Agree |
|
| 4. Knowledge Translation - The IAPH should review its efforts in support of KT and work to improve knowledge translation at the level of Aboriginal communities. The efforts expended by the IAPH in the area of KT do not always appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IAPH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. | Agree |
|
| 5. Ethics - While encouraged to persist with its already significant efforts in the area of ethics, given the important need for ethics guidelines in Aboriginal health research, the Institute should endeavour to have the newly developed guidelines implemented as quickly as possible. | Agree |
|
| 6. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree |
|
1 Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.