Mid-Term Evaluation of the Institute of Cancer Research (ICR)
Executive Summary
December 2005
1. Executive Summary
1.1 Overview of CIHR
1.2 Institute of Cancer Research (ICR)
1.2.1 Mandate
1.2.2 Strategic Orientations
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Institute of Cancer Research (ICR)
The Canadian cancer research community is well-established, successful, and has been relatively well funded. Historically, the Medical Research Council (MRC), the National Cancer Institute of Canada (NCIC), cancer specific voluntary organizations and both federal and provincial government agencies, funded the cancer research community. With the CIHR building upon the progress of the MRC, in 2000-2001 the ICR became a new partner in the cancer research community. Building on the investigator-based research funded by CIHR, the ICR focuses on supporting strategic research programs in areas of identified research need. ICR financial documentation1 reported that the ICR had an annual budget of $7,196,000 in 2004-05 for funding strategic initiatives, and receives an Institute Support Grant of $1M per year from CIHR.
As stated in ICR's Draft Performance Report 2000 2004, the mandate of ICR is: "The CIHR Institute of Cancer Research will support research to reduce the burden of cancer on individuals and families through prevention strategies, screening, diagnosis, effective treatment, psycho-social support systems, and palliation."
The ICR has developed six strategic orientations to carry out its mission and to guide its activities. Under each specific strategic orientation, the ICR has defined three more detailed objectives. The strategic orientations are to:
- Play a leadership role in setting the Canadian strategic cancer research agenda;
- Support and develop cancer research initiatives, programmes and projects that respond to that agenda;
- Develop and support capacity building in cancer research;
- Lead in co-ordinating funding of cancer research by all national, provincial and local organisations in Canada;
- Seek collaboration/partnership with other cancer agencies and stakeholders; and
- Stimulate research required to facilitate dissemination, transfer and translation of knowledge from research findings into policies, interventions, services and products.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to the ICR;
- forty-four key informant interviews with ICR staff and IAB members, researchers and students, stakeholders and partners, as well as a focus group discussion with IAB members (held at an IAB meeting);
- case studies of two ICR initiatives; and
- a telephone survey of 172 funded and 57 non-funded researchers affiliated with the ICR.2
2. Evaluation Results
The majority of researchers and stakeholders were aware of and familiar with the mandate of the Institute, and overall judged it as appropriate. The need for an organization like ICR that focuses on strategic priorities in cancer research was identified. The Institute undertook a national consultation (Delphi process) to identity research priorities for the Institute, which collected input from a variety of different groups within the community (e.g., researchers, stakeholders, government). The evaluation found that the research community generally supported the research priorities resulting from the consultation process. A few researchers consulted for the evaluation indicated that they were surprised by the strategic priorities chosen (in particular palliative and end-of-life care), but overall, they made sense to most researchers.
Overall, the ICR is perceived as needed by the community to support the funding of strategic priorities and to develop research excellence. While various alternatives to ICR were suggested, most felt it was too early to start to consider alternatives at this point. The most frequently suggested improvements to the model at this point were to provide more autonomy and additional funding to ICR so that it can be more effective in fulfilling its mandate.
There was support for the CIHR virtual Institute model. Most of those consulted for the evaluation indicated that the virtual Institute model was appropriate for the Canadian situation. The virtual model was viewed as a good fit given Canada's geographic dispersion, and the level of resources allocated to health research.
The evaluation found a positive relationship between level of involvement in the Institute and the perception of effectiveness of the Institute's activities. This was confirmed in interviews with the ICR staff who explained that the strategic focus of the ICR had been at the organizational level for the earlier years of the Institute as it worked with the cancer community partners to develop a national research agenda. Those most closely involved with the ICR would be aware of this strategic choice of level of activities, and thus more likely to take a more encompassing view of the Institute compared with those who have been less directly involved with the Institute, and thus less aware of this choice.
Overall, it was viewed that ICR has made a contribution to CIHR's expected results particular in creating knowledge and building communities. Overall, there was agreement across respondent categories that, although ICR has made a contribution to the creation of new knowledge in the area of cancer research, the research results will only be realized in coming years.
Areas where less success has been perceived are in knowledge translation and contributing to the ethics mandate of CIHR. Respondents reported that ICR has contributed to the development of knowledge translation (KT) activities, however KT progress has not been as evident as other ICR priorities. It should be noted that other Canadian cancer organizations have KT as their top priority.
ICR's achievement with palliative and end-of-life care was reported as exceptional. The most frequently cited impact was in capacity building with activities that have had impacts on what was a small, fragmented research community.
IAB/staff, stakeholders, and researchers interviewed reported mixed views regarding the strategic and operational planning mechanisms used by ICR. There were some "surprises" that resulted from the planning exercise to identify strategic priorities for the Institute, however, once identified, they appeared to be readily accepted by most members of the community.
Respondent groups expressed mixed opinions regarding consultation and communication. Communications and consultations with key stakeholders, particularly at the organizational level, appear to have been very effective and appropriate. Consultations and communications with individual researchers were deemed to be less effective. Given the strategic focus of the ICR on the organizational level for its initial developmental period, this mixed result is to be expected.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of ICR. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities;
- maintain the current structure and operation of the IAB;
- maintain current planning mechanisms, as they are effective in identifying priorities;
- sustain its effective work undertaken to date in the area of collaboration and partnerships;
- sustain its current capacity building activities;
- sustain its work in the area of developing foundations for knowledge creation; and
- maintain its efforts in contributing to the transformative vision of CIHR through its IAB representation, its RFA requirements, and its support to strategic initiatives favouring the integration of researchers from non-biomedical and clinical research areas.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Consultation and Communication As the Institute moves into the next phase, it should place more emphasis on communications with individual researchers. The communications and consultations have been very effective with the stakeholder communities given the priorities of the Institute.
Recommendation 3:
Knowledge Translation The ICR should review its efforts in support of KT. The efforts expended by the ICR in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the ICR are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 4:
Ethics The ICR should review its efforts in support of ethics. The efforts expended by the ICR in the area of ethics do not appear to have a large profile among IAB members, researchers and stakeholders. The area could be reviewed to determine to what extent the efforts are likely to result in the impacts that the ICR are hoping to achieve.
Recommendation 5:
Performance Monitoring and Reporting In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired, comment on Recommendation 1, that suggests continuing certain activities that are going well:
| Recommendation | Response | Action Plan |
| 2. Consultation and Communication - As the Institute moves into the next phase, it should place more emphasis on communications with individual researchers. The communications and consultations have been very effective with the stakeholder communities given the priorities of the Institute. | Agree |
|
| 3. Knowledge Translation - The ICR should review its efforts in support of KT. The efforts expended by the ICR in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the ICR are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. | Agree |
|
| 4. Ethics - The ICR should review its efforts in support of ethics. The efforts expended by the ICR in the area of ethics do not appear to have a large profile among IAB members, researchers and stakeholders. The area could be reviewed to determine to what extent the efforts are likely to result in the impacts that the ICR are hoping to achieve. | Agree | Appoint new IAB Ethics designate
|
| 5. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree |
|
1Microsoft Excel file (2005) "ICR Annual Report 2004-2005 – ISI Table.xls"
2Note that the survey of researchers was a cross Institute survey conducted by EKOS Research Associates.