Executive Summary of the Integrated Results of the Institute Mid-Term Evaluations

Other format


PDF version
(191 KB)

July 2008

Table of contents

Executive Summary

Management Response to Recommendations

Executive Summary

Overview of CIHR

The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications.

CIHR was created under the CIHR Act that came into force on June 7, 2000.

Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.

In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:

  • outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
  • outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
  • transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.

These strategic outcomes will be enabled through:

  • effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
  • organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.

CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country. CIHR emphasizes multidisciplinary approaches to addressing health problems.

The 13 Institutes are as follows:

  • Institute of Aboriginal Peoples' Health (IAPH)
  • Institute of Aging (IA)
  • Institute of Cancer Research (ICR)
  • Institute of Circulatory and Respiratory Health (ICRH)
  • Institute of Gender and Health (IGH)
  • Institute of Genetics (IG)
  • Institute of Health Services and Policy Research (IHSPR)
  • Institute of Human Development, Child and Youth Health (IHDCYH)
  • Institute of Infection and Immunity (III)
  • Institute of Musculoskeletal Health and Arthritis (IMHA)
  • Institute of Neurosciences, Mental Health and Addiction (INMHA)
  • Institute of Nutrition, Metabolism and Diabetes (INMD)
  • Institute of Population and Public Health (IPPH)

Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field, and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.

CIHR's research funding for 2004-05 was $619 million (up from $275 million in 1999-2000.) Total expenditures including administration were $666 million in 2004-05, compared to $289 million in 1999-2000. In 2004-05, $84 million was evenly divided and allocated collectively to Institutes to fund strategic research and $13 million was provided in support for the operations of the Institutes. Funds for strategic research within CIHR> (including strategic research funded by the Institutes and by CIHR) represent about 30 per cent of overall research funds (the remainder is allocated through the CIHR open competitions).

Evaluation Objectives and Issues

The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct an evaluation of its activities, outputs and outcomes at the end of the first five years of operation. The goals of the evaluation were the following:

  • to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
  • to provide input into the Five Year (Quinquennial) Review of Institutes required by the CIHR Act.

The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.

Methodology

The evaluation consisted of four main lines of evidence:

  • a review of documents and administrative data relating to the 13 Institutes and CIHR overall;
  • 565 key informant interviews with CIHR Vice Presidents, Governing Council members, Institute staff and IAB members, researchers and students, stakeholders and partners associated with the 13 Institutes, as well as one stakeholder and five IAB focus groups;
  • case studies of two to four initiatives for each Institute; and
  • a telephone survey of 1,676 funded and 568 non-funded researchers affiliated with the CIHR Institutes.1

Relevance and Alternatives

It is clear from this overview of the 13 Institute evaluations that CIHR as a whole and each of the 13 Institutes remain a relevant and needed component of support to health research in Canada. Their mandates and strategic priorities are regarded as appropriate and relevant, validating both the initial assignment of mandates to Institutes and the strategic directions marshalled by Institutes within their mandates. It should be noted that all Institute mandates are defined more broadly than the strategic priorities on which they have chosen to focus. Views were also expressed for most Institutes that their mandates are very broad, given their available resources. A few concerns were noted within Institutes that have given priority focus to some domains and may now need to turn greater attention to others. The Institutes are broadly acknowledged by key informants as the most appropriate mechanism to make a difference in their particular research domains. Across all 13 evaluations, no alternatives were consistently identified as potentially more effective. Overall, the benefits of the Institute delivery model are believed to outweigh the disadvantages.

Effectiveness

Overall, the evaluations showed that the Institutes have performed well, making substantial progress in fulfilling their mandates and strategic priorities.

While it is generally considered too soon to be thoroughly assessing the outcomes of health research generated through the Institutes, from the activities and preliminary research related results to date, the greatest effectiveness was noted in the areas of creating new knowledge, developing health research capacity, and developing partnerships and collaborations through a broad range of innovative initiatives. Through their funding of strategic initiatives in research and capacity building, Institutes have come to exercise significant leadership in the research agendas in their mandated research areas, and to leverage health research capacity through partnerships in multiple sectors. Capacity building, through expanded concepts and tools, is recognized as a key strength of all Institutes.

Although a priority, knowledge translation has been a significant challenge for the majority of Institutes, with the exception of those that had a higher pre-existing capacity (primarily in themes 3 and 4 where KT knowledge and infrastructure was already in place). Similarly, the evaluations consistently found that there is a lack of clarity as to Institutes' roles and responsibilities in meeting CIHR's ethics mandate.

The Institutes have succeeded in contributing to the achievement of CIHR's transformative vision for health research in Canada, particularly through fostering enhanced interdisciplinarity. This includes supporting the development of research and training that integrates biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health, and expanding research areas of strategic importance. The Institutes are contributing to CIHR's overall objective of creating new knowledge both directly, through the support of strategic research, and indirectly, through the creation of foundations for future knowledge generation.

Delivery

All 13 Institutes have been successful in establishing effective organizations and program delivery. The evaluation results showed that all 13 Institutes have used strategic and operational planning mechanisms effectively, and that the consultative processes used in these mechanisms have strengthened linkages and engagement of stakeholder communities. Institutes are uniformly seen as having been able to identify and respond to emerging issues. There is a generally high level of satisfaction with Institute Advisory Board functioning. While all 13 Institutes have made considerable efforts in terms of consultation and communication, the evaluations also suggested some need to address gaps and issues in internal communications between Institutes and CIHR corporate, and external communications between Institutes and their communities. While significant improvements have been made in terms of internal communications, problems still exist in terms of coordination of communication activities and in timeliness, while better external communication is needed to ensure respective communities are aware of the activities and aims of the Institutes.

With respect to the role of central CIHR functions, the evaluations found a range of experiences and identified needs for improvements in several areas.

A consistent finding from the stakeholder interviews across all evaluations was that all 13 Institutes are not sufficiently resourced in light of their mandates and that their mandates are too broad to be achieved with current resource levels. There is also a question in the mind of the Evaluation Steering Committee as to whether the mandate of any Institute can ever be fully 'achieved.'

The evaluation also explored views on the funding for open versus strategic investments. While there was support among many stakeholders for revised funding formulae, there was no consensus on the appropriateness of the 70:30 split between investigator-initiated and strategic research.

Recommendations

The following recommendations are made by the Evaluation and Analysis Unit and are based on the evidence presented within this report and the 13 Institute Evaluations. The recommendations were reviewed and accepted by the Evaluation Steering Committee to ensure they were reasonable on the basis of the evidence.

Institute Funding

Given the consistent finding from stakeholder interviews across all Institute evaluations that Institutes were not sufficiently resourced to achieve their mandates and may be trying to accomplish too much with too little by way of financial resources, we recommend that:

1.0 CIHR assess the current funding formula for Institutes, including both the Institute Support Grant and research funding to help ensure that the Institutes' mandate and strategic priorities are achievable.

2.0 Absent increased funding, CIHR should consider focusing on fewer, key priorities and activities that can be accomplished within available resources.

Themes

The evaluations revealed that there is a common understanding and acceptance of the role of each Institute in supporting the CIHR research themes (biomedical; clinical; health systems and services; and the health of populations, societal and cultural dimensions of health and environmental influences on health).

While there is acceptance of the principle, there is no guidance or clear direction regarding the Institutes' roles and responsibilities in funding and/or contributing to all four research themes. We recommend that:

3.0 CIHR should clarify whether it is necessary that each Institute fund projects in each of the four theme areas, or whether it is appropriate that Institutes as a group fund projects covering each of the four theme areas, and/or if there are other means of "including" all themes in Institute activities (including inter-theme cooperation with other Institutes, giving voice to all thematic areas in various discussions and fora, etc.).

Once Institute roles and responsibilities are clarified, then, working with the Institutes, clear expectations and measures should be set that address the funding levels and/or other Institute contributions to the four CIHR themes. Progress against these targets should be carefully monitored.

Knowledge Translation

A consistent finding from key informant interviews across all Institute evaluations was that the Institutes were generally not clear as to their roles and responsibilities in meeting CIHR's Knowledge Translation (KT) mandate. We recommend that:

4.0 CIHR should clearly define and articulate the roles and responsibilities of all parties responsible for the implementation of the CIHR KT Strategy. In addition, these roles and responsibilities need to be clearly communicated throughout CIHR and externally to all stakeholders.

5.0 CIHR should establish relevant performance measures and targets to monitor the implementation of CIHR's knowledge translation mandate.

Ethics

There is a wide variance across Institutes with respect to their performance on the delivery of the CIHR ethics mandate, with some Institutes being very active and others doing very little. A consistent finding from stakeholder interviews across all Institute evaluations was that the Institutes were not clear as to their roles and responsibilities in meeting CIHR's ethics mandate. We recommend that:

6.0 CIHR should define and articulate the Institutes' roles and responsibilities to address the overall CIHR ethics mandate. It is also recommended that relevant performance measures and targets be established to monitor the implementation of CIHR's ethics mandate, particularly within the Institutes.

Institute Delivery Model

The evaluations of the Institutes revealed the advantages of the Institute delivery model have so far outweighed the disadvantages in the perspective of Institutes and their stakeholders. The disadvantages that were identified include issues around Scientific Director (SD) transitions and were found to be particularly significant. It is recommended that:

7.0 CIHR fully analyze practical and effective means of achieving the aims of the Institute delivery model while minimizing its disadvantages and managing associated risks and issues. This should include:

  • further assessment of the strengths and weaknesses of the Institute delivery model;
  • effective management of Institute transitions; key elements that need focus are:
    • loss of momentum as an Institute transitions from one SD to another;
    • clarifying the obligations to Institute staff that are employed through Institute operating grants and therefore not entitled to the same job security as CIHR corporate employees.

Communications

With respect to performance on communications, the Institute evaluations revealed inconsistent results across the 13 Institutes, both within CIHR and from CIHR to its external partners and stakeholders. We recommend that:

8.0 Communications Branch should clarify with the Institutes their communications expectations and ensure that Institute-specific information is appropriately shared internally. We also recommend that the Director of Communications continue to work with Institutes to ensure that the key results and achievements of all of our research, as well as Institute priority areas, are communicated clearly to external audiences.

Institute Advisory Boards

While Institute Advisory Boards are generally found to be operating effectively, concern has been raised by senior management and Governing Council members that the potential exists for IABs to operate as governance mechanisms, rather than in the proper role as advisory bodies. We recommend that:

9.0 CIHR should work to ensure that all current and future IAB members are aware of the 'advisory' nature of their role and that the accountability relationships between CIHR> and the Institute Scientific Director be clearly articulated.

The following recommendation results from the evaluation process, as opposed to the findings and conclusions of the evaluations. It was not an issue explored through the evaluations.

Data Quality and Management

There is variation between the data tracked and reported centrally at CIHR and the data tracked and reported at some Institutes. Some examples of inconsistencies include the number of partners for strategic initiatives, the numbers of projects supported, whether the funding is from open or strategic budgets etc... There are also questions of data quality due to researchers' self-selections during the application process and using an inadequate classification system for research overall. The result is that there is variation in the way in which data are collected and presented across Institutes and at CIHR centrally. These inconsistencies negatively impact the overall ability to measure and report on performance in a consistent, accurate and credible manner. We recommend that:

10.0 CIHR should conduct a review of data management policies and practices across CIHR and document data quality issues. The purpose of the review should be to enable improved accuracy and consistency through creation and implementation of a strategy that clearly defines roles, responsibilities and accountabilities to ensure that data are tracked and reported accurately, consistently and reliably in the future.

Management Response to Recommendations

For each recommendation made above, Management has provided feedback on progress made. In the following tables, each Management response is provided with an action plan, a timeline, a contact responsible for coordinating the action and, if there was consultation, who was consulted.

Some of the recommendations have been fully addressed while some others will require a follow-up at a later date.

Responsible for Coordination of Management Response: Kelly VanKoughnet Acting VP Research (Pierre Chartrand, VP Research)

Recommendation - Institute Funding

Given the consistent finding from stakeholder interviews across all Institute evaluations that Institutes were not sufficiently resourced to achieve their mandates and may be trying to accomplish too much with too little by way of financial resources, we recommend that:

1.0 CIHR assess the current funding formula for Institutes, including both the Institute Support Grant and research funding to help ensure that the Institutes' strategic priorities are achievable

Agree - CIHR must develop processes to support our strategic investment planning. Such processes must recognize that the full budget (open and targeted components) of CIHR is intended to support the Institutes in advancing their mandate and strategic priorities.

Action Plan:

  • Create RKTC and specifically the sub-committee on Planning and Partnership responsible for providing recommendations to RKTC on CIHR's research and knowledge translation strategy and the production of a rolling multi-year investment plan in order to deliver on the strategy.
  • Review Institute Support Grant Program in association with renewal of related TBS review of Terms and Conditions.

Timelines for completion are:

  • RKTC - inaugural meeting February, 2007
  • Subcommittee - inaugural meeting April, 2007
  • Rolling multi-year budget in place to support future planning Fall, 2006
  • Review and modifications (as required) to the ISG Program - April, 2008.

Responsibility:

  • President, supported by VP Research
  • SD chair, CFO
  • SPP (subcommittee)
  • CFO, supported by Director, Institute Affairs

Consultation:
Extensive consultation at the executive and staff levels occurred in the creation of RKTC and its subcommittees. SPP in now responsible for continuing consultation in relation to our investment strategy and expanding that consultation to include partners and other stakeholders where appropriate.

2.0 Absent increased funding, CIHR should consider focusing on fewer, key priorities and activities that can be accomplished within available resources.

Agree, but this is a particular challenge given the complexity of CIHR's structure with 16 different cost centres having a budget they need to manage and spend strategically.

Action Plan:

  • Create a forum and decision-making mechanism to further refine priorities and focus investments for multi-institute/cost centre initiatives where potential for impact is greatest.
  • Define a timeline/planning structure to support strategic investment planning
  • Develop methodology/criteria to support decision among proposals for strategic investment
  • Develop a strategy to reduce the complexity of CIHR programming

Timelines for completion are:

  • RKTC and SPP created (Feb-April, 2007)
  • Planning framework - December, 2007
  • SPP Pilot methodology/criteria for decision-making - Feb, 2008
  • RKTC long term strategy to focus CIHR resources on high value, high impact strategic research initiatives. - Retreat April 2008
  • TIPs Project - Towards a Better, Simpler CIHR: Streamlined the portfolio of existing funding programs into a core set of approximately 20-25 funding tools comprising the CIHR Toolbox October 2007. Refinements to this toolbox continue. (see actions under #3)
  • Implementation of Funding Opportunities Database (June 2007): Standardizes and simplifies presentation of funding opportunities on the CIHR web site and offers searching and browsing capabilities, facilitates ongoing roll-out of e-submission and e-review, includes a 'watch' feature (to notify users of changes to opportunities and upcoming deadlines).

Responsibility:

  • President, supported by VP Research
  • SPP (SD Chair, CFO)
  • SPP (SD Chair, CFO)
  • RKTC
  • VP Research
  • VP Research

Consultation:
SPP Chair, Vice-chair and members consulted in defining the elements of a work-plan for SPP.

In addition, Governing Council has provided clear guidance to management to implement a forward strategy to allow CIHR to focus resources on high value, high impact initiatives while continuing to reduce complexity of programming wherever possible.

Recommendation - Themes

The evaluations revealed that there is a common understanding and acceptance of the role of each Institute in supporting the CIHR research themes (biomedical; clinical; health systems and services; and the health of populations, societal and cultural dimensions of health and environmental influences on health).

While there is acceptance of the principle, there is no guidance or clear direction regarding the Institutes' roles and responsibilities in funding and/or contributing to all four research themes. We recommend that:

3.0 CIHR should clarify whether it is necessary that each Institute fund projects in each of the four theme areas, or whether it is appropriate that Institutes as a group fund projects covering each of the four theme areas, and/or if there are other means of including all themes in Institute activities (including inter-theme cooperation with other Institutes, giving voice to all thematic areas in various discussions and fora, etc.).

Once Institute roles and responsibilities are clarified, then, working with the Institutes, clear expectations and measures should be set that address the funding levels and/or other Institute contributions to the four CIHR themes. Progress against these targets should be carefully monitored.

Agree - That it is useful to clarify roles and responsibilities.

However, the most important factor is that ALL health research areas of expertise with the potential to contribute to a given funding initiative are engaged to tackle a given challenge.

As well, it must be clearly recognized that the degree of engagement of each theme will differ across the institutes and across initiatives due to the nature of the health and research challenges being addressed.

It is also important to note that the contribution of open (untargeted) funding to advancing the mandate of each institute must also be considered when reviewing the contribution of each Institute within each theme. While these programs are often viewed as separate from the Institutes, they are in fact a foundational element of each Institute's programming/funding.

Action Plan:

  • Continue to monitor the contribution of each pillar in both the open and targeted funding programs.
  • Review funding program designs and structures to ensure that the portfolio of program vehicles is well adapted to support research in all themes (some vehicles may be better suited to one theme than another).
  • Review the theme engagement strategies used in the context of specific initiatives to ensure that the potential contributions of all themes are appropriately captured to tackle priority health challenges. (incorporated into review/approval of new initiatives)

Timelines for completion are:

  • Ongoing
  • Phase 1 review - March 2008
  • Annual review - ongoing
  • Ongoing

Responsibility:

  • KTC
  • VP, Research
  • RKTC (SPP)

Consultation:
n/a

Responsible for Coordination of Management Response: Liz Stirling Director, Knowledge Synthesis and Exchange (KSE), Acting Director, Michelle Gagnon (October 2007-April 2008)

Recommendation - Knowledge Translation

A consistent finding from key informant interviews across all Institute evaluations was that the Institutes were generally not clear as to their roles and responsibilities in meeting CIHR's Knowledge Translation (KT) mandate. We recommend that:

4.0 CIHR should clearly define and articulate the roles and responsibilities of all parties responsible for the implementation of the CIHR KT Strategy. In addition, these roles and responsibilities need to be clearly communicated throughout CIHR and externally to all stakeholders.

Agree

Action Plan:

  • To clarify what is meant by KT at CIHR, the KT Portfolio has refined CIHR's definition of KT and has identified 2 types of KT to be undertaken at CIHR: end of grant KT and integrated KT. This message has been given to RKTC, IRT, IABs and published (AEM, 2007)2
  • KSE is in the process of implementing the Knowledge Translation (KT) Assessor Project, which will provide criteria and guidelines pertaining to the integration of KT activities into applications for funding and KT review criteria for reviewers. It will also include the development of descriptions of KT across CIHR research themes that will be posted on the website and KT training modules for researchers and for decision makers.
  • KSE is developing standard end of grant and integrated KT related text for the RFA template that can be incorporated into RFAs, where appropriate.
  • KSE is a partner on the Dec. launch of the STIHR and has included KT-related text in the RFA including KT training expectations and sample activities that the funded training initiatives from across CIHR's research themes can undertake
  • KSE is working on a few other initiatives that are meant to help communicate about KT with internal and external audiences, including the multiple stakeholders that are involved in the process and also serve as educational tools. These include the KT Handbook and the KT Casebook.
  • The KT Portfolio, including the KSE Branch is in the process of developing a communication framework for the portfolio based on the vision and mandate of each of its branches. The document also outlines the roles and responsibilities for each branch vis-à-vis the institutes. An accompanying communication plan for the document is under development.
  • The KSE Branch has begun to revamp its website starting with the content of the About KT pages. A comprehensive KSE web strategy is under development in consultation with web services and in the context of the current efforts to revamp CIHR's website by December 2008.
  • A staff person dedicated to KSE communications, including the website and building an internal network of KT stakeholders focused on sharing knowledge and collaboration will be hired by end of 2007.
  • KSE intends to develop an annual/bi-annual report for Governing Council and other audiences, which help clarify the roles and responsibilities of the branch and provide corresponding examples.
  • KSE will focus on communication with the Institutes through its current activities (e.g. a KSE staff person is assigned responsibility to provide advice and act as a resource for each Institutes), by providing updates to the IRT and through ongoing collaboration on projects (e.g. RFAs ).
  • The KSE Branch has organized a number of opportunities for linkage and exchange with external stakeholders this year and intends to continue to do so. These provide an opportunity to communicate about CIHR KT efforts and roles to the external community and to open up opportunities for collaboration. Examples include the October 2007 National Roundtable on partnerships and Knowledge Translation and the upcoming, inaugural KT Funders Forum taking place in February 2008.

Timelines for completion are:

  • Done
  • 2007/08
  • 2007/08
  • 2007/08
  • 2007/08
  • Ongoing
  • 2007/08
  • 2007/08
  • Ongoing
  • Ongoing
  • Ongoing

Responsibility:

  • VP KT and Senior Policy Analyst, KT Portfolio
  • Senior Policy Analyst, KSE
  • Senior Policy Analyst, KSE
  • Director, KSE
  • Senior Policy Analyst, KT, Portfolio, Director, KSE and KSE Staff Leads
  • Director, KSE, KSE staff leads and Senior Policy Analyst, KT Portfolio
  • Director, KSE and KSE staff leads and Senior Policy Analyst, KT Portfolio
  • Director, KSE
  • Director, KSE
  • Director, KSE, KSE Staff Leads and Senior Policy Analyst, KT Portfolio
  • Director, KSE, KSE Staff Leads and Senior Policy Analyst, KT Portfolio

Consultation:

  • Communications
  • Web services
  • Institutes (at November 2007 Institute Roundtable and on an ongoing basis)
  • Evaluation and Analysis Branch
  • Partnerships and Citizen Engagement Branch
  • Research Portfolio

5.0 CIHR should establish relevant performance measures and targets to monitor the implementation of CIHR's knowledge translation mandate, particularly within the Institutes.

Agree

Action Plan:

  • Short and long-term performance targets for KT at CIHR are included in the KT Strategy 2004-2009. A report on progress in relation to the strategy will be undertaken to inform KT strategic planning as part of the Blueprint 2 process.
  • Knowledge translation indicators are included in the Common Performance Measurement and Evaluation Framework for the 13 Institutes. As part of the international review process, each Institute reported on its progress vis-à-vis these indicators. As part of ongoing efforts to improve indicator development, tracking and use, KSE and the KT Portfolio Senior Policy Analyst are working with the Evaluation and Analysis Branch and the Institutes to revise the current set of KT indicators, particularly in light of the CIHR Impact s Framework. It is important that indicators not only cover funding to promote KT, but also Institute activities (for example Senate presentations) to enable KT. Also, Institute initiatives (e.g. integrated KT Initiatives such as the IPPH Centres for Research Development initiative; the workshop held in Summer 2007 led by IG) and KSE Branch projects (e.g. the KT Assessor project) will also feed into ongoing KT indicator development.
  • The KT Portfolio Senior Policy Analyst is leading a project with the Evaluation and Analysis Branch to develop an approach to classify and track knowledge translation projects, funding and applicants in CIHR databases that will advance efforts across CIHR to develop KT indicators and to report more accurately in this area.
  • In collaboration with the Institute of Health Services and Policy Research (IHPSR) KSE supported a KT evaluation of the Interdisciplinary Capacity Enhancement teams grant program to help understand the contribution of this tool to KT and to identify relevant KT indicators that might be used across CIHR.
  • KSE is participating in the IRT led project to enhance clarity in regards to outcomes and impacts of Institute funded initiatives and how to effectively report on these in Institute annual reports. This project should ultimately also help support indicator development and similar efforts across CIHR.

Timelines for completion are:

  • Ongoing
  • Ongoing
  • Ongoing
  • Final ICE KT Evaluation report has been submitted to CIHR
  • 2007/08

Responsibility:

  • Director, KSE and Senior Policy Analyst, KT Portfolio
  • Director, KSE and Senior Policy Analyst, KT Portfolio
  • Senior Policy Analyst, KT Portfolio
  • Director, KSE and Senior Policy Analyst, KT Portfolio
  • Senior KSE Specialist

Consultation:

  • Evaluation and Analysis Branch - Director and responsible managers
  • Institutes (at November 2007 Institute Roundtable and on an ongoing basis)

Responsible for Coordination of Management Response: Francis Savage (Acting) Burleigh Trevor-Deutsch Director, Ethics

Recommendation - Ethics

There is a wide variance across Institutes with respect to their performance on the delivery of the CIHR ethics mandate, with some Institutes being very active and others doing very little. A consistent finding from stakeholder interviews across all Institute evaluations was that the Institutes were not clear as to their roles and responsibilities in meeting CIHR's ethics mandate. We recommend that:

6.0 CIHR should define and articulate the Institutes' roles and responsibilities to address the overall CIHR ethics mandate. It is also recommended that relevant performance measures and targets be established to monitor the implementation of CIHR's ethics mandate, particularly within the Institutes.

Agree

Action Plan:

  • The Ethics Office will consult with each Institute over the next 4 months with a view to clarify their roles and responsibilities in respect of ethics and to design a process to ensure their ongoing and timely engagement in CIHR's ethics projects.
  • After consulting the Institutes and the SCE, the Ethics Office will prepare a report on how to address this recommendation
  • A collective performance management framework will be developed in consultation with the Institutes
  • In consultation with the SCE, the Ethics Office will also prepare a strategic plan which will identify its ethics priorities relevant to the Institutes, and include an action plan, performance measures and targets for each.

Timelines for completion are:

  • Consultations will be completed by July 15, 2008
  • Report will be completed by August 15, 2008
  • To be completed by May 30, 2009
  • Plan will be completed by August 15, 2008

Responsibility:

  • Jaime Flamenbaum & Burleigh Trevor-Deutsch
  • Jaime Flamenbaum & Burleigh Trevor-Deutsch

Consultation:

  • Consultations were made with AD-OBIS and IAB-EDs

Responsible for Coordination of Management Response: Gaëtan Cyr Director, Institute Affairs

Recommendation - Institute delivery model

The evaluations of the Institutes revealed the advantages of the Institute delivery model have so far outweighed the disadvantages in the perspective of Institutes and their stakeholders. The disadvantages that were identified include issues around Scientific Director (SD) transitions and were found to be particularly significant. It is recommended that:

7.0 CIHR fully analyze practical and effective means of achieving the aims of the Institute delivery model while minimizing its disadvantages and managing associated risks and issues. This should include;

  • further assessment of the strengths and weaknesses of the Institute delivery model;
  • effective management of Institute transitions; key elements that need focus are:
    • loss of momentum as an Institute transitions from one SD to another;
    • clarifying the obligations to Institute staff that are employed through Institute operating grants and therefore not entitled to the same job security as CIHR corporate employees

Agree, however see comments where with the time passed since the evaluation, steps have been undertaken to mitigate the issues raised.

Action Plan:

  • As part of the renewal exercise for the Institute Support Grant (ISG) Program's Treasury Board Terms and Conditions, a detailed review of the virtual Institute funding model is underway, with preliminary results indicating the special requirements and strengths of the Institute delivery model outweighing its weaknesses.
  • Institute's ADs and Director, Institute Affairs to hold a work session, with part of the agenda focusing on a review of issues under the Institute delivery model.
  • GC approved a number of extensions to existing SD terms, in order to stagger SD renewals and limit future planned transitions to one in July and one in December per year.
  • Ottawa-based Assistant Directors (OBIS) identified as anchor person for their Institute's transition, with role aimed at maintaining continuity of core business and facilitate transfer of knowledge to new SD and new Institute staff at new Host Institution.
  • Incoming SDs informed on benefits of employee retention & encouraged to use flexible approaches for staffing models, to enable knowledge retention and smooth transitions (e.g., including short to medium-term contracts for existing Institute staff via the new SD/Host Institutions).
  • Increase OBIS staffing model, EMC approved increase to 2 full-time employees per Institute based in Ottawa (up from 1.5 FTEs) to help facilitate Institute staff retention and continuity with SD> transitions.
  • Business case presented/approved to strengthen Institute Affairs team in Ottawa.
  • Institute Affairs to manage the Institute Roundtable (IRT) Committee (Director Institute Affairs named IRT Exec. Secretary), with the Director, Institute Affairs and the IRT well positioned to advise on transition issues.
  • Institute's ADs and Director, Institute Affairs to hold a work session during the Dec IRT focusing on lessons learned and possible improvements to Institute transitions.
  • Create new position of Manager, Institute Affairs to lead SD orientation and operational aspects of Institute transitions.
  • Institute's ADs and Director, Institute Affairs to monitor and document any further issues that arise with transitions and discuss mitigating strategies.
  • Under the Institute model, Institute employees hired and based at the host institutions are employees of the host institutions (not of CIHR) and are bound by the rules, policies and practices of the respective host institutions. This is clearly identified in both the Institute Agreements and the ISG Manual launched in November 2005.
  • A legal review confirmed low risk that a claim by a host employee (Institute-based employee) that he/she was in law a CIHR employee would succeed.

Timelines for completion are:

  • To be completed by March 31, 2008
  • Completed Dec 07
  • Completed May 06
  • Completed October 06
  • Completed - in place since October 06
  • Completed April 06
  • Completed May 07
  • Completed Feb 06
  • Completed Dec 07
  • Completed Dec 07
  • Ongoing
  • Completed Nov 07

Responsibility:

  • CFO
  • Institute ADs
  • EVP
  • Director, Institute Affairs
  • Director, Institute Affairs
  • EVP
  • Director, Institute Affairs
  • EVP
  • Institute ADs
  • Director, Institute Affairs
  • Director, Institute Affairs
  • CFO

Consultation:

  • James Roberge, Francis Savage, Gaëtan Cyr
  • Gaëtan Cyr, IRT
  • Gaëtan Cyr
  • Gaëtan Cyr
  • Gaëtan Cyr, Scientific Directors
  • Gaëtan Cyr, RPPC
  • Gaëtan Cyr
  • Gaëtan Cyr
  • Gaëtan Cyr, IRT
  • Gaëtan Cyr
  • Gaëtan Cyr
  • CIHR Legal Counsel, IRT
  • CIHR Legal Counsel, TBS Legal Counsel

Responsible for Coordination of Management Response: Karen Spierkel Director, Communications

Recommendation - Communications

With respect to performance on communications, the Institute evaluations revealed inconsistent results across the 13 Institutes, both within CIHR and from CIHR to its external partners and stakeholders. We recommend that:

8.0 CIHR should work with the Institutes to clarify communications expectations and to ensure that Institute-specific information is shared between CIHR central and Institute staff. We also recommend that the Director of Communications continue to work with Institutes to ensure that the key results and achievements of Institute-funded research, as well as Institute priority areas, are communicated clearly to external audiences.

Agree, however, see comments attached. The time passed since the evaluation was undertaken has allowed us to address all of the issues raised.

Action Plan:

  • Development and implementation of 3-year communications strategy for CIHR and approval by RPPC; the strategy specifically addressed the working relationship between communications and the institutes, and outlined responsibilities for each in order to implement strategic institute communications activities - Completed
  • Hiring of full staff complement; each staff person is assigned to specific institutes in order to deliver communications services to those institutes more directly - Completed
  • Development of Institute communications plans and implementation of the tactics; communications plans were developed for each of the 13 institutes, and these are updated every year, and presented to Institute IABs and staff - Completed and ongoing
  • Development and implementation of national and regional media strategies; these strategies are developed annually to raise profile of CIHR results and successes and highlight institute initiatives - Completed and ongoing
  • Redesign of CIHR and all Institutes websites, to better highlight Institute related information, news and results - Completed
  • Development and implementation of Creative Services Business Case for essential services: translation, editing, writing, production in order to provide better services to institutes - Completed
  • Organization of 2 journalist workshops in cooperation with the Institute of Genetics and IHSPR (06/07); 2 more Media workshops are planned for 07/08 with INMHA and INMD. These workshops raise the profile of the institutes' research area, and create relationships with journalists covering science in the field. This allows the institutes to better communicate the results of research to media. - Completed and ongoing
  • Creation and organization of CIHR Cafés Scientifiques to foster health research awareness. 20 Cafes are planned for 07/08 to be held across Canada as part of a National CIHR Café Scientifiques program. These Cafes are developed with the institutes on themes relevant to that institute, allowing the institute to undertake public engagement activities and better communicate the results of research undertaken in the field to lay audiences. Subjects already covered have included mental health, obesity, aging and population health. - Completed and ongoing

Timelines for completion are:
n/a

Responsibility:

  • Karen Spierkel
  • Karen Spierkel
  • Angela Prokopiak
  • Angela Prokopiak
  • Karen Spierkel/Richard Dubois
  • Louise Séguin-Guénette
  • Angela Prokopiak
  • Christian Riel

Consultation:
Each of the Institutes were consulted in the development of the overall CIHR Communications strategy. The strategy was presented to RPPC, Institute Roundtable and EMC for approvals and buy-in. Furthermore, Institute Scientific Directors and Staff are consulted on development of Institute specific communications plans, media strategies and special projects such as Café Scientifiques, Media Workshops or individual web sites.

Responsible for Coordination of Management Response: Kathryn Moore Director, Governance

Recommendation - Institute Advisory Boards

While Institute Advisory Boards are generally found to be operating effectively, concern has been raised by senior management and Governing Council members that the potential exists for IABs to operate as governance mechanisms, rather than in the proper role as advisory bodies. We recommend that:

9.0 CIHR should work to ensure that all current and future IAB members are aware of the 'advisory' nature of their role and that the accountability relationships between CIHR and the Institute Scientific Director be clearly articulated.

Agree

Action Plan:

IAB Chairs Meeting

  • The purpose of the meeting is within the context of CIHR moving to an Institute-centric organization. This annual one day meeting, hosted by the Governance Unit, allows IAB Chairs to come together in a forum where they will be able to exchange ideas and best practices, be cognizant of key priorities and directions of CIHR and within each others' Institutes, and generally, work collectively toward the realization of CIHR's mandate.

IAB Orientation Manual

  • Orientation is provided to all Institute Advisory Board members following their appointment to any one of CIHR's thirteen Institutes. The Roles and Responsibilities of IABs are clearly delineated in the Orientation Manual:
  • IABs act in an advisory capacity to the Scientific Directors of the Institutes and to the Governing Council with respect to the full range of Institute activities. These activities include deliberation and evaluation on Institute health research priorities, and strategies and implementation plans for engaging the broader research community, universities, government, private sector, voluntary organizations, patient groups and other stakeholders and the Canadian public.
  • A copy of the orientation manual is shared with all new Scientific Directors as part of their transition process

CIHR Recruitment of new IAB members

  • Every year, CIHR seeks members to fill voluntary membership positions for its thirteen Institute Advisory Boards. The recruitment advertisement (web and E-Alert) informs applicants of the IABs advisory capacity, i.e.:
  • Diversity of expertise and experience are keys to the success of IABs. Each of CIHR Institute IABs is made up of 16 volunteers from Canada and abroad who meet at least twice a year to advise the Institute on health research priorities. They also consider how best to shape and implement health research strategies in line with these priorities, and to accelerate the flow of knowledge into health benefits.

Timelines for completion are:

  • Once a year.
  • Manual is available and update periodically.
  • Ongoing.
  • Ongoing process

Responsibility:

  • Governance Unit
  • Governance Unit is responsible for providing the chapter on responsibilities of the IAB members. The Governance Unit is also facilitating the production of Institute orientation manuals.
  • Governance Unit

Consultation:
n/a

Responsible for Coordination of Management Response: Peggy Borbey Director, Analysis and Evaluation

Recommendation - Data Quality and Management

There is variation between the data tracked and reported centrally at CIHR and the data tracked and reported at some Institutes. Some examples of inconsistencies include the number of partners for strategic initiatives, the numbers of projects supported, whether the funding is from open or strategic budgets etc.. There are also questions of data quality due to researchers self-selections during the application process and using an inadequate classification system for research overall. The result is that there is variation in the way in which data are collected and presented across Institutes and at CIHR centrally. These inconsistencies negatively impact the overall ability to measure and report on performance in a consistent, accurate and credible manner. We recommend that:

10.0 CIHR should conduct a review of data management policies and practices across CIHR and document data quality issues. The purpose of the review should be to enable improved accuracy and consistency through creation and implementation of a strategy that clearly defines roles, responsibilities and accountabilities to ensure that data are tracked and reported accurately, consistently and reliably in the future.

Agree This review should be taken up as an organization priority, not only by the Evaluation and Analysis Branch, but to include key other players such as EA Branch, Policy, Communications, ITMS, and the 13 Institutes.

Action Plan:

  • Initiate consultations with other players (ex. Institutes) to raise CIHR issues and needs
  • Review existing CIHR information policies, guidelines, and work processes related to data quality
  • Review other organizations' guidelines and frameworks related to data quality
  • Present project plan to IMIT Subcommittee of EEMC
  • Create guide documenting CIHR needs and proposed data quality framework
  • Develop implementation plan including estimation of the technical/financial and human resource implications as well as prioritization and scheduling of any specific change activities that will be required.
  • Begin implementation of the project according to plan.

Timelines for completion are:

  • Completed
  • Completed
  • Completed
  • Completed
  • March 2008
  • July 2008
  • Nov-Dec, 2008

Responsibility:

  • Evaluation & Analysis Branch
  • Evaluation & Analysis Branch
  • Evaluation & Analysis Branch
  • Evaluation & Analysis Branch
  • Evaluation & Analysis Branch
  • Evaluation & Analysis Branch, Research portfolio staff, ITMS, Policy, Finance
  • Evaluation & Analysis Branch, Research portfolio staff, ITMS, Policy, Finance

Consultation:

  • Bey Benhamadi, Manager of Data and Analysis
  • Peggy Borbey, Director, Evaluation and Analysis
  • Evie Gray, CIO
  1. Note that the survey of researchers was a cross-Institute survey conducted by EKOS under separate contract.
  2. Relevant publications include: Lost in Knowledge Translation; Theoretical Underpinnings of Knowledge Translation; CIHR Research: How to Translate Health Research Knowledge into Effective Healthcare Action