Canadian Institutes of Health Research Evaluation Plan 2010/11 - 2014/15

Other format(s)


PDF version
(281 KB)

Evaluation Unit
Evaluation, Internal Audit and Risk Management Branch
Strategy and Corporate Affairs Portfolio
October 2010

Table Of Contents

List of Tables

About the Canadian Institutes of Health Research

Mandate

The mandate of CIHR is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).

Vision

CIHR's vision is to position Canada as a world leader in the creation and use of health research knowledge that benefits Canadians and the global community.

Achieving the mandate and vision

Established in 2000, CIHR promotes a solutions-focused, multidisciplinary and collaborative approach to health research. Its unique structure brings together researchers from across disciplinary and geographic boundaries through its 13 Institutes. Currently, CIHR supports more than 13,000 health researchers and trainees in universities, teaching hospitals and other health organizations and research centres across the country. The agency's mandate is to support health research in a transparent process that meets the highest international standards of excellence and ethics in four research areas: biomedical; clinical; health systems and services; and the social cultural and environmental factors that affect the health of populations.

Institutes

CIHR's 13 Institutes share responsibility for achieving the fundamental objective of CIHR and have their own distinct strategic plans that are aligned with the overarching directions, mandate and vision of CIHR. The Institutes promote and build upon Canada's firm foundation of research excellence, engage the research community and encourage interdisciplinary, integrative health research and knowledge translation. Through their Scientific Directors and Institute Advisory Boards and under the oversight and guidance of CIHR's Governing Council, the Institutes' mandate is to work with stakeholders to forge a health research agenda across disciplines, sectors and regions that embraces scientific opportunity and reflects the emerging health needs of Canadians, the evolution of the health care system and the information needs of health policy decision makers.

1. Background and Context

1.1 Purpose

The purpose of this Evaluation Plan (EP) is to:

  • Provide schedules of evaluations to be conducted over the next five years;
  • Provide context around Treasury Board requirements for producing an EP and how this will be used within CIHR;
  • Describe the approaches used to identify and prioritize CIHR and horizontal evaluations;
  • Describe resource requirements and utilization within the Evaluation Unit to deliver scheduled evaluations;
  • Identify risks and success factors for conducting evaluations.

1.2 Treasury Board Requirements for Evaluation Plans

The primary purpose of the Evaluation Plan is to assist CIHR's President in ensuring that credible, timely and neutral information on the ongoing relevance and performance of direct program spending is available to support evidence-based policy decision-making. The EP helps to ensure accountability for the results achieved by CIHR policies and programs.

The EP also serves a range of other purposes including:

  • Assisting the President in confirming that the information needs of CIHR are being met;
  • Providing program managers with an opportunity to ensure that planned evaluations can be timed to support program redesign;
  • Providing program managers and the Head of Evaluation1 with an annual platform to engage in a dialogue about the development and implementation of performance measurement strategies that effectively support evaluations;
  • Providing units responsible for developing Reports on Plans and Priorities (RPPs) and Departmental Performance Reports (DPRs) as well as other groups engaged in strategic planning and reporting activities an opportunity to identify when evaluations will be available to inform their work;
  • Providing the Head of Evaluation with an opportunity to initiate regular communication and consensus building on evaluation needs and priorities across CIHR;
  • Serving as a management tool for the Head of Evaluation, allowing for the planning of workflow and human resource needs for the coming years, reflecting on progress and incorporating lessons learned from previous years to strengthen the function.

The Policy on Evaluation2 requires that the President ensures the development of a five-year rolling evaluation plan and confirms that this plan:

  • Aligns with and supports CIHR's Management, Resources and Results Structure;
  • Supports the requirements of the Expenditure Management System including strategic reviews; and
  • Includes all ongoing programs of grants and contributions as required by section 42.1 of the Financial Administration Act.

In order to ensure evaluation coverage of PAA categories containing large numbers of programs and initiatives, the Evaluation Unit will undertake a sampling approach to establish the scope of the evaluation; further detail on this can be found in Section 2.3 of this document.

The President is responsible, with advice from CIHR's evaluation committee (EC)3 for approving the EP which is then submitted to the Treasury Board of Canada Secretariat (TBS). Evaluation coverage requirements that should be met and reflected in the EP include:

  • All Direct Program Spending,4 excluding grants and contributions, to be evaluated every five years;
  • All ongoing programs of grants and contributions to be evaluated every five years5.

The Head of Evaluation is responsible for developing and annually updating the rolling five-year EP and ensuring that it meets the requirements outlined above. This includes responsibility for identifying and recommending a risk-based approach for evaluations, submitting an approved EP to TBS and ensuring that evaluations include clear and valid conclusions about the relevance and performance of programs.

1.3 Evaluation at CIHR

CIHR has been building internal capacity for evaluation, performance measurement and analysis since 2001. The Evaluation Unit works alongside an Impact Assessment Unit which focuses on the wider impacts of CIHR's programs and a Data Analysis group which leads on data systems and information services. The Evaluation Unit itself is primarily focused on supporting evidence-based decision-making and strategic planning across CIHR, as well as on meeting or exceeding the requirements of the TB suite of policies on evaluation. Meeting or exceeding TB policy requirements provides management with assurance on the credibility and neutrality of evaluation processes and products. CIHR's strategic plan, the Health Research Roadmap6, references the importance of the evaluation function in helping the agency to achieve organizational excellence and demonstrate impact.

The Evaluation Unit has not been fully staffed during 2010 due to staff changes and recruitment challenges, and was for some time operating with three of eight FTEs. Recent recruitment has seen staffing recently reach specified levels, although finding suitably experienced and qualified evaluation staff remains a challenge, as is the case for many government departments and agencies. Staffing challenges have resulted in some limited delays in the completion of scheduled evaluation projects, although these have not impacted on any evaluations of programs with TB submissions or on horizontal initiatives.

The Evaluation Unit has recently completed a re-organization within the agency since the last EP was submitted and is now part of a new Evaluation, Internal Audit and Risk Management Branch, within CIHR's Strategy and Corporate Affairs Portfolio. The new Head of Evaluation (the Director of Evaluation, Internal Audit and Risk Management)7 has a direct reporting line to the President of CIHR, responding to neutrality concerns raised in this year's Management Accountability Framework (MAF) Assessment report on the function.8

Figure 1 - Reporting Line between Head of Evaluation and CIHR Presiden

Figure 1 - Reporting Line between Head of Evaluation and CIHR President

The chart below shows new governance arrangements for CIHR's Evaluation Committee as of August, 2010. The Executive Management Committee, chaired by the President of CIHR, is now CIHR's Evaluation Committee. The Subcommittee on Performance Measurement (SPM) plays a strategic and advisory role on the design and conduct of evaluations. The Chair of SPM presents substantive evaluation items to EMC, supported by Evaluation Unit staff. The Evaluation Unit can also be requested to bring items for information to the Audit Committee as required.

Figure 2 - Evaluation Governance Structure at CIHR

Figure 2 - Evaluation Governance Structure at CIHR

1.4 Developing the Evaluation Plan

The 2010/11 Evaluation Plan is based on a refresh of EPs from 2008/9 and 2009/10 that have been submitted to TBS by CIHR. A full planning and consultation exercise is scheduled for the 2011/12-2015/16 plan, following the new TBS guidance on developing EPs9.

This refreshed EP has been developed using the following approach:

  • Scoping the evaluation universe using the Program Activity Architecture (PAA)10, MRRS and supplementary information from program managers;
  • Reviewing and refreshing the risk-based criteria used in previous CIHR EPs (materiality; program lifecycle; accountability history; strategic/corporate priorities);
  • Consultation with senior managers and program managers at CIHR (list provided in Appendix);
  • Consultation with the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC) on tri-agency scheduling of horizontal evaluations.

1.5 Scheduling changes from 2009/10 Evaluation Plan

Any significant amendments to the 2009/10-2013/14 evaluation schedule submitted to TBS are provided in the table below, with a rationale provided for each change. As CIHR's PAA categories have changed between 2009/10 and 2010/1111, a crosswalk has been provided to allow for comparisons between the two PAAs to be made.

Table 1 - Significant changes between 2009/10 and 2010/11 Evaluation Plans

Program Evaluation PAA 2010/11 PAA 2009/10 Change Rationale
Partnerships for Health System Improvement (PHSI) & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Knowledge Translation Programs) 1.4.2 2.3.2 & 3.1.1 Evaluation brought forward to 2010/11 from 2011/12 Change to PAA increased materiality (addition of $9M of Partnership Programs) and associated risk.
Strategic Initiatives (including Regenerative Medicine and Nanomedicine and the Canadian Light Source Program) 1.4.1.1 1.2.1
1.2.2
2.1.5
2.1.6		 Evaluation timeline increased - completion date of evaluation report now in 2011/12 CIHR's International Review will provide much of the required data for this evaluation. The new timeline allows for the evaluation to be more cost-effective by significantly reducing the amount of additional data collection required.
Collaborative Health Research Partnerships Program (CHRP) & Proof of Principle Program (POP I and II)
(Research Commercialization Programs)		 1.3.1 3.2 Evaluation delayed to 2011/12 An initial evaluability analysis showed that insufficient data was available for a full evaluation. Strategies will be developed by program managers with advice from the Evaluation Unit for data to be collected; the evaluation is planned to be conducted in 2011/12.
Open Research (Operating) Grants Program - Evaluative Study of the Historical Outputs, Outcomes and Impacts 1.1.1 1.1.1 Report completion delayed from 2009/10 to 2010/11 Resource constraints delayed completion
Business-Led Networks of Centres of Excellence Program - Review of relevance and effectiveness 1.3.2.2 3.1.2 (part) Review of relevance and effectiveness scheduled for 2010/11 Tri-Agency evaluation which was not specified on CIHR 2009/10 schedule.

1.6 Reports to be submitted during 2010/11

The following table lists reports that will be submitted to Treasury Board during 2010/11.

Table 2 - Reports to be submitted during 2010/11

Program Evaluation PAA 2010/11 PAA 2009/10
Open Research (Operating) Grants Program - Evaluative Study of the Historical Outputs, Outcomes and Impacts 1.1.1 1.1.1
Randomized Control Trials Program 1.1.2 1.1.2
Salary Support Programs 1.2.1.1 2.1.1 & 2.1.5
Canada Research Chairs (led by Social Sciences and Humanities Research Council) 1.2.1.2 2.1.3

1.7 Approval of the Evaluation Plan

As required by the Policy on Evaluation, (section 6.1.7) this Evaluation Plan has been reviewed by CIHR's Evaluation Committee and approved by CIHR's President.

The Confirmation Note, as outlined in the Treasury Board Secretariat's Guide to Developing Departmental Evaluation Plans, has been included as an appendix to this document.