CIHR 5-Year Evaluation Plan 2008/09 – 2012/13

[ Table of Contents ]

4. Evaluation Unit Activities

4.1. Evaluation Activities for 2008/09 – 2012/13

Fiscal Year 2007/08 has been the first year for the Evaluation Unit operating at almost full staffing capacity. In that year, a number of small and large-scale evaluation projects have been undertaken.

The current year, 2008/09, is devoted to beginning the process of fulfilling the CIHR 5-Year Evaluation Plan requirements. The evaluation commitments for 2008/2009 reflect high priority program evaluations, previously identified by CIHR management, such as the Open Operating Grants program (OGP) evaluation, evaluation of CIHR team-type programs, Randomized Control Trials, as well as TBS requirements (Pandemic Preparedness Research Initiative) and Tri-agency evaluations (Indirect Costs, Canada Research Chairs program). Finally, several evaluation studies were carried over from 2007/08 to be completed this fiscal year, such as the evaluation of the Strategic Training Initiative in Health Research (STIHR), Canada Graduate Scholarships Program (CGS), Doctoral Research Awards (DRA) and the evaluation of the Interagency Advisory Panel and Secretariat on Research Ethics (PRE/SRE).

Table 2 below documents the Agency's planned program evaluations for the upcoming 5 years (2008/09 – 2012/13). It builds and expands upon the results of the prioritization exercise. Where appropriate, each commitment has been operationally defined in terms of specific projects, approaches and resource needs.

In some instances, the evaluation information needs of managers have necessitated the re-clustering of programs within and between PAA categories. For example, the Team Grant Program PAA category has been expanded to include all 10 CIHR team-type programs, historical and existing, to ensure that management will get sufficient evidence as to the value-added of the 'team approach' to funding and conducting research. In order to evaluate the OGP, largest CIHR program, it is broken down into multiple evaluative studies, spread across several years, thereby making the evaluation more manageable and focused.

Other adjustments have included the merging of pairs of PAA categories into single evaluative studies (e.g., Training Programs and Strategic Training Programs; Salary Programs and Strategic Salary Programs), which will enable an analysis of, for example, the effectiveness of a strategic versus an open approach to funding, among others.

All evaluations included in the 5-year schedule will address value for money by covering core issues proposed by the TBS Directive on the Evaluation Function. The actual scope of projects for years 2009/10 to 2012/13 will be further defined by resources available, the status of previous evaluation projects and information needs of CIHR management following an annual re-profiling of the Plan by the Evaluation Unit.

Top of Page

Table 2. Proposed Evaluation Projects for 2008/09 – 2012/13.

[Please note that more precise information is known for projects slated for an evaluation in the current year than future years. Evaluations not led by the CIHR's Evaluation Unit are highlighted in grey].

PAA Sub-Activity Category	Basic stats 2007/08	MRRS Outcomes and Expected Results (CIHR MRRS, 2007)	Proposed Evaluation(s) (if known)	Preliminary Evaluation Foci	Start Year	Rationale / Resources / Comments
1.1.1. Open Operating Grant Program	3898 grants funded $383M $98.3K/ per grant	Expected Results: Health research advances knowledge and leads to broader impacts such as informing decisions, impacts on health and the economy Appropriate resources available that enable world class, investigator formulated research across all areas of health research Output: Research grants	1. Peer review evaluation	Peer Review	2008/09	Given its magnitude, size and dated evaluative information available to date, this program requires comprehensive assessment that will be conducted over several years to capture evaluation issues best. Several evaluative studies are being proposed and various issues of interest will be explored (e.g., assessing the peer review process especially in the context capturing research excellence; the program’s scale and scope; international competitiveness and other issues). The evaluations are to be conducted internally with some tasks contracted out. The following budgetary allocations have been proposed: $25,000 for 2008/09; $40,000 for 2009/10 and $35,000 for 2010/11.
			2. Historical outcomes and impacts assessment	Health research impacts	2008/09
			3. Program’s meta-evaluation	Overall effectiveness and relevance	2010/11
1.1.2. Randomized Control Trials Program	74 grants funded $32.3M $436.7K/ per grant	Expected Results: Better health research, research training and mentoring through funding of excellent teams Appropriate resources available that enable team grants to enhance research and training Output: Research grants	Program’s effectiveness (analysis of the final reports data)	Health research impacts	2008/09	The RCT evaluation was identified as a high priority as a result of multiple factors: the program’s high profile on the international arena, large per grant spending ratio, the decision to merge RCT into OGP, and the immediate availability of data collected through final reports (Consolidated Standards of Reporting Trials-CONSORT). This evaluation is to be conducted internally. The framework is now being developed in order to ensure that information needs of senior management are adequately met and sufficient evidence about the program’s outcomes and effectiveness is generated.
1.1.3. Team Grant Program	109 grants funded $26.7M $244.7K/ grant	Expected Results: Better health research, research training and mentoring through funding of excellent teams Appropriate resources available that enable team grants to enhance research and training Output: Research grants	Evaluation of all team-type programs at CIHR: - CAHR - CGSP - Groups - Teams - ET - ICE - IHRT - NET - PHSI - RRG (For Acronyms, refer to Appendix A)	Team Research Productivity Capacity development Knowledge translation	2008/09	The evaluation strategy was developed by the Evaluation Unit in the Spring of 2008. The proposed evaluation will include 10 team-type programs (ongoing and historical) that are spread out across different PAA categories. Such an approach will enable evaluators to provide solid evidence on the value-added of the team approach to funding research and, potentially, enrich the theory on team performance and productivity. Special attention will be devoted to developing evaluative tools for such horizontal issues as capacity development and knowledge translation. The project’s budget includes $25,000 in 2008/09.
2.1.3. Canada Research Chairs Program	693 grants funded $86.4M	Expected Result: World-class research capacity is enhanced in Canadian universities, research institutes and hospitals through the attraction and retention of excellent researchers Output: Research chairholder is established	TBD externally	Capacity Development	2008/09	External evaluation (Tri-Agency) Lead: CRC Secretariat (SSHRC) The evaluation is a requirement for the 2009/10 year. The planning will start in November 2008. The CRC program is intended to attract and retain some of the world's most accomplished and promising researchers; therefore, it will be of interest to CIHR to consider CRC outcomes in the context of other capacity development programs.
1.2.4. Pandemic Preparedness Research Initiative	40 grants funded $2.8M $70.5K/ grant	Expected Results: Health research advances knowledge and responds to objectives of Pandemic Preparedness Strategic Research Initiative Appropriate resources available that enable world-class research related to Pandemic Preparedness Output: Research grants	TBD externally	National Health Priorities	2008/09	External evaluation Lead: Institute of Infection and Immunity A formative evaluation will be carried out in 2008/09. The proposed evaluation will look at the relevance and effectiveness of the Initiative’s task group, researcher and stakeholder perceptions on initiative design and implementation, and success of activities aimed at fostering linkages and exchanges.
n/a Indirect Costs Program		n/a	n/a	TBD Externally	2008/09	External evaluation Lead: CRC Secretariat (SSHRC) A comprehensive evaluation of the program to be carried out during its sixth year of operation is a TB requirement. The primary focus of the evaluation is the continued relevance of the program and its cost-effectiveness as well as its results, expected outcomes and impacts. The project is currently scheduled to start in September 2008. CIHR will be participating in a working group.
1.2.2. Large Strategic Initiatives Programs	172 grants funded $20.8M $121.2K/ grant	Expected Results: Health research advances knowledge and leads to broader impacts such as informing decisions, impacts on health and the economy and responds to objectives of large, cross-cutting strategic initiatives Appropriate resources available that enable world-class research responding to cross-cutting strategic health priorities Output: Research grants	Cluster evaluation of large strategic initiatives	Health Research Priorities Health Research Impacts	2009/10	Evaluating large, cross-cutting strategic initiatives is important from the decision-making perspective. The proposed evaluation will look at the model of funding strategic initiatives holistically (rather than success of each individual initiative) and will build upon the previous Obesity Initiative evaluation framework and impact assessment study, as well as others. It will be conducted internally by the Evaluation Unit.
2.2.1. Research Resources and Collaborations	420 grants funded $49.4M	Expected Results: High quality research resources are available for excellent health research as a result of effective funding programs Appropriate resources available that enable adequate research resources for health research Output: Research grants	TBD	Partnerships Citizen engagement	2009/10	The proposed timing for this evaluation may need to be re-examined on the materiality and overall scores, given that the team-type programs that constitutes 2/3 of the budget for this category are evaluated with the Team Grant PAA. The scope of the evaluation will be determined through future consultations.
2.1.1. Salary Support Programs	416 awards funded $21.1M $50.8K/ award	Expected Results: Medical/Health, University and Research Institution staff have dedicated time for health research Appropriate resources available that enable world-class health researchers across all areas of health research to devote more time to health research responding to research opportunities Output: Salary Awards and research allowances	Programs’ relevance and effectiveness	Capacity Development	2009/10	In 2004/05, the Task Force on Career Support examined CIHR’s suite of programs offering salary awards to health researchers and made a series of recommendations. It is expected that a more comprehensive evaluation of salary programs and their role in enabling robust research environment will be performed in 2009/10 by the Evaluation Unit.
2.1.5. Strategic Salary Support Programs	50 awards funded $2.7M $54.8K/ award	Expected Results: Medical/Health, University and Research Institution staff have dedicated time for health research on strategic initiatives Appropriate resources available that enable world-class health researchers to devote more time to health research in strategic areas of health research Output: Salary Awards and research allowances	Programs’ relevance and effectiveness	Capacity Development	2009/10
n/a External Review of CIHR		n/a	n/a	TBD	2010/11	As required by the CIHR Act (2000), the Agency should conduct a quenquennial review of CIHR’s Research Institutes. The Evaluation and Analysis Branch will take the lead for this high profile project which will include a review of the slate of Institutes.
1.2.1. Strategic Priority Operating Grant Program	430 grants funded $38.2M $88.9K/ grant	Expected Results: Health research advances knowledge and leads to broader impacts such as informing decisions, impacts on health and the economy and responds to objectives of strategic initiatives Appropriate resources available that enable world-class Institute-framed research responding to strategic health priorities Output: Research grants	TBD	Health Research Priorities	2010/11	The scope and methodology for evaluating the strategic priority operating grant program will be developed in conjunction with the evaluation of the large strategic initiatives and the open operating grants program.
1.2.5. Expensive Drugs for Rare Diseases Initiative	1 grant funded $11.6M $11,600K/ grant	Expected Results: Additional information available regarding the use of enzyme replacement therapies to treat patients suffering from Fabry Disease Output: Research grant	TBD externally	National Health Priorities	2010/11	External evaluation Lead: Scientific Oversight Committee This initiative is a joint investment by the Government of Canada, participating provinces and the private sector, with the objective of gathering therapeutic effectiveness information on the rare genetic Fabry Disease. The Independent Scientific Oversight Committee is responsible for monitoring, evaluating and communicating publicly the results of the initiative’s research.
2.3.2. Partnership Programs	263 grants funded $9.1M $34.7K/ grant	Expected Results: Relevance and quantity of health research increased through innovative national and international alliances Appropriate resources available that enable partnered health research Outputs: Research grant; Training Awards; Salary Awards	Cluster evaluation of partnership programs	Partnerships Knowledge Translation	2010/11	The Partnership program provides grants to enable national/international programs to coordinate health research activities with stakeholders. Given similar goals/scope of the program with the KT program, the feasibility of a joint evaluation will be further examined.
3.1.1. Knowledge Translation Funding Program	192 grants funded $9.5M $49.5K/ grant	Expected Results: Health research is translated more effectively as result of funding programs Appropriate resources available that enable world-class knowledge translation of health research Output: Research grants	Knowledge Translation Study	Knowledge Translation Partnerships	2010/11	Evaluation of two types of KT (end-of-grant and integrated KT) will be at the core of the proposed evaluative study. CIHR Knowledge Exchange and Synthesis Directorate will be involved.
3.2.1. Commercialization Funding Programs	490 grants funded $98.9M $201.9K/ grant	Expected Results: Health research is commercialized more effectively as a result of funding programs Appropriate resources available that enable commercialization of health research Output: Research grants	Outcomes and impacts of commercialization of health research	Knowledge Translation Partnerships Health Priorities	2010/11	An initial evaluability assessment is to be carried out, followed by development of the performance measurement and evaluation framework ($35,000 budget for 2009/10). Future evaluative activities are contingent upon the framework.
1.2.3. HIV/AIDS Research Initiative	199 grants funded $17.3M $87.1K/ grant	Expected Results: Health research advances knowledge and responds to objectives of HIV/AIDS Research Initiative Appropriate resources available that enable world-class research related to HIV/AIDs Output: Research grants	TBD externally	National Health Priorities	2010/11	External evaluation Lead: PHAC 4 federal departments are responsible for delivering this Federal Initiative, launched in 2004/05 (renewal in 2010). The Public Health Agency of Canada will lead the evaluation.
2.1.2. Training Support Program	796 awards funded $24.2M $30.5K/ award	Expected Results: Highly trained individuals available for health research Appropriate resources available that enable superior health research trainees in all areas of health research to be qualified to pursue careers in health research Output: Training Awards and research allowances	Cluster evaluation of all training programs	Capacity Development	2011/12	Training programs have been well evaluated in 2006-2008 as individual programs. Building upon this work, an over-arching evaluation that looks at the effectiveness of all training programs (incl. capturing excellence) is proposed.
2.1.6. Strategic Training Support Program	299 awards funded 36.3M $121.6K/ award	Expected Results: Highly trained individuals available for health research in strategic priority areas of health research Appropriate resources available that enable superior health research trainees to pursue careers in strategic areas of health research Output: Training Awards and research allowances	Cluster evaluation of all training programs	Capacity Development	2011/12
1.2.6. National Anti-Drug Strategy Treatment	0 grants funded	Expected Results: Appropriate resources available that enable world-class research related to improving the effectiveness and informing policy and program development of and for addiction treatments (for youth, First Nations and Inuit peoples) Output: Research grants	TBD externally	National Health Priorities	2011/12	External evaluation Lead: Health Canada This is a new initiative with first grant programs launched in early 2008. Part of Government’s National Anti-Drug Strategy, the evaluation will be led by Health Canada.
2.1.4. Canada Graduate Scholarship Programs	747 awards funded $13.9M $18.6K/ award	Expected Results: Increased capacity to meet demand for highly qualified personnel in the academic, public and private sectors Appropriate resources available that enable superior health research trainees in all areas of health research to be qualified to pursue careers in health research Outputs: Training Awards	TBD	Capacity Development	2011/12	Tri-Agency Lead: CIHR This Tri-Agency evaluation will be led by CIHR and its Evaluation Unit. It will build upon the 2007/08 study in conjunction with the evaluation of CIHR’s training programs.
3.1.2. Networks of Centres of Excellence Grants	11 grants funded $27.5M	Expected Results: Strong linkages and partnerships created between university, government and industry, and other users (NGOs), resulting in the transfer of knowledge and exploitation of leading-edge research results with economic or societal benefits to Canada Output: Networks supported	TBD externally	Capacity Development Partnerships	2011/12	External evaluation (Tri-Agency) Lead: NSERC The evaluation will build upon the 2006 study. It will be led by the Natural Sciences and Engineering Research Council of Canada.
2.3.1. Institute Support Grant	14 grants funded $13M	Expected Results: Effective domestic and international health research agendas as well as alliances and partnerships in areas related to the mandate of each Institute Outputs: Discussion documents and other publications Communications regarding funding opportunities and funding results Relationships and networks Established Strategic Priorities	TBD	TBD	2012/13	Institute support grants are being redesigned following extensive consultations to address TB concerns. The timing of this evaluation would allow to prepare for renewal of the terms and conditions of this grant in 2013/14.
2.4. Ethical, Legal, Social Issues	50 grants funded $1.5M $30.8K/ grant	Expected Results: Health research conducted more ethically as a result of effective funding programs Output: Research grants	TBD	Ethics	2012/13	This evaluation is to be carried out internally, building upon the 2008/09 PRE/SRE study.

Top of Page

4.2. Performance Indicators

As part of the evolving performance measurement system at CIHR, the Evaluation Unit will report on its performance using a small set of key performance indicators:

Percentage of annual work plan and 5-Year Evaluation Plan accomplished. This could be a measure of the Unit's ability to adhere to project schedules and to efficiently manage the evaluation process.
Quality and usefulness of products and services, as assessed by clients. This could be accomplished via a combination of post-evaluation project client surveys, together with an annual survey of all senior managers and key stakeholders.

Top of Page

4.3. Resource Requirements

The Evaluation Unit has a resource base of 8 FTEs. The Unit's staff consists of the following personnel: one manager, one evaluation and quality control officer, 4 evaluators¹⁰ and 2 junior evaluators. One additional evaluator position is being proposed for the coming fiscal year.

It is expected that much of the evaluation work will be conducted using in-house resources.

In addition to undertaking program evaluation projects as defined in previous and current evaluation plans, the Evaluation Unit dedicates considerable amount of time to providing support and advice to other organizational functions. Approximately 20 to 30% of evaluators' time is devoted to client support. The Unit supports program management in assessing the design, delivery and effectiveness of policies, programs and initiatives (e.g., advice on results-based management and accountability frameworks, conduct of program-level evaluative studies, performance measurement activities and data analyses).

If you require more information about client support and for contact information, please refer to the next section.

Top of Page

4.4. Evaluation and Analysis Contacts

Anyone requiring the assistance of the Evaluation and Analysis Branch or needing more information should contact:

Peggy Borbey, Director
Evaluation and Analysis Branch
Knowledge Translation Portfolio
Telephone: 613-941-4350
E-mail: peggy.borbey@cihr-irsc.gc.ca

For more information about this Evaluation Plan, program evaluation projects or client support, contact:

Laura McAuley, Manager
Evaluation Unit / Impact Assessment Unit
Evaluation and Analysis Branch
Knowledge Translation Portfolio
Telephone: 613-948-2283
E-mail: laura.mcauley@cihr-irsc.gc.ca

We will be pleased to offer our support and provide you with the information you need.

Your feedback and comments are welcome.

10. One evaluator is currently on maternity leave. She will return to her duties in March 2009.