[ PDF (233 KB) | Help ]

---

Synthesis Report of Meetings

February 23, 24, and May 18, 2005
A Framework for Measuring the Impact of Health Research
September, 2005   

Executive Summary

Table of contents

Section 1: Introduction and Background
Background to the Consultation Process

Section 2: Results of an Expert Meeting in February, 2005
International Panel Discussion
Academic Presentations
Identification and Discussion of Issues
Approaches to measuring the impact of health research
Conclusions and Recommendations:

Section 3: Results of Stakeholders Meeting May 18

Section 4: Framework for Measuring the Impact of Health Research
1. General Considerations
2. Rationale and Definitions of Key concepts
3. Dimensions
4. Methodology
5. Develop Measurement Processes and Indicators

Appendix A - Participant List
Appendix B - Participant List
Appendix 1: CIHR Institutes common logic model
References  

Section 1: Introduction and Background

Canada's federal government has invested over $13 billion since 1997 in research and innovation. Federal health research funding exceeds $1.3 billion annually at present,¹ an amount equivalent to 0.73% of federal spending in fiscal 2003-04 ($177.1 billion).²

The benefits of health research are recognized widely in public opinion surveys, which have consistently found that over 85% of the public support health research funding. The Kirby Commission and the Romanow Commission both recognized the value of health research in terms of health benefits, with the latter also pointing to the economic spin-off to the national economy as a result of health research.^3,4

Recent policy developments have highlighted the importance of demonstrating the value of investments in research and innovation.ⁱⁱ Government initiatives and documents have included:

• a cabinet commitment (Dec. 2003) to assess government investments in R&D;
• a mandate in the new position of National Science Advisor (2004) to ensure that federal research investments are delivering results;
• the Council of Science and Technology Advisors report in 2003 that called for greater accountability for federal research investments; and
• a report by the auditor general in 2004 that raised concerns about accountability in federally funded research organizations.

The funding increase allocated to CIHR in the 2004 budget specified greater investment in commercial applications of research and required CIHR and other Granting Councils to track and report on the results of funded research.

CIHR's Blueprint (2004) identified a commitment to 'Develop and implement a framework that enables the 'evaluation of the organization's performance and the value of its programs of research support'. [italics added]

The concern with documenting the results of health research investments in Canada is consistent with a broadly based international movement. This report includes a brief description of work in the UK and Australia to measure performance of research granting institutions.

The circumstances outlined above demonstrate the institutional requirement for CIHR to develop viable methods to measure its performance and to estimate the beneficial impact of the health research it funds. CIHR also requires information about potential impact to guide future investments and set priorities.

Background to the Consultation Process

During 2004 CIHR developed a project charter for an initiative to measure return on investment (ROI). The objective of the CIHR ROI project was to seek advice from leaders of Canada's health research community on:

• methods to measure returns on investment in health research by CIHR as an agency; and
• measures that can be used to establish benchmarks and gauge progress in realizing the value of health research.

The project was vetted by CIHR's Standing Committee on Performance Measurement, Evaluation and Audit (SCPMEA). Funding agencies in the UK and Australia were consulted to identify common areas of approach and determine their interest in participating. Academics who had made key contributions to the literature on measuring research impact or ROI were also contacted. A small group from the Canadian academic community and senior levels of the federal government were invited to attend a meeting in February, 2005 to review the state of knowledge about ROI in health research and to provide advice on the development of a conceptual framework to measure ROI. Two background papers were prepared to summarize relevant information in the areas of policy and methodology.^5,6

Additional work was carried out after the February meeting to develop a framework that incorporated advice from the meeting and that addresses the multi-dimensional nature of CIHR's research activities. A draft framework was reviewed at a high-level meeting of stakeholders on May 18. Broad agreement was reached at that meeting and suggestions were made to enhance the framework.

This report summarizes the February meeting in Section 2 and the May meeting in Section 3. The draft framework is described in Section 4.

Section 2: Results of an Expert Meeting in February, 2005

An expert meeting was held February 23 and 24 in Ottawa. A list of participants is contained in Appendix A. This section presents a brief synopsis of presentations at the meeting and a discussion of issues in measuring the impact of health research. One of the issues deserves to be clarified at the outset, however, in order to clarify terminology that will be used in the remainder of this report. Return on investment is a term that normally measures the future stream of net profits that flow from an investment. In the case of research, a broader definition has been suggested:

"the economic and social outcomes of innovation in terms of competitiveness, wealth creation and the quality of life. Return on investment goes beyond private sector returns from commercialization to include social benefits relating to improvements in the quality of life of citizens and the diffusion of knowledge."⁷

This definition informed discussion at the expert meeting, but participants agreed that many in the research community and public sector audiences would tend to interpret return on investment in a more traditional, financial sense. As a result, it was agreed that the term impact of health research is preferable to ROI. Impact will be defined in the CIHR framework in terms of a five-dimensional view that addresses concerns of different stakeholder communities. It includes economic returns as one of the five categories within which impact is described.ⁱⁱⁱ

There were three presentations as part of an international panel discussion and two academic presentations.

International Panel Discussion

The purpose of the international panel was to review evaluation activities of funding agencies in the UK, Australia and Canada and to gain insight about the challenges in moving from performance evaluation to measuring the impact of research investment on a national level.

UK and Wellcome Trust perspective
Wellcome Trust is an independent research funding charity. In 2003-04 Wellcome Trust and the Medical Research Council were the two largest funders of biomedical research in the UK, spending £402 million and £416 million respectively.⁸

The UK has developed a Science & Innovation Investment Framework that includes a public service agreement (PSA) and a set of PSA target metrics to 'measure the UK's relative international research performance in science and engineering.'⁹ The PSA target metrics and indicators are organized within seven thematic areas (Table 1). The indicators provide performance measures that focus largely on the scientific community. The UK Office of Science and Technology (OST) and the Organization for Cooperation and Development (OECD) are the main sources of data for the indicators.

Table 1:

Wellcome Trust uses a two step approach to evaluate its funded research activity. The first step uses a high-level, organization assessment framework that links organizational aims and objectives to indicators of progress. The second step focuses on initiative-based evaluation frameworks and measures outputs, outcomes and impacts. Examples of outcome indicators for the strategic aim of improving the knowledge base of biomedical sciences include publications and major scientific discoveries. The evaluation process includes key informant interviews, annual reports, research narratives and case studies. Case studies are considered especially useful in advocacy to demonstrate the achievements of research activities.

Australian and NHMRC perspective
The National Health and Medical Research Council (NHMRC) is a statutory body within the portfolio of the Commonwealth Minister for Health and Ageing. The NHMRC is responsible for allocating Commonwealth funds for health and medical research, providing advice on all aspects of health care policy and considering ethical issues.¹⁰ The NHMRC had a research budget of Aus $427 million in 2003-04.

In 1999 the Australian government released the Health and Medical Research Strategic Review, known as the Wills Review¹¹, which created a vision for health and medical research in Australia up to 2010. In response to the review the Australian government doubled the NHMRC budget over the next five years and specified that there should be 'significant improvements in the quality and effectiveness of health and medical research, and measurable benefits to the health and wealth of Australia.'

The NHMRC developed a performance measurement framework in order to demonstrate the positive results of its research funding (Table 2). A number of national performance measurement initiatives are underway, largely in response to an investment of $5.3 billion announced by the Australian Government in May 2004 to improve Australia's science and technology performance. NHMRC is actively participating in these initiatives.

Table 2: NHMRC Key Performance Indicators for 2003-06

Outcomes	Indicator
1. Creating new knowledge	Increased investment leading to high impact research Growth in internationally competitive knowledge
2. Enhancing capacity to innovate	Expanded and mobile research workforce Increased international recognition of Australian researchers Growth in access to and availability of facilities and equipment
3. Utilising knowledge	Increased uptake of NHMRC health advice and information Improved transfer of knowledge into health policy and practice Increased commercial activity
4. Ensuring high ethical standards	Improved support, advice and guidance on ethics issues in health and for conducting research Compliance with NHMRC ethical guidelines
5. Strengthening communications and collaborations	Growth in collaborations and partnerships Increased engagement with the community
6. Regulating embryo research and maintaining the prohibition of human cloning	An effective national system of regulation Compliance with legislation concerning embryo research and maintaining the prohibition of human cloning
7. Achieving high standards of governance and accountability	Effective planning, monitoring and reporting Effective information management Effective governance arrangements

Source: NHMRC

Australia has also had experience with initiatives to measure returns on investment in health research. These include:^iv

The Australian Society for Medical Research (ASMR) commissioned a study in 2003 by Access Economics, a private sector economic consulting firm.¹² This study extended a methodology originally developed as part of a series of papers sponsored by the Lasker Foundation in the US.¹³

The study estimated rates of return to Australian health R&D of up to $5 for every dollar invested. The study was used to lobby government for additional research investments. The study uses assumptions that were considered to be controversial.

The Australian Research Council (ARC) commissioned a study in 2003 by the Allen Consulting Group, another private sector economic consulting firm, to examine the return on investment from ARC-funded research.¹⁴ There was general consensus among government departments that participated in the study that the resulting report asked the right questions and helped to push debate surrounding return on investment forward. According to the report, the total social rate of return on ARC investment in Australia is 39% compared to an average rate of return of all publicly funded R&D of 25%. The report was described as being very focused on ARC activities and it was not clear how generalizable results were to other funding agencies.

The NHMRC undertook two studies in 2003 to provide evidence of outcomes and return on investment: an international benchmarking study and an economic evaluation of the outcomes of particular NHMRC funded research.

The international benchmarking study was inconclusive in many respects because:

• the data available for comparison purposes between countries and systems is often quite dated, making useful conclusions very difficult;
• there is a wide divergence in the way international agencies collect and report data;
• many of the OECD countries included in the study at the time did not have comparable performance indicators.

The economic evaluation of outcomes study found that it was extremely difficult to actually pinpoint the value that NHMRC funding provided to a health research endeavour through the commercialisation process. An alternative case study approach to examine successful companies and individual researchers was also unsuccessful because the respondents were unable to see any benefit flowing to them and were reluctant to cooperate.

Canada and the CIHR perspective
CIHR was established in 2000, succeeding the Medical Research Council. In 2005/06, CIHR's annual budget is $700 million. CIHR's mandate is: "To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system..." The inclusion of knowledge translation in CIHR's mandate imposes a responsibility to define the impacts of funded research in terms of economic, population and health system benefits as well as other performance measures that focus on the research community.

Developing evaluation and analysis capacity is a corporate priority. Significant steps include establishing a Standing Committee of Governing Council on Performance Measurement, Evaluation and Audit (SCPMEA) and establishing a corporate unit responsible for evaluation and analysis - both in fiscal year 2002-03.

Present evaluation activities of CIHR include:

• development of a common Institute performance evaluation framework;
• identification of key indicators (input, output, outcome);
• case studies;
• formative evaluation completed in fall 2005; and
• 5-year external review June 2006.

CIHR is developing a Management Resources and Results Structures (MRRS) Reporting Framework in conjunction with Treasury Board. This reporting structure is required by the federal government to report inputs, outputs and outcomes from CIHR's funding activities (Table 3). Indicators to measure progress in each activity are being developed.

Table 3: Summary of CIHR's Management Resources and Results Structures (MRRS) Reporting Framework

Strategic Outcome	Activities	Expected Result
1. Outstanding Research	Fund health research (Investigator initiated and strategic)	Health research supported to create health knowledge responding to opportunities and priorities
2. Outstanding Researchers in Innovative Environments	- Fund health researchers and trainees - Fund research collaboration and other grants to strengthen the health research community - National and international alliances and priority setting - Inform research, clinical practice and public policy on ethical, legal and social issues (ELSI) related to health and health research	Strong health research community able to undertake outstanding research
3. Transforming Health Research into Action	- Support activities on knowledge translation, exchange and use - Support national efforts to capture the economic value of health research advances	Health research adopted into practice, programs and policies for improved health of Canadians, a productive health system; stimulation of economic development through discovery and innovation

Source: CIHR

Common themes
There is a convergence of outcome objectives among the three agencies (Table 4). All three include knowledge creation, human resource development and knowledge translation as their three most important outcomes. All three also include ethics as key activities or outcomes. Public engagement is explicit in the strategic aims of Wellcome Trust and NHMRC, while for CIHR, it is an enabling factor for achieving its mandate (partnerships and organizational excellence are the other enabling factors). The U.S. National Institutes of Health (NIH) reports activities within three outcome areas, two of which correspond to the first two conceptual categories in Table 4: (1) research programs, (2) research training and career development and (3) research facilities. These common themes indicate that there is broad agreement on the purposes of fostering health research. They also suggest that there is considerable potential for international collaboration in developing methods to measure the impact of health research in both the academic community and within society.

Table 4: Similar Themes in Reporting Research Outcomes

CIHR Strategic Outcomes	Wellcome Trust Strategic aims	NHMRC Outcome-Output Framework
1. Outstanding Research	Knowledge base: Biomedical sciences and their impact on society.	1. Creating new knowledge
2. Outstanding Researchers in Innovative Environments	Resources: Provide researchers with infrastructure & career support	2. Enhancing capacity to innovate.
3. Transforming Health Research into Action	Translation: Ensure maximum health benefits are gained from biomedical research.	3. Utilising knowledge

There was also agreement among representatives of the three agencies about the main challenges in identifying impacts or returns to society from research. These include:

• Linkages between research outputs and outcomes in the form of health, prosperity and well-being are often difficult to trace, typically when knowledge develops incrementally over time.
• Some of the most significant impacts are intangible and difficult to measure through conventional methods - for example, improved health, longevity and a population better prepared to reach its potential as the result of improved knowledge.
• Attribution of returns to specific funding agencies or even countries can be difficult as new knowledge is typically the result of multiple researchers working on certain issues and multiple research agencies funding the research.
• Different players have different priorities. Commercial returns, for example, are more important to industry and some governments, than to researchers and the general public.
• Different conceptual approaches may be necessary for different kinds of health research.

Academic Presentations

The purpose of the academic presentations was to assist in understanding current approaches to measuring the impact of health research funding.

The 'Payback' Framework For Assessing The Impact of Health Research.
Professor Martin Buxton and Dr. Stephen Hanney; Health Economics Research Group, Brunel University, UK.

Prof. Buxton summarized the payback framework developed at the Health Economics Research Group (HERG) as "a categorisation of the types (and hence the relevant measures and indicators) of 'payback', and a conceptual framework of the way and context in which research 'payback' may occur." The model was originally developed in 1996 and has been refined in a number of applications since that time. The most recent application was a study of the returns from arthritis research carried out for the Arthritis Research Campaign (ARC), a major funder of health research in the UK.¹⁵

The Payback framework consists of a system to categorize the payback from health research and a logic model that can be applied to research projects in order to identify the payback within each category (Figure 1). The categorization of benefits is multidimensional:

• knowledge production
• research targeting, capacity and absorption
• informing policies and product development
• health and health sector benefits:
  including health gain; cost savings; service/delivery
  improvements; equity gains
• broader economic benefits

Figure 1

The payback framework has been used to assess a number of research projects, most recently 16 case studies of projects funded by the ARC. The process of assessment was described in the Executive Summary of the ARC project report (pg. xv):

"Using the information collected from document and literature reviews, semi-structured key informant interviews and bibliometric analysis, each of the 16 cases was written up as a narrative organised according to the structure provided by the payback framework. Using a common structure facilitates comparative analysis, allowing us, for example, to identify the factors associated with the successful translation of research. We employed two approaches to our cross-case analysis. The first was based on a qualitative assessment of the case studies based on a discussion within the project team^v of the key observations made by each member of the team. The second involved a novel method of scoring the case studies on the five payback categories."

Conclusions from the work are:

• a multidimensional perspective on payback is valuable and appeals to multiple stakeholders;
• a logic model helps focus thinking, provides consistency and need not be restricting;
• payback assessment is feasible with relatively modest resources; and
• formal analysis of payback can begin to address research management questions and it produces the illustrative 'good news' stories that can be very influential.

In comments during the presentation, Prof. Buxton stated that he considered the framework to be a useful way to look at a body of research and conceptualize it. The payback model was not intended to be an economic return on investment approach, although it includes a dimension that can capture the broader economic benefits of research. Problematic issues in assessing payback include long time lags between research, adoption of results and final outcomes; problems of attribution where there are numerous incremental advances in knowledge; and the need to apply judgment in assessing payback. One of the key issues in any attempt to measure the impact of research is the unobserved counterfactual - we do not know what would have happened if the knowledge produced by research had not been made available.

For Better or Worse: ROI from Medical R&D
David M. Cutler; Harvard University
Dr. Cutler has carried out extensive research on the effects of health research on prevention, treatment and health care costs. His presentation was based on a recently published book, Your Money or Your Life.¹⁶

Health research in the US accounts for less than 5% of total health spending, well below most high-technology industries where research costs are usually about 10% - 15% of sales. Medical R&D results in new treatments, which add costs to health care but provide options for treatment that often extend lives or improve quality of life. Dr. Cutler cited advances in treatment for cardiovascular disease (CVD), treatment of low birth weight infants and treatment of mental illness as areas where very notable advances have been made as the result of research.

Dr. Cutler estimated the present value of the total lifetime costs of treating CVD as $30,000 per capita for persons aged 45 in the US. Average life expectancy for persons aged 45 in the US population has increased by 4.5 years since 1950 due to reductions in death from CVD. Improved treatment is responsible for approximately two-thirds of this gain in longevity, or a gain of 3 years of life. The remaining one-third is due to behavourial changes, which have been influenced by advances in knowledge resulting from research, such as reductions in smoking rates. Comparing benefits in the form of improved longevity to lifetime costs produces an estimate of $10,000 cost per life year gained. He asked the panel for threshold values used to determine upper limits for cost-effectiveness of new treatments in their respective countries (usually expressed as cost per quality adjusted life year (QALY)). Panel members estimated $10,000 to $50,000 in Canada and £30,000 in the UK. Dr. Cutler estimated $100,000 in the US.

Estimates of economic benefits and costs indicate a ratio of 4 to 1 for CVD treatment. Cost-benefit ratios would range up to 7.1 to 1 in the case of heart attacks and depression (Figure 2).^vi In another recent publication Dr. Cutler estimated the cost-effectiveness of behaviourial change in reducing deaths from CVD as 30 to 1.¹⁷

Figure 2:

Source: David M. Cutler, 2005.

Dr. Cutler indicated that 'two-thirds of Americans rank health care as a top item for an expanding economy.' The population is aware that over-treatment in some instances results in lower efficiency than possible. He stated that as a cultural value, however, Americans are prepared to tolerate waste in order to have access to treatment when needed.

Identification and Discussion of Issues

A number of issues were identified and discussed during the panel meetings. This section briefly reports the opinions expressed around major issues.

Incorporating Efficiency Concerns in Benefit-cost Analyses of New Treatments.
Dr. Cutler's benefit-cost calculations raised issues about the way in which new treatments are evaluated. Medical interventions may be either cost-saving or cost-increasing. Laprascopic surgery for gall bladders is much less expensive than invasive surgery, for example. Cost per QALY is an accepted approach to comparing relative costs of treatment. This approach to evaluating health research was compared to an investment portfolio in which investment decisions favour alternatives that promise the greatest rate of return, taking into account the risks associated with uncertainty. There is a concern, however, that such an approach might disadvantage minority groups with complex health problems.

Questions were raised regarding the diffusion of new technology and the value at the margin of extending certain treatment to persons with relatively mild severity. This problem is exacerbated in health care systems by the fact that persons who are candidates for treatment often are not in a position to make informed judgments about its advisability, and providers who do have the necessary information are often subject to financial incentives that favour treatment (especially in fee-for-service practice).

The question of how to value health services research was also raised. It was noted that some forms of research produce knowledge or technology outputs that can be adopted quickly while other types of research, such as health services and policy, which are more focused on efficiency, may evolve more slowly.

The issue of how gains in life expectancy have been used in advocating increased research funding was raised. Work commissioned by the Lasker Foundation in the US, and replicated by the Australian Society for Medical Research, concludes that major uncounted increases in social value have been realized by life-extending health advances. One estimate concluded that the value per capita of uncounted increases in personal income (utility) in the US from gains in longevity were double or triple the amounts of increase in health expenditures per capita between 1980 and 1990 and close to the average increase in non-health personal consumption during that period.¹⁸

Dr. Cutler pointed out that the Lasker Foundation approach was to emphasize the value of research and development rather than the costs of health care. Canadian economists at the meeting expressed concerns about the need to focus on efficiency. They thought that approaches that stressed the value of research might be used to justify wastage or over-treatment in present health systems.

The discussion around this issue illustrates a dilemma in measuring returns on investment in health research. Important gains to society are often intangible and difficult to trace to specific research knowledge. Gains in treating CVD, for example, have been attributed by Dr. Cutler to three relatively broad categories: invasive treatments, pharmacotherapy and behavioural change. Skepticism about the efficiency with which the first two factors are used at the margin in health care systems is reasonable if the objective is to do better. But will it be possible to develop an evaluation methodology that measures the beneficial impact of health research in shaping modern health care systems and also identifies prospects for future gains in efficiency?^vii

Spending on health research must be balanced with other priorities for public spending.
Work commissioned by the Lasker Foundation (cited above) has argued that the value of health research, in terms of increased life expectancy of the US population, can be estimated at about two trillion dollars using methodologies developed by prominent economists to assign a value to years of life. Approaches such as these suggest that, in the words on one participant, 'we are so far from the cost-benefit frontier in health research that almost any additional investments can be justified'.

Participants agreed that although arguments based on the value of life have merit, additional public investments in health research are limited by budget constraints and the need for government to balance spending among many important public priorities. It is also important to realize that most health research results in 'public goods' in the form of treatments or prevention strategies that are available to the health system and the population without financial return to those who fund the research. Where monetary returns are possible, there will be a greater tendency for the private sector to invest - which explains why most corporate research investments are in the form of R&D for patentable drugs or equipment. Even then, many point out that patented products often are possible only as the result of earlier research funded by government or charitable organizations.

Do government attitudes favouring commercialization threaten basic research values?
Concern was expressed that emphasis on commercial applications of research has the potential to tilt attitudes towards a market based approach that does not adequately consider risk or spillover effects. The precautionary approach, which has long been the standard in health care, is in danger of being supplanted by an approach that uses risk assessment in decisions about whether or not to approve new treatments that have potentially dangerous side effects. Recent experiences with drugs such as Vioxx illustrate that side effects can have negative impacts for both society and drug manufacturers.

Research is multi-faceted and impacts should be evaluated along different dimensions.
The panel reached agreement on this issue. It has several dimensions for CIHR.

1. CIHR's mandate is complex and covers goals such as excellence by scientific standards, the creation of new knowledge, knowledge translation leading to improved health, more effective health services and products and a strengthened Canadian health care system. Each of these dimensions may require specific evaluation criteria.
2. CIHR organizes activities around four research themes, which may require different approaches to measuring impact or return on investment.
   • biomedical research;
   • clinical research;
   • health services and policy; and
   • population and public health.
3. CIHR funds both investigator-initiated research (70% of its research investments) and strategic research into areas identified as high priority for health systems (30% of research investments). Examples of strategic research include assembling a team to sequence the SARS genome during the epidemic of 2004 and a project to define acceptable wait times for certain conditions (which is presently in the request for assistance phase). Methods to evaluate strategic research may need to be more specifically focused on achieving the purpose of the research than methods to evaluate investigator-initiated research, which can have unforeseen beneficial effects over a period of years

Approaches to measuring the impact of health research

The methodology background paper prepared for the meeting identified three broad categories to measure the impact of health research.

1. Achievement of national or institutional goals, which include health research goals adopted by CIHR in its strategic plans, academic achievement and goals set by government for Canada's standing in the global research forum.
2. Societal value of health research, which potentially can be determined by measuring the contributions of research to the achievement of health goals. Improvements in targeted health status indicators and reduction of death rates from major health threats are examples of such goals.
3. Economic measures, including traditional concepts such as the value of health care costs avoided, reduction of productivity losses due to illness or accidents, commercial value of research innovations and the intrinsic value of added years and enhanced quality of life.

The Buxton and Hanney (HERG) Payback approach uses five categories to define research impact:

1. Knowledge production
2. Research targeting and capacity building
3. Informing policy and product development
4. Health and health sector benefits
5. Wider economic benefits

A third and closely related approach to evaluating the impact of health research was suggested by Dr. Renaldo Battista:

1. Self-referential measures, which use measures of research outputs that are specific to the scientific community, such as numbers of researchers trained, scientific publications and citations.
2. Instrumental effects on decision-making, medical treatments or health indicators.
3. Cultural shift, which describes the ways in which research contributes to a knowledge-based society.

This approach embodies an expanding circle of influence that includes dimensions for both the audience for research results and the specific outputs and outcomes for health research. Members of the expert group suggested the approach could be described in terms of concentric circles expanding outward and that a series of sub-divisions could be defined depending on the area of focus (e.g. university, funding agency, medical care sector, administrative policy and the role of scientific insight in cultural orientation).

Intellectual similarities exist in the three approaches. At this stage of the conceptual framework development, participants favoured the five dimensions in the Payback approach to clarify definitions of how impact should be measured. As mentioned in the introduction, participants thought it better to use the term 'impact' rather than the phrase, 'return on investment' for purposes of clarity with respect to what should be measured in evaluations of health research.

The Payback categories can be seen as a generic set of definitions that contribute to the language of evaluation. Dr. Buxton and his colleagues have also developed a specific methodology for evaluating research and the definitions have been refined through use in their evaluation activities. The expert group thought that CIHR could use the definitions in conjunction with a number of methodologies that may be required in order to measure impacts relative to its mandate. Minor changes can be made to category definitions, where necessary, to accommodate CIHR's requirements.

Conclusions and Recommendations:

The meeting reached several conclusions, which are included in the Methodology section of the draft framework. The meeting made recommendations for an approach to developing methods for measuring the impact of health research but it did not produce recommendations for specific measures of health research impact.

Recommendations for the development of methods were:

• A variety of approaches should be used as appropriate for subject area and stakeholder concerns. These will include case studies or narratives and indicators of achievement in specific areas defined by the five dimensional impact categories.
• Work on the Burden of disease methodology should be strengthened. Key participants in developing this methodology in Canada are Health Canada, Statistics Canada and the Canadian Institute for Health Information.

CIHR continued work on a framework for measuring health research impact after the February meeting. The draft framework was presented to a meeting of stakeholders on May 18. The results of the May meeting are described in the following section.

Section 3: Results of Stakeholders Meeting May 18

A meeting of stakeholders from government, academic and health research sectors was held in Ottawa, May 18. A list of participants is contained in Appendix B. Objectives for the meeting were:

1. Develop among health research stakeholders and opinion leaders a common understanding of the range of potential impacts of health research;
2. Refine the draft framework for measuring the impact of health research;
3. Discuss next steps to implement this framework.

The meeting featured a presentation of the draft framework followed by a general discussion of its content, which included advice for CIHR in taking the framework forward and opportunities for collaboration between CIHR and stakeholder groups.

Framework Content

Suggestions about content covered several topics, including the boundaries of the framework itself and the content of specific categories for measuring impact. Key issues and suggestions are summarized below.

Are specific impacts positive or otherwise?
The question of whether or not commercialization of discoveries should be a goal was raised. It was pointed out that economic value depends on the nature of products produced and not on the fact that jobs are created within national economies. A closely related point was raised about the role of research and development (R&D) in keeping certain drugs under patent and whether this strategy by pharmaceutical companies represented a negative impact in the form of protection against competition, which in turn leads to higher prices.

With respect to the first concern, the framework is being developed within current value systems. The federal government considers an increase in commercial activity to be an important outcome of Canada's innovation strategy. An impact framework that measures commercialization in terms of commonly accepted criteria, such as jobs created, value added to GDP and contributions to Canada's reputation as an innovative society, will address important public goals. It may not be feasible to evaluate individual examples of new commercial activity, for reasons that include the uncertainty of long-term impacts and the current state of knowledge about commercialization, which is in an early stage of development.^viii

With respect to the issue of pharmaceutical R&D, the framework is being developed to measure the impact of research funded by CIHR. While this research includes pharmaceutical products and medical devices, it is normally basic research carried out with a view to discovery. The difference between basic research and product R&D is recognized within the research community, although the boundaries may be controversial in some instances. The nature of the peer review processes for CIHR funded research tends to guarantee that funded biomedical and clinical services research will be almost exclusively basic research.

While the points discussed above will be relevant to discussions of the framework as it evolves over time, they should not delay progress in the development of methods to measure impact. It was noted that methodology requirements built into the framework will attempt to distinguish between social rates of return and commercial profits in cases where it is appropriate to do so. A related consideration is that a variety of approaches will be used to identify impact, including qualitative evaluation as well as quantitative measurement. Qualitative evaluations will be especially important to measure progress in achieving CIHR's mandate of improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

General impacts on scientific processes
Participants pointed out that the research environment plays an important role in stimulating high quality research. Knowledge accumulation drives the system of research and academic excellence. These considerations argue for a comprehensive view of the importance of high quality research as an academic value, and of public support for this value system.

Impacts on health policy
The issue of how to influence public policy was discussed. One participant expressed the opinion that political cycles have an unpredictable effect on government decision-making. The federal Finance Department representative pointed out that budget cycles were very important considerations in decision making. Since investments in research must be defended within the entire range of budgetary responsibilities, it is important to have appropriate information in the fall of one year in order to influence budgetary support in the following spring. He also pointed out that budget documents require CIHR to demonstrate results in terms of two criteria: excellence in research and commercialization of research discoveries. CIHR accountability in terms of research excellence is important to government as well as the scientific community. Participants agreed that research impact information would be an important factor in evidence-based decision making in the area of health and health research funding policy.

Some goals may conflict
The draft framework identifies cost savings and efficiency as potential benefits of health research. Participants pointed out that many research discoveries have the potential to increase treatment costs. While this often will be the case with clinical services, it is important to recognize other considerations that add complexity to calculations of cost-effectiveness:

• Some new treatments are genuinely less expensive than existing therapies - for example techniques that replace major surgery or that avoid periods of in-hospital treatment.
• Population health and health services research can lead to a greater role for prevention or identify opportunities to reorganize delivery systems in ways that are more efficient than present systems.
• Treatment costs should be balanced against human capital gains wherever possible. One participant expressed the opinion that increases in life years gained and increases in productivity were related outcomes of our health system and that they should be linked in the framework if possible.

Performance measurement and research impact
There are two separate issues to be addressed in evaluation of CIHR's activities:

1. Evaluate performance in terms of corporate objectives.
2. Understand how research contributes to health, social and economic progress.

There is an ongoing process of performance evaluation at CIHR, which includes a five-year review by an external panel of experts that will be completed in 2006. The research impact framework incorporates that work and expands it to include the second issue. The contribution of research to health, social and economic progress extends beyond the activities of CIHR to include health research sponsored by other funding agencies and by the higher education sector.

Audience concerns
The framework identifies five different audience groups and lists concerns for health research that are germane to each group (see Table F1 of the Framework). In discussions of audience concerns it was pointed out that contributions to international health and recognized excellence in Canadian research were important values for most of the audience segments (these items have been added to the Framework).

Suggestions for Impact Categories
Indicators are identified in the framework and grouped within the five categories used to measure impact (see Table F3 of the Framework). The following suggestions were made (these suggestions have been implemented in the framework as revised for this report).

1. Include prevention in the category of health sector benefits and highlight its importance.
2. Clearly separate impacts on individual and population health from impacts on the health care system.

Ways to move forward
One participant pointed out the importance of a balanced and flexible framework. He suggested that the robustness of the framework could be tested by attempting to apply it to each of CIHR's four research themes (pillars).^ix

The timeframes within which impact should be measured were debated. The framework indicators and options indicate that three and five year evaluations of CIHR performance would be key timeframes for impact measurement. Participants pointed out that some activities, such as bibliometric analyses, should be an ongoing process since they contribute to appropriate targeting of future research and to documenting the present state of knowledge - both important considerations in funding new research.

The identification of long term impacts was also discussed. The relevant timeframe for realizing the full impact of a particular research achievement varies considerably and it may be necessary to consider the time dimension on a case by case basis. Participants did not agree on an appropriate timeframe for retrospective case studies, although the value of retrospective analysis was recognized. One participant pointed out that it was not enough to just measure impact in retrospective studies but that the state of knowledge and research values at the time the research was initiated were important contextual considerations.

The feasibility of creating packages of impact measurements for each audience identified in the framework was questioned. CIHR was advised to concentrate on key impact areas and avoid trying to do too many things, which might create risks for successful implementation.

One participant expressed a concern that most case studies or anecdotes concentrate on research that has had positive results.On the other hand, it was pointed out that a random sampling approach could miss research projects that have had highly significant impacts on prevention or treatment. Sampling from an entire program of research is important to test if methodologies are sound - for example, we need to know if a finding of no impact is a fair assessment of the research in question or if the methodology was not sensitive enough to measure impacts that did materialize. Research is not expected to bear fruit in each investigation, however. Curiosity driven research may settle a question or contribute to existing knowledge without having a significant impact that can be measured within the framework. Yet the successes can have a huge impact. Dr. Bernstein, CIHR's CEO, noted that it might be necessary for a funding organization to fund 990 grants without major impact in order to 'find ten that change the world.'

Several participants suggested a process of testing indicators to determine their robustness and to identify sources of information. The extent to which indicators could be influenced by CIHR activities is also an important issue as some of the indicators identified in the framework are subject to many influences. In some instances it may be necessary to commission research into the indicators themselves in order to clarify their value and importance.

Collaboration
There was broad agreement on the value of collaboration in taking the framework forward. All research agencies in attendance expressed agreement with the conceptual design and content of the framework and volunteered to work with CIHR to refine and test it. There was general agreement on the importance of testing indicators and of adopting a flexible and multi-dimensional approach to the framework development. One provincial agency volunteered to begin testing during the summer if desired. Another provincial agency stated that it has staff working on the issues of impact and indicators and that it would be willing to work with CIHR to these ends. One participant stressed that it would be important to 'get started' on the process of measuring impact and suggested a series of retrospective case studies. It was also acknowledged that it will be important to 'get it right' and to achieve buy-in from stakeholders, and that refinements and measurement will likely take some time to come to fruition.

Summary
Dr. Cy Frank, Scientific Director of CIHR's Institute of Musculoskeletal Health and Arthritis, summed up the day's results on behalf of CIHR. He noted that there was agreement among those present that the framework was appropriately designed and that it is a useful vehicle for measuring the impact of health research. The Framework document reviewed at the meeting provided a good starting point for further development and the meeting had produced valuable suggestions for refinements and filling gaps. The next challenge would be to develop realistic indicators that can produce the desired insights into research impact. CIHR can work with its partners in this endeavour. There is an opportunity for Canada to be a leader in the international community in measuring the impact of health research.

The action plan for future development will be to incorporate revisions agreed to at the meeting and then have the Framework document reviewed by the SCPMEA. Following this review, CIHR will work with other stakeholders to develop indicators. On behalf of CIHR, Dr. Frank stated that the meeting had been a 'huge success'. He and Dr. Bernstein thanked all of those who had participated.

Section 4: Framework for Measuring the Impact of Health Research

1. General Considerations

The CIHR Framework for measuring the impact of health research includes the rationale for undertaking activities to evaluate health research, definitions of key concepts, identification of the audience for impact information and the types of information required to address audience concerns. The methodology to be used in measuring impact is described, including the CIHR Institutes Common Evaluation Logic Model, categories to be used to describe impact and a process for developing indicators within each of the impact categories.

2. Rationale and Definitions of Key concepts

The rationale for measuring the impact of health research is to enhance understanding of how health research contributes to social and economic progress.

Definitions:
Health Research: CIHR adopts a broad approach to health research built around four research themes^x:

Biomedical research:
Research with the goal of understanding normal and abnormal human functioning, at the molecular, cellular, organ system and whole body levels, including development of tools and techniques to be applied for this purpose; developing new therapies or devices that improve health or the quality of life of individuals, up to the point where they are tested on human subjects. Studies on human subjects that do not have a diagnostic or therapeutic orientation.

Clinical research:
Research with the goal of improving the diagnosis, and treatment (including rehabilitation and palliation), of disease and injury; improving the health and quality of life of individuals as they pass through normal life stages. Research on, or for the treatment of, patients.

Health services and policy research:
Research with the goal of improving the efficiency and effectiveness of health professionals and the health care system, through changes to practice and policy. Health services research is a multidisciplinary field of scientific investigation that studies how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviours affect access to health care, the quality and cost of health care, and, ultimately, Canadians' health and well-being.

Population and public health research:
Research with the goal of improving the health of the Canadian population, or of defined sub-populations, through a better understanding of the ways in which social, cultural, environmental, occupational and economic factors determine health status. Output: The actual products or services created. In the context of evaluating health research, outputs would usually be expressed in terms of new knowledge, techniques for treatment or products such as pharmaceuticals or devices.

Outcome:
The key results of an initiative (can be short or longer term). For example, a new medical device may be the output of a specific research initiative. The outcome may be more efficient diagnosis or treatment of a disease.

Impacts:
In the context of evaluating health research, the overall results of all the effects that a body of research has on society. Impact includes outputs and outcomes, and may also include additional contributions to the health sector or to society. Impact includes effects that may not have been part of the research objectives, such as contributions to a knowledge based society or to economic growth.

Performance Measurement:
Collection and monitoring of information relating to the ongoing performance of an organization.

Evaluation:
A systematic assessment of policies, programs or initiatives to determine success in meeting objectives, including the interests of its stakeholders.

Human Capital:
Human capital refers to the achievement of economic potential by individuals. In the context of capacity building, human capital refers to academic training and the development of specialized skills.

Goal Oriented Research:
Research projects designed to further knowledge about the protection and promotion of health. Includes both investigator-initiated research and strategic research where topics are identified by funding agencies.

Knowledge Translation:
CIHR defines knowledge translation as the exchange, synthesis and ethically-sound application of knowledge - within a complex system of interactions among researchers and users - to accelerate the capture of the benefits of research for Canadians through improved health, more effective services and products, and a strengthened health care system. This broad definition can include for example, the translation of health research results into forms that will influence decision-making in the health policy or medical practice sectors, or the development of commercial products from health research.

Payback Model Categories to Measure the Impact of Health Research ^xi:

Knowledge production:
The contributions to knowledge from a research project or a body of research involving multiple projects. Knowledge production is usually measured through contributions to scientific publications and patents or invited presentations (e.g. conferences) but includes knowledge fed more directly to users through commissioned reports etc.

Research targeting and research capacity:
Benefits to future research activity.  This includes the use of research information to improve targeting of future research; individual and group development of research skills and research capacity; development of the capability to use existing national or international research.

Informing policy and product development:
Clinical and administrative benefits, including the development of informed information bases upon which to make decisions, and the application of research findings in policy development (at all levels of policy). Includes development of clinical practice guidelines. Benefits for product or process development where research findings feed into commercial decisions and developments. (This category has been modified in the CIHR framework, where product development and commercialization of research findings will be included in the category, Economic benefits).

Health and health sector benefits:
Improvements in life expectancy and quality of life through advances in prevention, diagnosis or treatment  made possible by research. Increased efficiency of service organisation. Improved equity in the health sector. (This category has been modified in the CIHR framework, where efficiency in the form of cost-effectiveness will be included in the category, Economic benefits).

Broader economic benefits:
Benefits to the economy that result from health research. These benefits can include economic returns from commercialization and contributions to the economy from improvements in workforce health. (This category has been modified in the CIHR framework, where it is called 'Economic benefits' and includes all economic impacts).

3. Dimensions

Audience for the framework indicators
Audience is a key dimension of the framework structure. The types of information considered to be most relevant to the interests of each community within the audience are identified in Table F1. CIHR can use impact data to assist in fulfilling its responsibilities for accountability to the stakeholder communities listed. There are also overarching concerns that are not associated with any single group. These concerns include international health and the importance of excellence in Canada's research achievements.

Table F1 Cross Classification of Audience for Impact Information, Concerns and Types of Information

Audience	Concerns	Type of Information
Higher education sector	- Academic Excellence - Knowledge production - Capacity building	Outcomes measurement criteria - e.g. indicators being developed for CIHR performance measurement.
Health professionals & administrators	- New treatments and diagnostic potential. - Productivity of resources used in health system.	- Biomedical and health services research achievements. - Population health indicators.
Society	- Improved health status. - Response to public health threats. - Efficiency and sustainability	- Improvements in key health status indicators (e.g. life expectancy, infant mortality). - Health expenditures.
Business sector	- Commercial potential	- Number of firms created to market innovations. - Net present value of commercialized research outputs.xii - International markets served (exports)
Government	- Public health and responses to health threats. - Health status. - Contribution to macroeconomic growth and productivity. - Efficiency and sustainability of both public and private health systems.	- Value realized for health research funding. - Most of the indicators identified for other communities are relevant.

CIHR requirements
CIHR requirements include the need to demonstrate that the agency is fulfilling its mandate and to provide accountability for the use of public funds. The two main requirements can be summarized as:

• justify research funding;
• assist planning and inform funding allocation decisions;

play a significant role in the international movement to measure impact of health research investments. Affordability is also an issue, since CIHR must balance its administrative costs and its research investments.

4. Methodology

Methodology Requirements
Based on decisions at the Feb. 23, 24 meeting, methodologies for measuring the impact of health research should meet the following requirements.

• Methodologies will need to address CIHR's mandate in specific terms.
• New methodologies should build on existing performance measurement work. This will produce a continuum between the two types of activity.
• A variety of approaches and measures are required to address CIHR's four research themes (pillars), strategic and investigator-initiated research.
• Methodologies should consider short term and long term impacts of research.
• Where appropriate, methodologies should distinguish between social rates of return and commercial profits - e.g. some innovations could have positive effects in one dimension and negative effects in another, while others could have positive effects in both.
• Even though different approaches are adopted for different aspects of CIHR's activities, it will be important to map the results back to a common conceptual framework.
• It will be important to view CIHR as part of a knowledge system. Links with others that have similar or complementary roles should be created and nurtured.
• It is important to distinguish if the Impact work will create a snapshot in time or be a continuous process. CIHR regards it as a continuous process.
• Involvement of other research funding agencies, both domestic and international, will be desirable in order to maximize insight and achieve efficiencies available by pooling efforts.

CIHR Common Evaluation Framework and Impact Categories

The Buxton-Hanney Payback categories will be used to guide the classifications of impact. Minor modification of the categories have been made to assist in classifying impacts relevant to CIHR and the Canadian research milieu.

CIHR's 13 institutes have developed a Common Performance Measurement and Evaluation Framework. The Common Evaluation Framework includes a logic model (Appendix 1) that describes inputs, institute activities, outputs, enabling outcomes and core outcomes. These outcomes contribute to a set of ultimate impacts, which describe CIHR's success in achieving its objectives within the scientific community. The logic model also includes three overall goals for health and the health sector. These goals are contained in CIHR's mandate.

The core outcomes are the focus for CIHR performance measurement in the Common Evaluation Framework. The Research Impact Framework will include the performance measurement work and, in addition, will attempt to measure progress toward achievement of the overall goals.

Figure F1 shows how the logic model categories relate to the impact categories that will be used in the Research Impact Framework. The Common Evaluation logic model and the Research Impact Framework can be aligned by expanding the logic model to include economic and social impacts, which go beyond the health sector.

Figure F1

Relationship between CIHR's Common Performance Measurement and Evaluation Framework and the Health Research Impact Framework

Measures of Impact
Knowledge production is usually measured through an analysis of contributions to the scientific literature (bibliographic analysis). A number of countries use bibliometric analysis to monitor the output of scientific publications and citations by their researchers. A recent study shows that while Canada ranks sixth among G8 countries in its share of total citations, it ranks fourth in the field of preclinical medicine and health.¹⁹

Research targeting and capacity comprises two types of impact. Capacity is usually measured through the number of highly skilled researchers produced in a country. Number of PhDs is a commonly used measure. In Canada, we can also measure the impact of the Canadian Research Chairs (CRC) program to determine how many researchers are being attracted or trained for health disciplines. Research targeting will seek to define how the outputs of specific research programs inform subsequent research agendas, lead to the development of new research questions or practices, and contribute to an incremental process of knowledge production. Special studies, possibly including surveys of researchers and/or successful funding grant recipients will be required to address this issue.

Informing policy is an adaptation from the category, 'Informing policy and product development' in the Payback Model categories. In its modified form this category concentrates on policy impacts at all levels, including clinical, administrative and government policy. The category includes indicators such as the number of clinical practice guidelines resulting from research. Qualitative measures, such as the use of research knowledge in clinical or administrative decision making, can also be used to measure success in this dimension of knowledge translation.

Health and health sector benefits includes impacts on health status through advances in prevention, public health and patient treatment. Case studies and periodic CIHR evaluations can address these issues. Documentation of health-preserving innovations and their use in clinical treatment will be another means of measuring progress. Progress through time in improving health status can be tracked by using specific measures of health impact that are related to health research. As a first measure we propose the reduction in potential years of life lost (PYLL) for specific high priority disease groups. Examples of high priority diseases are heart disease and cancer - CIHR currently has research investments of approximately $100 million in each. While it is usually difficult to attribute reductions in mortality to specific research initiatives, it will be possible to compare progress in developing treatments for specific diseases with reductions in mortality from those diseases.

Economic benefits The category, 'Economic benefits' is an adaptation from 'Wider economic benefits' in the Payback Model categories. This adaptation will consolidate indicators of economic value within a single category. It also includes approaches that have been used by other institutions or in academic research, including proposed methods to measure financial return on investment.

A recent literature review by Buxton et al classified the economic measures into four broad categories.²⁰ Table F2 lists each method and summarizes opinions expressed at the meeting of experts in February about the feasibility of using each approach in the impact framework. It should be noted that the discussions of these methods were not extensive and the opinions noted do not constitute conclusions.

Table F2

Expert Opinion about the Feasibility of Approaches to Measuring the Economic Value of Health Research

Economic Measure	Feasibility of the approach for CIHR Impact Framework
Commercialization Increases in commercial activity as a result of new innovations and products.	Government requirement which needs to be addressed.
Direct cost savings: Savings in health care and related expenditures as a result of new treatments.	Qualified agreement
Human capital Gains to the economy from a healthier workforce.	Endorsed more rigorous work on Burden of Disease estimates
Value of life and health Assigns value to life years saved as the result of new treatments.	This approach was not considered to be an area of priority. Concerns were expressed about its ability to measure efficiency gains or losses.

Commercialization
The first priority in measuring economic benefits will be to establish estimates of the contribution of CIHR funded research to commercialization of research discoveries. Statistics Canada carries out periodic surveys of commercialization in the higher education sector. Preliminary results from the latest survey show that institutions for higher education and hospitals received $51 million in income from IP commercialization in 2003 and held $52 million in equity in publicly traded spin-off companies.²¹ The survey does not break down income by field of study, but it does provide field of study for number of patents issued in the survey year and total patents held. Information from the survey is also used by the Association of Universities and Colleges of Canada (AUCC).

Economic returns to patent holders are just one component of economic returns from commercialization of research discoveries. Value added by spin-off firms and by private sector firms that purchase the rights to use IP are not measured across the economy. Statistics Canada held two meetings on commercialization measurement during 2004.^22,23 At present there is no comprehensive framework for economy wide measurement of commercialization, but strategies for the future are being considered.

The performance of biotechnology firms is a highly aggregate indicator of the importance of research discoveries in economic activity. CIHR is one of several participants in a project to develop biotechnology statistics, which is funded by the Canadian Biotechnology Strategy. Statistics Canada has developed a Biotechnology Use and Development Survey as part of this project. Preliminary results show that there were 262 innovative biotechnology firms in the human health sector in 2003.^xiii These firms had 9,194 employees with biotechnology-related activities and revenues of $2 billion.²⁴

CIHR can undertake discussions with Statistics Canada and the AUCC to clarify the contribution of health research to commercial returns on investment. Special studies and follow-up surveys of participants in The Small and Medium-Sized Enterprise Research Program (SME) and researchers who have received CIHR Proof of Principle (POP) grants can also be carried out. The SME is a partnership between CIHR and biotechnology companies targeted to university spin-off companies and new commercial ventures by researchers. The POP program supports university based researchers to establish the marketability of an invention or discovery and then move it toward commercial viability.

Direct cost savings
Health technology assessments have the potential to document savings from the use of specific technologies in specific jurisdictions. Jacob and McGregor (1997)²⁵ have measured the financial and policy impact of specific health technology assessments in Quebec. Lehoux et al (2000)²⁶ have examined the pathways through which health technology assessments affect health decision making. The Canadian Coordinatinating Office for Health Technology Assessment (CCOHTA) provides a national focus for the study of cost-effectiveness of new technologies and medications. Collaboration with CCOHTA to define the impact and cost savings from relevant health research can be pursued.

Human capital
The human capital approach seeks to measure the economic value of improvements in workforce health. Work underway to measure the burden of disease in Canada, which includes estimates of the costs of lost production, may provide baseline data to measure human capital gains that result from health research.

Value of life and health
The impact of health research on broad economic measures, such as GDP growth, is difficult to measure with any degree of certainty. GDP is affected by many factors and there is a well established system in place, in the form of national income and expenditure accounts, to classify and measure the economic determinants of GDP growth. The contributions of indirect factors, such as a healthy population, are not measured in present accounting systems. Some analysts have argued that GDP should be broadened to include estimates of the value of life and health, but such an undertaking would be beyond the scope of the Health Research Impact Framework.

Social Benefits and Well-Being
Social benefits are not included as a separate impact category, but are important outcomes within the last two categories (health and health sector benefits & economic benefits). The literature on broader measures of social benefits, such as equity and social cohesion, will be monitored for possible adaptation of findings to the impact framework.

Two recent studies for the Advisory Council on Science and Technology acknowledge the importance of health and economic progress as indicators of well-being and social benefits. A study by Sharpe and Smith (2005), identified five dimensions through which well-being could be defined: economic, environmental, health, social and cultural.²⁷ The report documented 38 national and international systems to measure well-being and social progress. Health status indicators were included in most systems. The authors concluded that it was feasible to measure contributions of research to well-being, but cautioned that it was important to define the dimensions of well-being and the type of research investments that are relevant to them.

Torjman and Minns (2005) have developed a sustainable development framework for the use of social indicators in science and technology research.²⁸ Health is a component of the social dimension in this framework. Their analysis demonstrates that certain indicators serve a dual purpose in terms of measuring progress toward the achievement of social objectives and other objectives. Indicators of health and economic progress are both examples.

Indicators for each impact category
A number of indicators have been defined within CIHR's Common Evaluation Framework and within the Management, Resources and Results Structure (MRRS) and plans are being put in place to collect the required data. Other indicators are being considered for future use. An example of indicators within each of the research impact categories is provided in Table F3.

Table F3
Indicators of Health Research Impact and Potential Sources of Information

Knowledge Production
1. Number of publications resulting from CIHR-supported research. 2. High peer review rankings of results of CIHR-funded research.		Bibliometric Studies
Research Targeting and Capacity
1. Extent to which Institutes have appropriately influenced the research, policy and/or practice agendas in their communities. 2. Percentage of Research Chairholders attracted or retained in Canada due to the CRC program.		Evaluations every 3 - 5 years. CRC database.
Informing Policy
1.Number of public policies influenced by ethical legal social issue (ELSI) principles. 2. Number of clinical practice guidelines by disease area influenced by CIHR funded research. (N)		Case studies. Evaluations every 3 - 5 years.
Health and Health Sector Benefits
Public health: Strategic research initiatives and their outcomes. (N) Health impacts: Impact of health research on PYLL for target disease categories (e.g. cancer, circulatory disease) (N)		Case studies Statistics Canada data. Special studies to establish links to health research.
Economic Impacts
Commercialization: 1. Number and nature of patents, spin-off companies and licenses for intellectual property (IP) generated from CIHR funded research. 2. Income from IP commercialization. 3. Case studies and follow-up surveys of commercial use of research funded by CIHR's Proof of Principle program.		Statistics Canada Statistics Canada Special studies
Cost savings: Estimates of the value of high impact innovations developed through health research in Canada (N)		Special studies
Human capital: Reduction in productivity lost through illness or injury due to innovations from research. (N)		Collaborative studies with Health Canada and Statistics Canada

Note: Most indicators are incorporated in CIHR's draft MRRS. Indicators followed by (N) are new.

5. Develop Measurement Processes and Indicators

There are many audiences for impact measures and many dimensions to be measured in a program as complex as CIHR's. We are proposing to apply multiple methods, which will produce a series of impact assessments to illustrate the full scope and diversity of CIHR's contributions to the well-being of Canadians. This will be an ongoing process with phased development of methods and measures of health research impacts.

Options
The options outlined below can be used as appropriate for subject area or stakeholder concerns.

1. Evaluations carried out every 3 to 5 years as part of CIHR's ongoing evaluation strategy. These evaluations will include bibliometric analyses, and external reviews.
   Strengths: The evaluations are comprehensive, funding is in place and continuity is assured.
   Limitations: Impacts are focused on the scientific community.
2. Case studies to describe the impact of health research in specific areas. The Payback Model methodology or other narrative approaches will be considered for appropriateness to specific studies.
   Strengths: Case studies can generate powerful stories of impact for several audiences and provide useful qualitative evidence on the value of CIHR funded research and knowledge translation.
   Limitations: Case studies may not be able to capture the longer-term effects of research that evolves over time with several organizations participating.
3. Collaborative studies with partner organizations (e.g. provincial research agencies). These studies may include commissioned research or critical literature reviews.
   Strengths: Studies can be targeted to specific research objectives identified by funding agencies.
   Limitations: Studies may be exploratory in nature.
4. Indicators produced by national organizations (see examples in Table F3).
   Strengths: Indicators can provide comprehensive data for national populations, trends through time and international comparisons.
   Limitations: It is often difficult to attribute changes in high level indicators to specific causes. For example, research is one of many contributing factors to changes in population health.

The approach to identifying indicators and studies of research impact will be guided by the following principles:

• Identify what information is needed.
• Identify what information is available and gaps in present knowledge.
• Decide what new measures should be developed.
• Determine who will collect data and the potential costs.
• Set priorities, based on perceived importance and costs.

Schedule for Development
CIHR is in the process of developing a schedule for implementation of the framework.

References

---

2. These initiatives and references to documents describing them are provided in more detail in section B, C and D of the Policy Backgrounder prepared for the February meeting.
3. The five dimensions for defining impact of health research are based on the Payback model developed by Prof. Martin Buxton and Dr. Steve Hanney. A presentation of the model and its applications was provided at the meeting by Prof. Buxton.
4. Most of the remaining text has been condensed from a background paper by Michelle Leggo, the NHMRC presenter. The material is presented in detail because it provides valuable insight from attempts to measure and compare economic benefits from health research.
5. The project team in the quote above refers to the team of researchers and consultants who conducted the payback assessment of the 16 ARC projects.
6. Benefits are calculated by using a value of $100,000 for every year of life in perfect health gained (equivalent to a QALY of 1). Cost is equivalent to the average lifetime cost for persons of a reference age (45 in the case of CVD). Future benefits and costs are both discounted at 3% per year. In the example using CVD, the present value of future benefits (3 additional years of life) from treatment is $120,000. Present value of future costs is $30,000.
7. For readers unfamiliar with the analysis of marginal effects in economics, the problem is similar to an evaluation of investments, in which average returns over the past few years do not guarantee equivalent returns for investments made today. Nor do averages tell us what one's returns might have been if the timing or balance of specific investments within a portfolio had been different (e.g. high technology stocks during the last decade).
8. The section of the Framework titled 'Measures of Impact' includes a discussion of the current state of development of methods for measuring commercialization in Canada.
9. CIHR's four research themes are biomedical, clinical, population health and health services.
10. Definitions taken from CIHR Grants and Awards Guide, www.cihr-irsc.gc.ca/e/22630.html#1-A3
11. Adapted from Wooding et al (2004) - Returns from Arthritis Research, Volume 1, Pgs. 12-14. These definitions have been reviewed by Professor Buxton, co-author of the report. They should not, however, be viewed as official definitions for the Buxton Hanney model.
12. Net present value of future returns is the standard methodology used in economic benefit-cost analysis. Limitations on information availability may require the use of less rigorous measures, such as annual gross sales and profits.
13. Innovative biotechnology firms are defined as private sector firms that currently use or develop biotechnology in their activities or strategies. Universities, government labs, not-for-profit firms and firms with less than $100,000 in R&D expenditure or revenues of less than $250,000 were excluded from the survey.

1. Finding a Balance in Federal Health Research Funding. Canadian Institutes of Health Research. Ottawa, 2004.
2. Annual Financial Report of the Government of Canada, Fiscal Year - 2003-2004. http://www.fin.gc.ca/afr/2004/afr04_1e.html#Expenses
3. The Health of Canadians - The Federal Role, Final Report, Volume Six: Recommendations for Reform, PART V: Expanding Capacity and Building Infrastructure (Chapter 12), The Standing Senate Committee on Social Affairs, Science and Technology, Ottawa, October 2002.
4. Commission on the Future of Health Care in Canada, Building on Values: The Future of Health Care in Canada - Final Report (pp. 76, 87), Roy Romanow, Commissioner, Ottawa, November 2002.
5. Developing a CIHR Return on Investment (ROI) Framework; Policy Backgrounder. CIHR February, 2005.
6. Developing a CIHR Return on Investment Framework: Methodology Background Paper, CIHR February, 2005.
7. Developing a Conceptual Framework for Benchmarking and Measuring Performance of Innovation in Canada. Discussion paper prepared by Global Economics for the Advisory Council on Science and Technology Secretariat. Ottawa, 2003.
8. Returns on research investment: UK and Wellcome Trust perspectives. Presented at the CIHR meeting Feb. 23-24. Liz Allen, Wellcome Trust, UK.
9. PSA target metrics for the UK research base, pg.4. Department of Trade and Industry, Office of Science and Technology, UK., 2004,
10. The National Health and Medical Research Council and Performance Measurement. Paper prepared for the CIHR meeting Feb. 23-24. Michelle Leggo, NHMRC. Canberra, Australia.
11. Health and Medical Research Strategic Review. www.nhmrc.gov.au/wills/contents.htm
12. Exceptional Returns - the Value of Investing in Health R&D in Australia. A report prepared for The Australian Society for Medical Research by Access Economics September 2003 - www.asmr.org.au/news/
13. Exceptional Returns: The Economic Value of America's Investment in Medical Research. Papers commissioned by the Funding First Initiative of the Mary Woodard Lasker Charitable Trust. 2002. http://www.laskerfoundation.org/reports/pdf/exceptional.pdf
14. A Wealth of Knowledge. Allen's Consulting Group. September 2003 www.arc.gov.au/pdf/ARC_wealth_of_knowledged.pdf/
15. Wooding S, S Hanney, M Buxton, J Grant (2004) The returns from arthritis research. Volume 1: Approach, analysis and recommendations. RAND Europe, Cambridge. http://www.rand.org/publications/MG/MG251/
16. Your Money or Your Life: Strong Medicine for America's Healththcare System. Cutler, David M. Oxford University Press, 2004.
17. Cutler, D.M. Kadiyata, S. The Return to Biomedical Research: Treatment and Behavioral Effects (pg. 156-157). In Measuring the Gains from Medical Research An Economic Approach. Murphy K. M. Topple, R.H. eds. The University of Chicago Press. 2003.
18. Nordhaus, W.D. The Health of Nations: The Contribution of Improved Health to Living Standards (pg. 29, 30). In Measuring the Gains from Medical Research An Economic Approach. Murphy K. M.Topel, R.H. eds. The University of Chicago Press. 2003.
19. King, DA, 2004. The scientific impact of nations. Nature. 430: 311-316.
20. Buxton M, S Hanney, T Jones 2004. Estimating the economic value to societies of the impact of health research: a critical review. Bulletin of the World Health Organization. 82 (10):733-739.
21. 2003 Survey of Intellectual Property Commercialization in the Higher Education Sector. Preliminary release No. 1 (December 9, 2004). Science, Innovation and Electronic Information Division (SIEID). Statistics Canada.
22. Gault, Fred and Susan McDaniel. Forthcoming. Summary: Joint Statistics Canada - University of Windsor Workshop on Intellectual Property Commercialization Indicators, Windsor, November, 2004. SIEID Working paper. Statistics Canada.
23. Earl, Louise, Fred Gault and Michael Bordt. Forthcoming. Summary: Meeting on Commercialisation Measurement, Indicators, Gaps and Frameworks, Ottawa, December, 2004. SIEID Working paper. Statistics Canada.
24. Biotechnology Use and Development Survey. Preliminary results for 2003 reported in Innovation Analysis Bulletin. Statistics Canada Catalogue 88-003-XIE, V7,1. February, 2005.
25. Jacob R, M McGregor (1997). Assessing the Impact of Health Technology Assessment. International Journal of Technology Assessment in Health Care. 13:1 68-80.
26. Lehoux P, R N Battista, J M Lance (2000). Monitoring Health Technology Assessment Agencies. The Canadian Journal of Program Evaluation. V1:2, 1-33.
27. Sharpe A, Smith J. (2005). Measuring the Impact of Research on Well-being: A Survey of Indicators of Well-being. Centre for the Study of Living Standards Report 2005-02.
28. Torjman, S, F Minns (2005). Sustainable Development Framework for Science and Technology Research: Social and Cultural Dimensions. Caledon Institute of Social Policy. Ottawa.