Advancing Population and Public Health Economics: Annotated Bibliography
Canadian Institutes of Health Research-Institute of Population and Public Health
February 2013
Introduction and Context
This annotated bibliography was developed as part of the Advancing Population and Public Health Economics project, which was a collaborative effort among the National Collaborating Centres for Public Health (NCCPH), the Canadian Institutes of Health Research-Institute of Population and Public Health (CIHR-IPPH), the Canadian Institute for Health Information-Canadian Population Health Initiative (CIHI-CPHI) and the Public Health Agency of Canada (PHAC). CIHR-IPPH provided leadership and supported this project in close consultation with partners. The aim of the project was to engage researchers and professionals with expertise in the area of population and public health economics and to stimulate dialogue about key areas for research and capacity-building in the field. These objectives were supported through several activities, including this annotated bibliography, a background paper an invitational workshop held in January 2013 and proceedings from this workshop.
This bibliography is a compendium of relevant works published between 1992 and 2012 that aim to elicit discussion among key stakeholders about the role of economic analyses in population and public health. It was a key input for the project’s invitational workshop and is also expected to have a broader reach. There is a considerable body of literature exploring population and public health economics and an even larger collection of works related to health economics more generally. As such, the list of publications included in this document is not intended to be exhaustive, but rather is meant to highlight current understandings in the field in relation to the project objectives.
This bibliography is divided into four sections†:
- Making the case for investments in population and public health;
- Theoretical and methodological considerations in population and public health economic analyses;
- Advancing the field of economics in population and public health; and
- Illustrative examples of economic analyses in population and public health.
Advancing Population and Public Health Economics – Annotated Bibliography Acknowledgements
The Canadian Institutes of Health Research-Institute of Population and Public Health (CIHR-IPPH) would like to acknowledge several individuals for their generous contributions of time and expertise to the Advancing Population and Public Health Economics Annotated Bibliography.
For searching the literature and preparing all annotations:
- Max Deschner, CIHR-IPPH Summer Student, BA Joint Honours Candidate, Political Science and Anthropology, McGill University
- Rachel Maclean, CIHR-IPPH Summer Student, MPH Candidate (health promotion), Dalla Lana School of Public Health, University of Toronto
For editing the annotated bibliography:
- Andrea Hill, CIHR-IPPH Student, BJ Combined Honours Journalism and Biology, Carleton University
- Emma Cohen, CIHR-IPPH Knowledge Translation and Communications Officer
For providing expert review:
- Rodrigue Deuboué, PhD Candidate, Institute of Public Health, University of Ottawa; CIHR-Institute of Health Services and Policy Research
For providing oversight and expert review:
- Erica Di Ruggiero, CIHR-IPPH Associate Director
- Sarah Viehbeck, CIHR-IPPH Senior Evaluation Associate
The contents of this bibliography may be reproduced in whole or in part, provided the intended use is for non-commercial purposes and full acknowledgement is given to CIHR-IPPH.
Methods
An initial list of key citations and grey literature was suggested by staff members of the aforementioned partner organizations. In addition, individuals with expertise in the field of population and public health economics were consulted for their suggestions of publications most relevant to the stated project objectives. A number of articles were also identified as being pertinent following a review of the University of Calgary’s Annotated Bibliography of Economic Evaluations of Public Health Interventions Part 21. The identification of relevant works occurred exclusively between May and August 2012.
Inclusion Criteria:
After a thorough review of all articles, select publications were annotated based on the following inclusion criteria:
- peer-reviewed articles and grey literature sources published in English or French between 1999–2012, and
- sources that address population and public health economics from a conceptual (i.e. theoretical and/or methodological) and/or empirical perspective, with preference given to publications that:
- discuss the nature of economic evidence needed for population and public health;
- highlight relevant economic methods and underlying economic theories in the field of population and public health;
- provide illustrative examples of economic analyses in population and public health;
- discuss the infrastructure needed to support the field of population and public health economics;
- explore the remaining gaps that have not yet been adequately addressed; and,
- reflect on unintended consequences of conducting (or not conducting) economic analyses in public health.
Exclusion Criteria:
Publications that center on the role of economic factors (i.e. socioeconomic status) in determining health, rather than on the application of economic tools and/or theories to advance the field of population and public health, were excluded. Articles with an emphasis on health economics more generally were also excluded unless the authors described how the key concepts presented were applicable to the field of population and public health economics. Lastly, due to the breadth of available literature, annotations for primary studies that looked at real-world health economic research were not developed (i.e. economic analyses or evaluations of specific population and public health interventions were not included). However, a list of illustrative examples of economic analyses in population and public health is included on page 23. These examples were suggested by staff members of partner organizations and by experts in the field of population and public health economics.
Making the case for investments in population and public health
Allin S, Mossialos E, McKee M, Holland W. Making decisions on public health: a review of eight countries (Section 3) [ PDF (515 KB) - external link ]. Copenhagen: World Health Organization [WHO]; 2004.
In this section of a larger resource published by the World Health Organization, Allin et al. provide an overview of the organization, financing and priority-setting mechanisms behind public health strategies in Australia, Denmark, Canada, Finland, France, Germany (North Rhine-Westphalia), the Netherlands and Sweden. They refer to published literature on each country, particularly in the area of economic analyses. The authors find that economic evaluation and monitoring of public health policies is generally underdeveloped in these countries. They argue that monitoring and evaluation should receive the most investment and attention in public health development. They are unsure as to whether limitations in economic evaluation stem from lack of political will or are simply a result of new approaches that have had insufficient time to mature. The authors note that all countries included in their analysis recognize the necessity of developing a more systematic methodology to determine priorities and choose among potential interventions. Australia and the Netherlands are cited as countries that are increasingly using economic evaluation and evidence of interventions’ effectiveness. The authors encourage the use of economic evaluation, noting that this will ultimately lead to a broader evidence base to inform responsible public health policy making. They argue that international collaborations are vital for achieving such an evidence base.
Brown L, Thurecht L, Nepal B. The cost of inaction on the social determinants of health, CHA-NATSEM Second Report on Health Inequalities [ PDF (882 KB) - external link ]. Canberra: National Centre for Social and Economic Modelling, University of Canberra; 2012.
In their second report on health inequalities, Brown et al. note that health inequalities are increasing significantly for the most disadvantaged Australians as a result of government inaction on the social determinants of health. This is discussed in terms of the number of affected people, overall well-being, ability to work, earnings from work, reliance on government income support and use of health services. Consistent with the findings from the Marmot Commission, the report posits that reducing health inequalities is a question of inclusion, fairness and social justice. Major economic and social advantages such as decreased health inequities, increased satisfaction with life, gains in employment, increased annual earnings, reduced income and welfare support, as well as health system savings, are being dismissed as a result of inaction. The report stresses both the individual and collective health and financial benefits that could be accrued by improving health equity in Australia.
Suhrcke M, Sauto Arce R, McKee M, Rocco L. Economic costs of ill-health in the European Region [ PDF (7,945 KB) - external link ]. In: Figueras J, McKee M (eds). Health Systems, Health, Wealth and Societal Well-being: Assessing the case for investing in health systems. Maidenhead: Open University Press; 2012. p. 61-100.
In this chapter from a larger volume, Suhrcke et al. explore the economic consequences and costs of ill health and the economic benefits of good health within the context of the World Health Organization European Region. They argue that the concept of economic cost must be precisely defined in order to engage in meaningful debate on this topic. The authors define economic cost in three ways: 1) social welfare costs, which relate to the value individuals attribute to better health; 2) micro- and macro-economic costs, which relate to issues such as employment and a country's economic growth; and 3) healthcare costs and their relation to ill health. The authors then discuss different measures of the costs of ill health. They point to three salient policy implications: 1) estimated costs of ill health can be considered the upper limit of the potential benefits of economic interventions; 2) demonstrating the negative effects of ill health on social welfare, economic growth and health expenditures might facilitate getting policy makers outside the health system on board; and 3) only by factoring social welfare costs into economic evaluations can policy makers underscore the true economic benefits accrued from interventions.
McDaid D, Suhrcke M. The contribution of public health interventions: An economic perspective [ PDF (7,945 KB) - external link ]. In: Figueras J, McKee M (eds). Health Systems, Health, Wealth and Societal Well-being: Assessing the case for investing in health systems. Maidenhead: Open University Press; 2012. p. 125-152.
In this chapter from a larger volume, McDaid and Suhrcke outline the economic argument for investing in public health and health-promoting interventions. They argue that much of the disease burden in the World Health Organization European Region could be mitigated through early interventions that affect change both inside and outside the healthcare system. According to the authors, three criteria guide the economic argument for preventive interventions: 1) evidence of effectiveness; 2) information on the associated costs and benefits; and 3) whether the potential for “market failure” justifies the intervention. The authors argue that economic evaluation of interventions is a valuable tool when making policies. It acknowledges resource scarcity and is therefore necessarily ethical, as it aims to achieve the greatest benefits from a limited budget. The authors point to three ongoing challenges facing economic evaluations in this field: 1) strengthening the evidence base on the effectiveness of interventions; 2) increasing the current evidence base from economic evaluations; and 3) facilitating the implementation of evidence-informed population health strategies. They call for greater economic analysis outside the United States in order to understand implementation in different sociopolitical contexts and increased intersectoral action to raise awareness of both health and non-health benefits accrued through preventive interventions.
Public Health Agency of Canada. Investing in prevention – The economic perspective: Key findings from a survey of the recent evidence [ PDF (705 KB) - external link ]. Ottawa: Public Health Agency of Canada; 2009.
This report presents findings from an overview of recent evidence on the economic incentives to invest in prevention, with emphasis on the economic costs of ill health for the healthcare system and wider society. The report is not a systematic review of literature, but rather aims to offer a summative view of current knowledge about the economic benefits of prevention in order to support policy makers. The report notes the economic potential of preventive health, demonstrates current trends in research through economic evaluation cases, highlights gaps in economic evaluation knowledge and points to methodological issues when using economic evaluation to inform decision making. The report analyzes peer-reviewed academic and grey literature for current information on preventive health intervention evaluations, research questions and methodological issues related to using evidence. The report utilizes the “four faces of prevention” highlighted by Goldsmith et al. (2004) to categorize preventive health interventions. The authors note the move towards evaluations that highlight the general economic benefits of good health for society (beyond mere health system concerns). More effort is needed to fill knowledge gaps for preventive interventions that target the “upstream” determinants of health, which can increase the potential for overall savings.
Theoretical and methodological considerations in population and public health economics
Au F, Prahardhi S, Shiell A. Reliability of two instruments for critical assessment of economic evaluation. Value in Health 2008;11(3):435-439.
In this study, the authors assess the reliability of two instruments designed to critically appraise economic evaluations: the Quality of Health Economic Studies scale (QHES) and the Pediatric Quality Assessment Questionnaire (PQAQ). A random sample of 30 articles related to economic evaluations of health promotion was independently evaluated by two of the authors twice each using the aforementioned critical appraisal instruments. The second assessment was conducted 12 months after the first in order to reduce memory influence. An interclass correlation coefficient was used to assess the inter-rater reliability of both instruments, as well as the level of agreement between the two instruments. Cronbach’s generalizability theory was also used to measure the sources of variation in quality scores of the studies. The results of the study demonstrated excellent levels of inter-rater reliability and acceptable-to-high levels of agreement between the two instruments. With respect to the generalizability assessment, the greatest sources of variation in the scores assigned to each article were systematic differences in the quality of the articles themselves, representing 56% of the variance. The authors conclude that the choice of instrument can be based on other criteria, such as simplicity, rapidity of use and ease of application (as is the case for the QHES) or precision in finding problems with a particular study, which is useful when detailed critique is necessary (as is the case for the PQAQ).
Baltussen R, Leidl R, Ament A. Real world designs in economic evaluation: Bridging the gap between clinical research and policy-making. Pharmacoeconomics 1999;16(5 Pt 1):449-458.
In this review, Baltussen et al. note the information that economic evaluations of clinical research should offer in order to effectively inform policy makers about decisions in healthcare. Policy makers need both information on cost-effectiveness that is internally and externally valid and assessments of a healthcare intervention’s overall budget and health impacts. Such assessments necessitate a sort of aggregate analysis to evaluation, which is generally not employed by current individual-oriented approaches. The authors note use of three principle conceptual frameworks for evaluation: randomised controlled trials (RCTs), observational studies and modelling. RCTs are ideal given their high internal validity, but their low external validity should be a point of caution. The authors suggest a three-step approach to increasing the practical utility of economic evaluation: 1) ensuring internal validity; 2) adapting results to the real world (external validity); and 3) assessing the net costs and outcomes of the intervention at the system level. Adding a population component to models and using disease and public health modelling are ways to improve economic evaluation of an intervention.
Bonneux L, Birnie E. The discount rate in the economic evaluation of prevention: A thought experiment. Journal of Epidemiology & Community Health 2001;55(2):123-125.
Bonneux and Birnie carry out a thought experiment in order to calculate the savings gained through elimination of cardiovascular disease in the Netherlands. The authors aim to demonstrate that, in the standard economic model of evaluation, the devaluation of long-term health benefits of prevention by constant discount rates is not a result of societal preference. In fact, they posit that at a discount rate of 6% a vaccination that would eternally prevent cardiovascular disease would likely be considered worthwhile at a price of more than 1,100 Euro. As an exponential model, the discount rate compounds devaluation of the benefits of preventive strategies started during one's youth or adulthood and attempts to offset degenerative diseases. The authors note that discounting is not problematic in itself, but rather the constant monotonous discount rate in the standard economic evaluation model is undesirable. They explain that empirical studies about time preferences provide no evidence for this monotonous constant discount rate. Ultimately, value judgments of preventive strategies are complex, non-objective and not readily dependent on theories. Bonneux and Birnie argue that in order to guide people and policy makers in their decisions about the value of prevention, economic evaluation requires better empirical estimates of time preferences and more representative time-dependent models of discounting that reflect these.
Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, Truman BI et al. Developing an Evidence-Based Guide to Community Preventive Services – Methods. American Journal of Preventive Medicine 2000;18(1):35–43.
Briss et al. offer an overview of the methods used to undertake systematic reviews and convert these into evidence-based recommendations for the Guide to Community Preventive Services. The guide makes recommendations for certain preventive interventions related to changing risk behaviors, reducing specific diseases or impairments and addressing ecosystem and environmental dilemmas. The authors note that the guide considers a preventive intervention's effectiveness, applicability of effectiveness data (to other populations and contexts), economic results and challenges for implementation among other things. They also discuss specific ways in which evidence can be gathered and transformed into recommendations. The authors posit that using an evidence-based approach to find ideal interventions can reduce errors in how information is gathered, identify gaps in research knowledge and improve users' competency in determining the merits of an intervention recommendation themselves. The benefits of using an evidence-based approach can also lead to more widespread accord around the development and implementation of community health programs.
The specific methods for reviews of economic evaluations in the Community Guide are discussed here:
Carande-Kulis VG, Maciosek MV, Briss PA, Teutsch SM, Zaza S, Truman BI, et al. Methods for Systematic Reviews of Economic Evaluations for the Guide to Community Preventive Services. American Journal of Preventive Medicine 2000;18(1):75–91.
This paper offers an overview of the methods used in the Guide to Community Preventive Services: Systematic Reviews and Evidence-Based Recommendations for undertaking systematic reviews of economic evaluations in health promotion and disease prevention interventions. These studies include analyses of cost, cost-effectiveness, cost-benefit and cost-utility. The methods of the guide were developed in such a way that they could be applied to studies using different methods of evaluation. The authors explain that these methods can be adapted to various systematic reviews of economic evaluations, particularly in public health. The methods use explicit criteria to select the studies to be included in reviews, as well as a standard data abstraction form to compare data across interventions. The authors note that this approach increases the utility of the gathered economic information for policy making.
Canadian Agency for Drugs and Technology in Health. Guidelines for the economic evaluation of health technologies: Canada [ PDF (779 KB) - external link ], 3rd Ed. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2006.
This guide aims to improve the credibility and quality of standardized economic information for decision makers in the Canadian public healthcare system. It offers standards for conducting economic evaluations and reporting evidence in relation to technologies that promote health; prevent and treat conditions; and improve long-term care among other things. The guide offers standards for a wider audience beyond those who only evaluate drug technologies and includes methodological updates that have occurred within the economic evaluation of health technologies. It provides a broad overview of economic evaluation from underlying principles and types of evaluation to discussions on effectiveness; modelling; valuing outcomes; and variability and uncertainty among other things. The guide includes a notable section on equity considerations that discusses intervention assumptions, implications, groups affected and distributional impacts. The appendices of the guide include information on the presentation of analysis results, reviews of existing economic evidence, the standard reporting format and a glossary.
Coast J. Is economic evaluation in touch with society’s health values? BMJ: British Medical Journal 2004;329(7476):1233–1236.
Coast discusses the evolution of economic evaluation methods and the deviation away from using welfare approaches and towards using non-welfare approaches that pursue societal objectives. The author argues that current non-welfare approaches to economic evaluation, such as cost-effectiveness analyses, fail to consider all of society’s health objectives and are too complex for policy makers to utilize. The author discusses three questionable assumptions of the non-welfare approach that are often overlooked: 1) the convergence between the objectives of decision makers and those of economic evaluation, namely to maximize health outputs; 2) the validity of amalgamating multiple outcomes into one simplistic outcome such as the quality-adjusted life year (QALY); and 3) the usability and applicability of these complex techniques to decision makers. In light of these questionable assumptions, Coast posits that economic evaluations should be restricted to the cost-consequences approach, whereby different options available to decision makers are contrasted in tabular form according to their respective costs and consequences. Coast argues that the cost-consequence approach would more closely meet the needs of decision makers and avoid extensive use of inadequate assumptions.
De Salazar L, Jackson S, Shiell A, Rice M. Guide to the Economic Evaluation of Health Promotion. [ PDF (1,826 KB) - external link ] Washington: Pan American Health Organization; 2008.
This guide provides an overview of economic evaluation for health promotion interventions. The authors note that economic evaluation has generally been used to measure the cost-effectiveness and cost-benefit of preventive and clinical interventions. This is typically based on the “health gain.” However, health promotion is not solely about health gain. It is “a positive concept emphasizing social and personal resources as well as physical capacities”. Concerned with social wellbeing, it includes activities to reduce health inequities, protect the individual’s right to health, and promote the broad determinants of health. Economic evaluation thus becomes more complicated as it deals with health promotion. Beyond introducing the concept of economic evaluation, the guide discusses why it is important, the steps involved in carrying out an economic evaluation, issues that may arise when evaluating more complex interventions and other issues associated with the use of economic evaluation methods. As a result of the political, social and technical characteristics of economic evaluation in health promotion, the guide stresses the necessity of involving a multi-disciplinary team when carrying out evaluations.
Donaldson C, Currie G, Mitton C. Cost effectiveness analysis in healthcare: contraindications. BMJ 2002;325(7369):891-894.
Donaldson et al. present a skeptical view of the economic basis of the incremental cost-effectiveness ratio. They note that this ratio is typically based on the comparison of a novel intervention against what is presently practiced. The authors explain that incremental cost-effectiveness ratios cannot be equated with cost-effectiveness as they often call for more resources. This leads to questions of broader resource distribution. Cost effectiveness is defined as the organization of inputs to production in the most technically efficient way, choosing the combination that minimises cost. Incremental cost-effectiveness ratios can thus be problematic as they don't always organize inputs in the most efficient way. The authors argue that economic evaluations should consider different kinds of efficiency questions and their implications for opportunity cost. Studies should not offer prescriptions if resource distribution issues are at play. Donaldson et al. conclude by acknowledging the general success of classical cost-effectiveness, but argue that guidelines are needed so concepts of efficiency are better accounted for. Although the authors make no explicit mention of population or public health, this article was included in the annotated bibliography as it is relevant to these domains because it questions the conceptual (theoretical and/or methodological) foundations of using the incremental cost-effectiveness ratio.
Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Economics 2002;11:415-430.
In this technical paper, the authors explain that economic evaluations in healthcare are frequently conducted prospectively alongside clinical trials and, as such, often fail to incorporate econometric techniques. Furthermore, the authors assert that cost-effectiveness analysis, the primary approach in economic evaluation, has traditionally been concerned with the estimation of incremental cost-effectiveness ratios, which are associated with a number of statistical problems and are not amenable to regression analysis. The authors thus suggest augmenting the standard net-benefit framework, which reformulates the cost-effectiveness problem, to generate a net-benefit statistic by utilising a regression approach. Using empirical data from an economic evaluation of The Program in Assertive Community Treatment — a model of care for persons with severe and persistent mental illness — the authors demonstrate how a regression-type framework can enhance the net-benefit method. Though not without its limitations, the net-benefit regression approach allows health economists to identify important subgroups, adjust for imperfect randomization and, most notably, exploit the plethora of established econometric techniques. The authors conclude that future econometric work might investigate the adoption of more sophisticated regression methods to the net-benefit framework, such as the minimum distance framework or the generalized method of moments framework. The paper does not explicitly refer to economic analyses of population or public health interventions. However, it is included here as it deals with methodological issues that can be applied to these areas of economic evaluation.
Kelly MP, McDaid D, Ludbrook A, Powell J. Economic appraisal of public health interventions [ PDF (141 KB) - external link ]. London: NHS Health Development Agency; 2005.
Kelly et al. make the case for routinely and consistently applying the mechanisms of economic appraisal to public health interventions. They argue that economic appraisal is underdeveloped and inherently complex and lay out some of the challenges and solutions with this approach to evaluation. They note that using a common economic framework for evaluation would foster a reliable and transparent decision-making structure. Economic evaluation should not differ too greatly from common economic frameworks, but should be flexible enough to account for the multi-dimensional, complex and layered outcomes of public health policies and interventions. Economic analysis needs to account for the determinants of health inequalities; behaviour change; differences between cause and effect; biological and social variation in the population; and inevitable changes in interventions during implementation. All of these things can be incorporated into analysis by incorporating available information (including assumptions) into the analysis and making these explicit. The authors provide an overview of a number of different methods of evaluation. They explain that, over the long term, the use of comprehensive cost-benefit analyses (CBA) at the societal level are best able to capture wide, cross-sectoral effects of public health interventions. However, given difficulties in applying CBA, cost-consequence analyses (CCA) are recommended to capture layered outcomes of public health interventions at the local level.
Lessard C. Complexity and reflexivity: Two important issues for economic evaluation in healthcare. Social Science and Medicine 2007;64(8):1754–1765.
Lessard recognizes the complex, uncertain and multifaceted nature of healthcare. She notes that with the growth of healthcare costs, decision makers and managers have increasingly looked towards health services research and, in particular, health economics for answers to ongoing challenges. Lessard points to economic evaluation as a way to offer rigorous data that can inform the effective allocation of resources to maximize the health of the community. She notes that the rationale for economic evaluation is utilitarian in nature and is thus population-based. It is concerned with issues of both equity and efficiency. Yet, she criticizes the positivist and subjective nature of economic evaluation. She calls for economists to employ complexity thinking and reflexivity when carrying out evaluations and to be more imaginative in the approaches to the data and methodologies they use. Complexity thinking promotes greater awareness of uncertainty, contextual issues, different perspectives and transdisciplinarity among other things. This helps bring out the different values underpinning established knowledge. Reflexivity improves quality, reliability, validity and relevance of research. It can also shed light on subjective assumptions and practices within evaluation, ultimately promoting the pursuit of more rigorous research. The methodological developments promoted by Lessard, though specifically aimed at economic evaluation in healthcare, are equally applicable to economic evaluation in population and public health. Both complexity and reflexivity can improve the external validity of research in these areas, which can lead to more informed and responsible politics.
Lessard C, Birch S. Complex problems or simple solutions? Enhancing evidence-based economics to reflect reality [ external link ]. In: Shemilt I, Mugford M, Vale L, Marsh K, Donaldson C (eds). Evidence-based decisions and economics: healthcare, social welfare, education and criminal justice. Oxford: Wiley-Blackwell; 2010. 162-172.
In this chapter, Lessard and Birch argue that the traditional approach to health economics, wherein average health outcomes in a sample population are used to assess whether an intervention “works,” is too constricting and fails to recognize the complex pathways in which health is produced. In order to more adequately address the contexts in which decision makers must operate, the authors suggest a shift away from the dominant Newtonian paradigm and positivist perspective. They also underscore the necessity of incorporating complexity theory into health economic analyses. Two existing broad-based economic models that have the capacity to incorporate complexity are highlighted: Grossman’s model of the demand for health and Evans and Stoddart’s framework for the determinants of health. The authors argue that acknowledging the complex adaptive systems in which health is produced will allow for a greater understanding of how and why interventions work and under what conditions. Ultimately, this will increase the real-world applicability of health economic evaluations.
Oostenbrink J, Al M, Oppe M, Rutten-van Mölken M. Expected value of perfect information: An empirical example of reducing decision uncertainty by conducting additional research. Value in Health 2008;11(7):1070–1090.
Oostenbrink et al. offer insights on how value of information (VOI) analysis can inform policy makers about the expected value of undertaking further research to support a decision. VOI is useful for economic evaluations in health as it can offer information on the implications of choosing the wrong intervention. It recognizes that currently available information is shrouded in uncertainty. The expected value of (partial) perfect information (EVPPI) estimates the value of removing uncertainty simultaneously on parameters in model-based decision making. EVPPI is able to identify specific parameters for which uncertainties contribute most to overall decision uncertainty. This is useful as the policy maker can decide which intervention to adopt, as well as whether or not more research on the decision is needed. In this study, the authors aim to determine EVPPI before and after gathering more information on the parameter of a probabilistic Markov model with the greatest EVPPI. The Markov model compared the five-year costs and effects per quality-adjusted life year (QALY) of treating individuals with various levels of chronic obstructive pulmonary disease (COPD) and one of three bronchodilators. EVPI per patient is calculated and then multiplied by the number of patients qualified for treatment in order to determine the population EVPI. In conclusion, the VOI analysis used in the study determines parameters for which more research is advisable. After undertaking further research on the most important study parameter, EVPI is reduced and could be further reduced with more research on the next most important parameter.
Shemilt, I, Mugford M, Drummond M, Eisenstein E, Mallender J, McDaid D, Vale L, Walker D, The Campbell & Cochrane Economics Methods Group (CCEMG). Economics methods in Cochrane systematic reviews of health promotion and public health related interventions. BMC Medical Research Methodology 2006;6(55).
Shemilt et al. explore approaches to applying economics methodologies to different Cochrane systematic reviews in public health and health promotion in order to contribute to improving methodological guidance on economics for reviewers. The Cochrane Database of Systematic Reviews was searched to find studies involving economic evaluation. This incorporated 21 Cochrane reviews and seven review protocols related to health promotion and public health. Data explaining the economic aspects of studies were extracted, summarized and qualitatively analyzed. None of the reviews or protocols involved formal economic evaluation, but 10 aimed to involve studies and data in economics. The authors find that reporting on the methodologies underpinning economics sections often lacks specificity and that reviews can tailor more specific search strategies that include criteria intended for effectiveness studies. Among other recommendations, the authors argue that there is ultimately a need to develop evidence-based guidance for reviewers in order to help them determine whether and how to include economic methods in a systematic review.
Shiell A, Hawe P. Test-retest reliability of willingness to pay. European Journal of Health Economics 2006;7:173-178.
Despite the increased application of the willingness-to-pay (WTP) approach in the economic evaluation of population health interventions, very few studies have examined the reliability of WTP measures. In this article, Shiell and Hawe present the results of a survey designed to assess the test-retest reliability of face-to-face interviews as a method of eliciting WTP values in a sample of the general population. During three occasions over a period of five weeks, a randomly selected sample of the general population was asked to provide WTP measures for a hypothetical intervention. Test-retest reliability — defined in the context of this study as the ability of the WTP approach to rank health states or health interventions in terms of their value consistency over time — was assessed by interclass correlation and by generalizability analysis. The test-retest reliability of the WTP instrument was found to be acceptable, though not substantial, and there was a statistically significant shift in mean value between the first and second assessments. The most significant source of variation was the participants themselves, though the interaction between participants and time; the interaction between participants and health state; and random error or omitted variables also accounted for significant variation. They conclude that the WTP technique is likely capable of consistently differentiating between health states over time. The volatility seen in the study was likely a result of measurement error, but might also be seen as evidence that undermines the claim that respondents have stable and developed preferences when they come to valuation exercises. The WTP approach might seem reliable, but it is not necessarily valid.
Shiell A, Sperber D, Porat C. Do taboo trade-offs explain the difficulty in valuing health and social interventions? The Journal of Socio-Economics 2009;38:935-939.
Shiell et al. describe willingness-to-pay (WTP) as an economic technique often used in health economics evaluation because there is little restriction in what can be included within its measurement. The authors note that although there are improvements in measurement and broad use of guidelines on WTP use, there are still inconsistencies and anomalies in results. This problematizes the notion that all values can be monetized and that all sources of value are commensurable. Shiell et al. point to two types of incommensurability: interdimensional (the difficulty of assigning monetary value to health) and constitutive (when some types of tradeoff are considered “taboo”). Taboo tradeoffs do not consist solely of consequentialist logic, but require the chooser to consider normative boundaries of the self and set-in values. To test the effects of taboo tradeoffs on WTP, study participants were asked to value three types of goods: a market-traded private good, a private healthcare good and a quasi-public good. The authors employed a qualitative/quantitative mixed-methods approach alongside a vocalized thought exercise as a way to better construe respondents’ values and the challenges they faced in answering questions. Ultimately, the authors found no conclusive evidence that taboo tradeoffs negatively affect one's willingness to participate in such a valuation activity.
Weatherly H, Drummond M, Claxton K, Cookson R, Ferguson B, Godfrey C, et al. Methods for assessing the cost-effectiveness of public health interventions: Key challenges and recommendations. Health Policy 1999;93(2-3):85–92.
Weatherly et al. identify the methodological challenges associated with economic evaluation of public health interventions and offer some suggestions for overcoming them. The authors analyzed five reviews concerned with the economics of public health and identify four principle methodological challenges within these: 1) attribution of effects (using approaches other than controlled trials to measure effects of interventions and accounting for effects over the longer term); 2) measuring and valuing outcomes; 3) identifying intersectoral costs and consequences; and 4) incorporating equity considerations. The authors reviewed empirical studies to determine how these methodological challenges had been approached in the past and held an expert workshop to discuss future approaches. From the initial search, 154 studies in areas such as alcohol and obesity and physical exercise met the inclusion criteria. However, most studies did not sufficiently deal with the four identified methodological challenges. The expert workshop compensated for the studies' lack of insight into overcoming the challenges by determining a number of possible steps forward. The authors note that the experts’ suggestions have generally not been applied in practice. Thus, there is a critical need for pilot studies and more methodological research.
Advancing the field of economics in population and public health
Ammerman AS, Farrelly MA, Cavallo DN, Ickes SB, Hoerger TJ. Health economics in public health. American Journal of Preventive Medicine 2009;36(3):273-275.
Ammerman et al. present the results from a survey that 1) assessed use of health economics by public health researchers and practitioners; 2) identified barriers to use of health economics; and 3) collected recommendations on how to overcome barriers. The term health economics is used to denote the branch of economics that deals with healthcare service provision, delivery and use. Of note is that more than half of the 294 respondents reported very little or no use of health economics in their work. The most commonly cited barriers to the use of economic resources were lack of experience in health economics; funding limitations; lack of time; lack of technical understanding and intimidation associated with economic theory; and lack of both tools for basic economic assessment and data on economic variables. Ammerman et al. underline the number of respondents who reported a willingness to incorporate health economics into their work and thus suggest greater emphasis be placed on increasing opportunities for collaboration between economists and public health professionals.
Cookson R, Drummond M, Weatherly H. Explicit incorporation of equity considerations into economic evaluation of public health interventions. Health Economics, Policy and Law 2009;4(2):231-245.
Cookson et al. argue that economic evaluations in public health have focused primarily on efficiency objectives — namely the maximization of health gain — and have failed to incorporate equity objectives despite the importance of equity as a policy objective in public health interventions. There are three types of equity considerations that may be made when determining allocation of scarce resources in public health: 1) concern to reduce a particular type of health inequity; 2) concern to give priority to improving the health of a particular group; and 3) concern to respect a particular ethical rule or procedure. In an attempt to facilitate the integration of equity considerations into economic evaluation in public health, the authors discuss four main approaches: 1) a review of the background information on equity; 2) health inequality impact assessment; 3) analysis of the opportunity cost of equity; and 4) equity weighting of health outcomes. The authors suggest that the first three approaches can be readily applied with little additional cost, while the fourth approach requires further methodological research.
A response to Cookson, Drummon, and Weatherly (2009) can be found here:
Shiell A. Still waiting for the great leap forward. Health Economics, Policy and Law 2009;4(2):255-260.
The author offers his insight on how to best incorporate equity into health economic evaluations. Cookson, Drummond and Weatherly suggest that future efforts be directed towards re-examining the equity implications of 1) interventions known to be cost-effective and likely to increase inequalities in health; and 2) interventions believed to be equity enhancing, but shown to be cost-ineffective. However, Shiell asserts that this is an inappropriate use of scarce resources given that most of the economic evidence relating to public health is focused on clinical and/or individual-lifestyle interventions. Shiell argues that what is most needed is an increase in the quality and quantity of primary evidence relating to the cost-effectiveness of interventions tackling the social determinants of health.
Eddama O, Coast J. A systematic review of the use of economic evaluation in local decision making. Health Policy 2008;86:129-141.
Based on a systematic review of the literature, the authors suggest there is limited use of economic evaluation in local decision making. Local decision making is defined both at the “meso” and “micro” levels. At the “meso” level, it is where bodies or actors within healthcare organizations make broad decisions about the allocation of care to certain groups of patients or diseases. At the “micro” level, local decision making is where decisions concerning the patient are made directly by healthcare providers. The barriers that limit the use of economic evaluation among local decision-makers are linked to three factors: institutional and political factors; cultural factors; and methodological factors. Despite the evidence suggesting that there are many barriers to the use of economic evaluation, the authors report an increase in the use of economic evaluation over time, particularly in the United Kingdom as well as Australia and Canada. This increase is less pronounced in the United States. Eddama and Coast conclude that not enough is known about the exact use of economic evaluation in local decision making and that further qualitative work is needed to enrich and explain the results from their systematic review.
Evers S, Salvador-Carulla L, Halsteinli V, McDaid D, & The Mheen Group. Implementing mental health economic evaluation evidence: Building a bridge between theory and practice. Journal of Mental Health 2007;16(2):223-241.
Evers et al. recognize that although there is increasing interest in using economic evaluation to inform strategic decision making on mental health policy, the capacity to use evaluation is underdeveloped. They describe some of the challenges faced in conducting economic evaluations and how these evaluations are used to support decision making related to mental health policy in Europe. Using a questionnaire and literature review as the basis of the study, the authors find that, although economic evaluations are increasingly being used, their quality is highly variable. Economic evaluations are typically focused on medications and few evaluations take place outside the health and social care sectors. Cost-effectiveness evaluations are more prevalent than cost-utility or cost-benefit analysis. Evers et al. conclude that there is much room for methodological and practical developments in economic evaluation, mainly with respect to outcome measurement and economic evaluations of complex non-health system interventions.
Godfrey C. Economic evaluation of health promotion [ PDF (2,435 KB) - external link ]. In: Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, Ziglio E (eds). Evaluation in health promotion: principles and perspectives. Copenhagen: WHO Regional Publications; 2001. p. 149-170.
Godfrey offers an introduction to the economic evaluation of health promotion, explaining that evaluation involves identifying, measuring and valuing both the inputs (costs) and outcomes (benefits) of different interventions. She explains that economic evaluation should be employed within an extensive grouping of different analytical tools and that researchers must recognize the difference between partial and full economic evaluations. Godfrey also notes that a main challenge for health economists and health promotion professionals is resolving the discrepancy between demand for and supply of cost-effectiveness information on interventions. Developing rigorous and practical evaluation designs is a critical step that should be taken to establish effectiveness to which economic evaluations can be added. The author suggests that policy makers need to be educated about economic evaluation and outlines the problems that can arise when a poorly-designed study is used to inform decision making.
Goldsmith LJ, Hutchison B, Hurley J. Economic evaluation across the four faces of prevention: a Canadian perspective [ external link ]. Hamilton: Centre for Health Economics and Policy Analysis, McMaster University; 2005.
Goldsmith offers economic arguments for preventive measures over curative measures using examples from five specific interventions in the Canadian context. She notes that prevention and cure are not inherently dichotomous, but that the former often lacks outright support as it has no immediate beneficiaries and typically has high initial costs and deferred benefits. A literature search for the study yielded 290 recommended prevention interventions and 23 interventions with potentially major population health impacts. Six hundred-seventy-two economic evaluations were identified from 154 remaining preventive interventions. Economic evaluations were categorized by type of intervention and a number of other factors. Evaluations of the five interventions were then summarized. Goldsmith posed three questions for each intervention in aggregating and analyzing results of their evaluations: 1) does the intervention produce a net benefit from the societal perspective?; 2) is the intervention cost-saving from the payer perspective?; and 3) if neither, might the intervention nevertheless be a worthwhile investment in health? Despite methodological and contextual diversity, Goldsmith finds there is consistency among evaluations. All of the interventions analyzed have a net societal benefit. To conclude, Goldsmith notes that unappraised evaluation evidence exists for many preventive interventions, however, economic evidence is non-existent for the recommended preventive interventions. She calls for more systematic reviews of effectiveness evidence and economic evaluation evidence, as well as economic evaluations of individual preventive interventions for which evidence is lacking.
Golaszewski T. Shining lights: Studies that have most influenced the understanding of health promotion’s financial impact. American Journal of Health Promotion 2001;15(5):332-340.
Golaszewski undertakes a review of the literature to identify studies that underpin the economic merit of health promotion. His findings offer moderate support for health promotion's financial value. A panel of experts was asked to identify the five to 10 most influential research articles providing evidence that supports or undermines the economic case of health promotion programs. Articles were published between 1980 and 2000. An initial nomination list of 23 studies was cut to 12 articles based on rankings by the panel. Inclusion and exclusion criteria included studies that: 1) examined the relationship between health risks and financial outcomes, or health promotion programs and financial outcomes; 2) provided strong and compelling financial data supporting the worth of health promotion; 3) had a high quality methodology; 4) answered an important question or replicated important findings with superior methodology; and 5) represented U.S.-based initiatives published since 1980. Data was extracted from the studies based on population characteristics, design, statistical tests, limitations, and results. Golaszewski makes several findings: 1) evidence supports an association between modifiable health risk factors and healthcare costs; 2) health promotion programs can effectively provide financial returns, mainly for healthcare costs and reduced absences; and 3) most economics-based research is initiated by private sources. The author highlights some methodological problems within this research domain (e.g. inability to properly randomize subjects into proper experimental and control groups). He concludes that more government and philanthropic funding is needed to bring about more quality research on the financial merits of health promotion.
Grosse SD, Teutsch SM, Haddix AC. Lessons from cost-effectiveness research for United States public health policy. Annual Review of Public Health 2007;28:365-391.
Grosse et al. discuss economic evaluation and its use in public health through population-level case studies and clinical preventive services' public health promotion strategies. They address cost-effectiveness analysis (CEA) and its methods, what should be made of findings from CEA, and how economic evaluations inform decision making at the policy level. Decision makers must grapple with questions of scarce resources when looking at which public health interventions to implement. Deliberations can be aided by financial evidence, but they are also inherently subjective and depend on a multitude of stakeholders. Grosse et al. note that economic evaluations are an important part of decision making, as is evidence of effectiveness and contextual information. They conclude that CEA findings are “a decision aid, not a decision rule” and that efforts must be made to standardize and make more transparent methods and assumptions of CEA.
Harris JR, Holman PB, Carande-Kulis VG. Financial impact of health promotion: We need to know much more, but we know enough to act. American Journal of Health Promotion 2001;15(5):378-382.
In this commentary, Harris et al. offer an overview of present efforts at the US Centers for Disease Control and Prevention to investigate the financial impacts of prevention and health promotion programs. They explain that, although the effectiveness of many preventive and health promotion activities is known, their financial impacts are not always clear. The authors point to several successful interventions outlined in the Guide to Clinical Preventive Services and the Guide to Community Preventive Services in an effort to show the financial benefits of investing in prevention and health promotion (particularly in the areas of tobacco control and vaccination). They discuss efforts to improve implementation of interventions, as well as the knowledge base for determining the financial impact of effective interventions (economic evaluations that involve cost-effectiveness analysis or cost-benefit analysis). The authors call for five research-related needs for financial impact: 1) more well-executed economic evaluations of interventions and well-evaluated implementations of health promotion programs and their financial impact; 2) common definitions and measures; 3) analyses responsive to business concerns (which could improve credibility and relevance of analyses as businesses may be less interested in the “societal perspective”); 4) a common framework for using research in terms of financial impact; and 5) collective experience of all involved players in order to increase relevance to decision-making. Harris et al. conclude that, although we need to know much more about the financial impact of health promotion, we know enough now to start acting. They point to the Guide to Community Preventive Services as a list of evidence-based actions that can help healthcare purchasers, workplace managers, and corporate citizens improve delivery of effective interventions. Several specific prescriptions are offered for these individuals.
Lessard C, Contandriopoulos A-P, Beaulieu M-D. The role (or not) of economic evaluation at the micro level: can Bourdieu's theory provide a way forward for clinical decision making? Soc Sci Med 2010;70(12):1948–1956.
Based on a review of the literature, the authors present the current state of knowledge on economic evaluation, evidence-based medicine and family physicians in healthcare. The authors suggest that although interest in and use of health economic evaluation has increased at the macro (policy) level of decision making, its application has remained limited at the meso (administrative) and micro (clinical) level. Given the scarcity of empirical studies on the use of economic evaluations in healthcare decision making at the micro level and the overall neglect for the social reality of clinical decision making in existing studies, the authors propose Bourdieu’s sociological theory as a research framework. They note there is a need to improve understanding of economic evaluation’s role in decision making from a disciplinary outlook different than health economics. Bourdieu’s theory could contribute to a deeper understanding of 1) the social processes that influence family physicians’ schemes of perception, thought and action with respect to the role of economic evaluation in clinical decision making; 2) the wider social relationships that influence family physicians’ willingness to contribute to efficient resource allocation; 3) the potential influence of economic evaluation on everyday clinical decision making; 4) the development of a comprehensive conceptual framework for understanding the role of economic evaluation in the decision-making process of family physicians; and 5) a deeper understanding of the theory and practice of knowledge translation. Ultimately, the authors suggest that the proposed research approach could lead to an increase in the use of economic evaluation in clinical decision making and to the development of effective knowledge translation strategies specific to the primary care sector. Although the authors focus on clinical decision-making, this article was annotated as the proposed research framework is potentially applicable to micro-level population and public health professionals.
Lorgelly PK, Lawson KD, Fenwick EAL, Briggs AH. Outcome measurement in economic evaluations of public health interventions: a role for the capability approach? International Journal of Environmental Research and Public Health 2010;7:2274–2289.
Lorgelly et al. discuss the limitations of exclusively focusing on health as an outcome measure in economic evaluations of public health interventions. These limitations are significant given the large proportion of interventions that seek to improve an individual’s quality of life beyond health. The authors suggest that using the capability approach for outcome measurement within economic evaluation may confer significant benefits. The approach, which measures wellbeing according to an individual’s capability rather than his or her functioning, may facilitate the development of an all-encompassing outcome measure. In order to operationalize this approach for use in economic evaluations, the authors argue it is essential to develop an index that assigns a single composite number to an individual’s capability and that future research works towards removing any of the conceptual challenges associated with the adoption of the capability approach.
Madden L, King L, Shiell A. How do government health departments in Australia access health economics advice to inform decisions for health? A survey. Australian and New Zealand Journal of Health Policy 2009;6:6.
Despite several instances in which economic evaluations have been used to support public health policy development in Australia, the authors suggest that health economic evidence is not always well-matched to the needs of government decision-makers. By surveying a purposeful sample of health and non-health economists employed by all departments of health in Australia and select government departments in New South Wales, the authors identify the range of mechanisms used to access economic evidence. All but one health departments report utilizing health economics expertise to inform decision making, with most departments using a mix of internal (e.g. staff training in health economics, health economist positions, and health economics units) and external (e.g. consultancy for services and collaborative research centres) mechanisms to secure economic evidence. Each mechanism reported by respondents was linked to the type of health economic functions that governments require to inform decision making for policy and planning purposes. Based on the findings, the authors conclude that, although economic evidence is being used to inform decision making, lack of uniformity in approach suggests health economics is a specialty area that necessitates further development.
Milstein B, Homer J, Briss P, Burton D, Pechacek T. Why behavioral and environmental interventions are needed to improve health at lower cost. Health Affairs 2011;30(5):823-832.
Using the CDC-developed HealthBound policy simulation model of the United States’ healthcare system, the authors estimate the relative and combined health and economic impacts of expanding health insurance; delivering better preventive and chronic care; and enabling healthier behavior and safer environments. Each simulated intervention was compared to the 2003 status quo scenario after 10 and 25 years along two summary measures, including deaths prevented and healthcare/intervention costs. Data from 2003 were the basis for all estimations in this study (the last year for which key data was available). In the first 10 years, all three interventions were likely to prevent avoidable deaths as compared to the status quo scenario, with the most lives saved by the care intervention, followed by protection and then coverage. The 10-year cumulative costs were lowest for protection, followed by care and then coverage. After 25 years, interventions designed to improve behavioral and environmental conditions were shown to be the most effective and the least costly, followed by better preventive and chronic care and then increased insurance coverage. The authors also report that, although each individual intervention has the potential to save lives and reduce costs, they are likely to be most effective when implemented in combination. In light of the study results, the authors argue that despite the fact that impact of behavioral and environmental interventions grows more gradually, upstream protective interventions should not be overlooked as they can save lives and alleviate demand on limited healthcare capacity.
Rush B, Shiell A, Hawe P. A census of economic evaluations in health promotion. Health Education Research 2004;19(6):707-719.
The authors present the findings from a systematic search of 412 pieces of relevant peer-reviewed and grey literature published between 1990 and 2001 on economic efficiency of health promotion interventions. The authors note a steady increase in the number of studies reporting some aspect of the cost-effectiveness of population health interventions in the past decade, though the literature remains desolate in comparison to the breadth of economic evaluations of health services more generally. Relevant articles were classified according to a four-part typology: type of health promotion intervention, risk factor addressed by the intervention, setting in which the intervention was situated and population most affected by the intervention. The majority of references were concentrated on select areas of health promotion practice, namely clinical interventions and behavioural programs. There was a scarcity of economic evidence for interventions designed to address the social and economic determinants of health and, as such, the authors suggest the application of economic methods needs to be improved to deal with more complex health promotion interventions. Lastly, the authors underscore the need to increase the applicability and accessibility of economic evidence for decision makers.
Shiell A. In search of social value. International Journal of Public Health 2007;52:333-334.
Shiell highlights the challenges of evaluating the cost-effectiveness of social interventions by using a walking school bus intervention as an example. The walking school bus involves a group of children walking to school with two adult supervisors. This stems from the fact that benefits of social interventions generally extend beyond health and often interact with one another. These benefits are multiple and multiplied through feedback loops. Shiell refers to a study that concluded the walking school bus was not a cost-effective intervention. He explains that this study did not capture all outcomes of the intervention, particularly those beyond health benefits, such as the administrative and managerial skills gained by parents who organize the bus and discussions amongst participants about how to make their neighborhood safer and healthier. The study thus ignored the intervention’s social value. Shiell argues that, from a methodological standpoint, there remain significant gaps in how economic evaluations approach social interventions. Although current approaches guarantee that all costs and benefits are identified in the study, they are not always measured and valued. Shiell notes that this may be acceptable in economic evaluations of healthcare, but it is not acceptable in the case of social interventions with broad non-health effects.
Shiell A, McIntosh K. Some economics of health promotion: what we know, don’t know, and need to know before deciding how much to spend on promoting the public’s health [ PDF (280 KB) - external link ]. Harvard Health Policy Review 2006;7:21–31.
Shiell and McIntosh review economic evidence on health promotion, exploring what is now known, what is not known and what needs to be known to make the case for investments in prevention. They explain that health promotion is not always an important item on the policy agenda because the economic evidence is not uniformly in favor of health promotion. Who pays for interventions and who benefits from them are always contentious issues. Moreover, there are many barriers to policy implementation. The authors offer a primer on economic evaluation, explaining cost-benefit analysis and the use of quality-adjusted life-year (QALY) measurements to compare different programs. They provide several examples of health promotion activities that both promote health and save resources (e.g. some vaccination and needle/syringe programs). They demonstrate that prevention is not universally less expensive or more effective than treatment; rather, the merits of preventive interventions need to be assessed individually. According to Shiell and McIntosh, a key shortcoming of the economic evidence of health promotion is the focus on clinical and behavioural programs, which are the easiest to evaluate. They call for more evaluations of “upstream” health determinants, which generate evidence on health promotion activities that address social, economic, or environmental risk factors. Understanding such issues might improve population health and reduce socio-economic health inequalities. They also encourage more evidence on how “healthy” public policy can be best implemented, which necessitates a better understanding of the political economy of public health and stakeholder values regarding resource allocation.
Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ: British Medical Journal 2008;336(7656):1281–1283.
Shiell et al. argue it is important to differentiate between complex multi-component health interventions and the complex systems in which interventions are implemented (i.e. primary healthcare, hospitals and schools). They further argue that these distinctions be applied correctly in economic evaluations. A complex intervention is created from multiple components that act both independently and inter-dependently. A complex system is adaptive to changes in local environment, is composed of other complex systems and behaves in a non-linear manner. Although complex interventions can be difficult to evaluate economically, this can feasibly be achieved given sufficient time, effort and resources because evaluation does not require new economic methods. However, economic evaluation of interventions aimed at changing complex systems presents a much greater challenge because of the emergent properties that characterize these systems as a whole. Outcomes cannot be measured based on an aggregate sum of individual changes. Evaluation of these interventions thus requires a new methodological approach that is sensitive to interactions between variables at different levels and accounts for non-linearities, multiplier effects and changes in individual values resulting from interventions.
Smith, R. The Relationship between reliability and size of willingness-to-pay values: a qualitative insight. Health Economics 2007;16:211-216 (published online 23 August 2006 in Wiley InterScience).
In a previous quantitative study (2004), Smith finds that the reliability of willingness-to-pay (WTP) values is an increasing function of the size of WTP communicated. Reliability is demonstrated through a “test-retest” study, whereby values at one time are compared to those at a later time. In this article, Smith presents the results of a qualitative study undertaken with the 2004 quantitative study. The qualitative study suggests that greater WTP values might require respondents to more closely consider which values they choose. This gives the WTP values greater reliability. The author explains that temporal reliability is an important component of validity and refers to the stability, consistency, or agreement of values over time. Smith suggests that low values of WTP might result from a respondent's “discretionary account,” where expenditure is more variable. The qualitative component of this study required interview respondents to complete a concurrent verbal protocol or “think aloud” procedure in which they verbalized their thought process as they responded to questions. The interview was followed by an “open-ended” discussion regarding anything the respondent felt was important to bring up. The author notes that the findings of the study could imply that there is greater confident in WTP values when they comprise a large portion of respondent income. This study is useful in that it contributes to the exploration of factors (through qualitative investigation) that might underlie quantitative evidence.
A response to Smith (2006) can be found here:
Shiell A, McIntosh K. Subject variation more than values clarification explains the reliability of WTP estimates. Health Economics 2008;17:287-292.
Shiell and McIntosh refute Smith’s 2006 study on willingness to pay (WTP) measures. Smith’s study suggested that test-retest reliability increases with WTP because individuals take more time to consider their valuation of more expensive or highly valued goods. Shiell and McIntosh argue that Smith’s study misconceives what reliability actually measures. Shiell and McIntosh find that 75% of the increase in reliability stems from greater subject variation (i.e. diverse views from study respondents about the value of good health). They offer an overview of what reliability is and its measurement using the intraclass correlation coefficient (ICC). They note that reliability is not “consistency, stability, or agreement,” as posited in Smith’s study, because it remains strong both when respondents provide the same answers and when responses change systematically over time. The relationship between reliability and WTP is affected by both subject variation and measurement error. Shiell and McIntosh call for better understanding of the concept of reliability in order to properly interpret the outcomes of repeated WTP measures and to properly differentiate between measurement error and systematic changes in values assigned to goods both within and between individuals.
Van Velden ME, Severens JL, Novak A. Economic evaluations of healthcare programmes and decision making: the influence of economic evaluations on different healthcare decision-making levels. PharmacoEconomics 2005;23(11):1075–1082.
Van Velden et al. carry out a literature review to determine the extent to which economic evaluations influence healthcare decision making at the macro (national or regional), meso (administrative) and micro (provider) levels. Thirty-six empirical studies on the effects of economic evaluation at these levels were included in the review. The literature search was carried out in 2004 using the MEDLINE database. Of an initial 2,400 screened articles, 28 empirical studies were selected for inclusion and an additional eight were identified through reference lists. Articles were sorted into six categories based on their underlying arguments: healthcare programme/manufacturer; costs/economics; patient/disease; political/policy; administration/user; and legal/regulatory. The authors found that economic evaluations have a limited influence on decision making at the macro and micro levels. Political and regulatory factors are major determinants of decisions at the macro level. Economic evaluations have the most influence at the meso level. The authors conclude that this influence is most pronounced for healthcare organizations. They note that economic evaluations are important factors in decision making, but cannot be considered the most important determinants of healthcare decisions at any of the levels discussed. Although this article does not explicitly mention economic evaluation in relation to population or public health, its findings are applicable to these areas of health. The authors note that researchers should concentrate on a decision making level when undertaking economic evaluations and that healthcare decision makers should improve their understanding of economic evaluations to use them more often and more effectively. This article was annotated as it was felt that the authors’ recommendations are applicable to population and public health researchers, practitioners and decision makers.
Wagstaff A, Culyer AJ. Four decades of health economics through a bibliometric lens. Journal of Health Economics 2012;31:406-439.
Wagstaff and Culyer review the field of health economics over the past 40 years using bibliometric “metadata” from EconLit, citation information from Google Scholar and topical classifications determined by the authors themselves. They note the growth of health economics, as well as its shifting topical and geographical foci. They compare authors, countries, institutions and journals based on the volume of publications in health economics and the influence assigned to these by different citation-based indices. The authors find that topics like “determinants of health and ill-health” and “health statistics and econometrics” are those of the 50 most-cited publications of the 2000s (nine each), while “medical insurance” and “supply of health services” are less popular than they were in the past (three and two, respectively). Six publications are classified under “public health.” “Health and its value” and “economic evaluation” are two topics that demonstrate no trend in popularity. In a list of the 300 most-cited publications in health economics over the last 40 years, “public health” and “determinants of health and ill-health” (includes a population health perspective) were the topics of 28 and 42 publications, respectively. The authors note the significant expansion of health economics in the developing world, though they judge Central Asia, Africa and the Middle East to be under-researched. EconLit is chosen as the database to identify relevant publications given its wide scope and its Journal of Economic Literature (JEL) classification scheme. To conclude, Wagstaff and Culyer note that health economics has yet to address some of the ethical dilemmas tied into this field, including “moral” issues related to healthcare investment decisions and opposition to the utilitarian foundation of economics.
Williams I, McIver S, Moore D, Bryan S. The use of economic evaluations in NHS decision making: a review and empirical investigation. Health Technol Assess 2008;12(7):iii, ix-x, 1-175.
Williams et al. report on the use of research evidence concerned with economic evaluation in healthcare decision making. They address how, and to what extent, health economic information is used in health policy decision making in the United Kingdom, as well as what factors are associated with the use or non-use of this research evidence. The authors use electronic databases (up to 2004) to undertake a systematic review of literature on the use of economic evaluation in decision making and barriers to use of economic evaluation. Qualitative methods including documentary analysis, meeting observation and semi-structured interviewing are employed to conduct five case studies. The authors find very few past systematic reviews in this area. They also note that some previous healthcare studies on the use of economic evaluation in decision making have problematic methodological approaches. They call for more qualitative research on the spectrum of policy decision-making levels.
Illustrative examples of economic analyses in population and public health2
Anderson DR, Serxner SA, Gold DB. Conceptual framework, critical questions, and practical challenges in conducting research on the financial impact of worksite health promotion. American Journal of Health Promotion 2001;15(50):281-288.
Baker M. Innis Lecture: Universal early childhood interventions: What is the evidence base? Canadian Journal of Economics 2011;44(4):1069–1105.
Australian Government Department of Health and Ageing. Returns on investment in public health. Canberra: Department of Health and Ageing; 2003.
Chokshi DA, Farley TA. The cost-effectiveness of environmental approaches to disease prevention. The New England Journal of Medicine 2012;367(4):295-297.
Emery JCH, Matheson JA. Should income transfers be targeted or universal? Insights from public pension influences on elderly mortality in Canada, 1921–1966. Canadian Journal of Economics 2012;44(5):247-269.
Gold L, Shiell A, Hawe P, Riley T, Rankin B, Smithers P. The costs of a community-based intervention to promote maternal health. Health Education Research 2007;22(5):648-657.
Grosse SD, Matte TD, Schwartz J, Jackson RJ. Economic gains resulting from the reduction in children's exposure to lead in the United States. Environmental Health Perspectives 2002;110(6):563-569.
Grosse SD, Waitzman NJ, Romano PS, Mulinare J. Reevaluating the benefits of folic acid fortification in the United States: Economic analysis, regulation, and public health. American Journal of Public Health 2005;95(11):1917–1922.
Martin A, Jones A, Mugford M, Shemilt I, Hancock R, Wittenber R. Methods used to identify and measure resource use in economic evaluations: a systematic review of questionnaires for older people. Health Economics 2012;21(8):1017–1022.
Mujica Mota R, Lorgelly PK, Mugford M, Toroyan T, Oakley A, Laing G, Roberts, I. Out-of-home day care for families living in a disadvantaged area of London: Economic evaluation alongside a RCT. Child: Care, Health & Development 2006;32(3):287-302.
Newall AT, Mark J, Beutels P. Economic evaluations of childhood influenza vaccination: A critical review. Pharmacoeconomics 2012;20(8):647-660.
Ozminkowski RJ, Goetzel RZ. Getting closer to the truth: Overcoming research challenges when estimating the financial impact of worksite health promotion programs. American Journal of Health Promotion 2001;15(5):289.295.
Pelletier KR. A review and analysis of the clinical- and cost-effectiveness studies of comprehensive health promotion and disease management at the worksite. American Journal of Health Promotion 2001;16(2):107-116.
Sperber D, Shiell A, Fyie K. Cost effectiveness of a law banning cellular-phone use by drivers. Health Economics 2010;19(10):1212–1225.
† Note that these categories are meant to be loosely interpreted as in many cases, publications are applicable to more than one category.
- Rush B, Shiell A. Annotated Bibliography of Economic Evaluations of Public Health Interventions (1990 - 2001) [ PDF (1,316 KB) - external link ]. Centre for Health and Policy Studies, University of Calgary; 2002.
- Please refer to the methods section of this bibliography for the criteria used to guide the inclusion of illustrative examples.