- Open Access
- Open Peer Review
Can cost-effectiveness results be combined into a coherent league table? Case study from one high-income country
Population Health Metricsvolume 17, Article number: 10 (2019)
Doubts exist around the value of compiling league tables for cost-effectiveness results for health interventions, primarily due to methods differences. We aimed to determine if a reasonably coherent league table could be compiled using published studies for one high-income country: New Zealand (NZ).
Literature searches were conducted to identify NZ-relevant studies published in the peer-reviewed journal literature between 1 January 2010 and 8 October 2017. Only studies with the following metrics were included: cost per quality-adjusted life-year or disability-adjusted life-year or life-year (QALY/DALY/LY). Key study features were abstracted and a summary league table produced which classified the studies in terms of cost-effectiveness.
A total of 21 cost-effectiveness studies which met the inclusion criteria were identified. There were some large methodological differences between the studies, particularly in the time horizon (1 year to lifetime) but also discount rates (range 0 to 10%). Nevertheless, we were able to group the incremental cost-effectiveness ratios (ICERs) into general categories of being reported as cost-saving (19%), cost-effective (71%), and not cost-effective (10%). The median ICER (adjusted to 2017 NZ$) was ~ $5000 per QALY/DALY/LY (~US$3500). However, for some interventions, there is high uncertainty around the intervention effectiveness and declining adherence over time.
It seemed possible to produce a reasonably coherent league table for the ICER values from different studies (within broad groupings) in this high-income country. Most interventions were cost-effective and a fifth were cost-saving. Nevertheless, study methodologies did vary widely and researchers need to pay more attention to using standardised methods that allow their results to be included in future league tables.
Given the constraints on health sector resources and the large demand for expanded or new health sector interventions, it is important to determine which interventions get the best value for money out of the limited resources available. As part of this prioritisation, cost-effectiveness is an important criterion for policymakers to consider, although it should be considered alongside other issues such as health equity, the size of the health gain, the timing of health gains and costs, issues around intervention feasibility and public acceptability, etc.
Previous international work has generated league tables for intervention cost-effectiveness from studies in multiple countries [1,2,3,4], for specific countries (e.g., the USA  and Australia [6, 7]), and for specific topic domains such as cancer care  and dietary sodium reduction . All of this work suggests that there are many cost-saving and cost-effective interventions that policymakers could adopt or enhance use of. Nevertheless, there are often method differences between studies that may limit the coherence of such league tables .
The country in our case study presented here (New Zealand (NZ)) is a fairly typical high-income country in which demand for health services is constantly increasing relative to available resources. However, New Zealand does have a single national agency that successfully constrains rising pharmaceutical prices (PHARMAC) and conducts cost-utility analyses as part of its decision-making processes. But elsewhere, in the health sector, there is no agency that routinely does such analyses and systematically prioritises health sector interventions. There is also no government-wide threshold for when gaining a quality-adjusted life-year (QALY) is deemed “cost-effective” or the relative importance of gaining QALYs in different populations as part of reducing health inequalities (given the health gaps between Māori (indigenous) and non-Māori in this country).
Despite this, a league table has been published of 21 pharmaceutical interventions in New Zealand . This work reported one intervention that was cost-saving and 20 with cost-effectiveness ratios ranging from NZ$771 to $142,000 per QALY (with a median value of the latter group of $9290, and mean of $25,300 [with these values calculated by us]). Other New Zealand-specific league tables include one for 12 dietary sodium reduction interventions (of which all but one were cost-saving ) and one for five tobacco control interventions (all cost-saving ). We have also developed an online interactive league table  that encompasses all the interventions modelled by the Burden of Disease Epidemiology, Equity and Cost-Effectiveness Programme (BODE3) Programme (covering tobacco control, nutrition, injury prevention, cancer prevention, and cancer treatment and management). However, in this particular study, we aimed to identify if other published cost-effectiveness analyses (CEAs) performed for New Zealand by different research teams could be included in a single reasonably coherent league table.
We first aimed to identify published New Zealand-specific CEAs that meet all of the following criteria: (i) the study was published in a peer-reviewed journal in the time-period 1 January 2010 to 8 October 2017; (ii) the study results included one of the following metrics: a cost per QALY or life-year (LY) gained or a cost per disability-adjusted life-year (DALY) averted; (iii) the study involved New Zealand epidemiological or cost data (even if New Zealand data was part of a combined study with other countries); and (iv) the study was not an output of the BODE3 Programme, since results from this programme all use a standard methodology and are detailed in an online interactive league table  (with a journal article about this league table pending).
Threshold for cost-effectiveness
A cost-effectiveness threshold from a policymaker's perspective will vary depending on what (dis)investment opportunities are being considered and what new (or requirement to reduce) funding is available, at a given time for a given investment horizon. Therefore, funding agencies usually do not have an official threshold—including in New Zealand. Nevertheless, as a guide, we note that on occasions, New Zealand Government agencies have quoted a value from the New Zealand Treasury of $38,110 per QALY (e.g., as per a Ministry of Health Report ), which is close to the World Health Organization guidance  of the per capita gross domestic product (GDP) of a country per QALY gained, i.e., $45,000 in NZ$ 2011 ).
Adjustment to NZ$ 2017
After abstracting the key information from each study, we then scaled the incremental cost-effectiveness ratio (ICER) values to 2017 NZ$ using the consumer price index (CPI) and purchasing power parity (PPP) (when translating from other currencies).
A total of 21 cost-effectiveness studies were identified and their key characteristics are summarised in Table 2 (with more detailed summary data and commentary in Additional file 1 Tables A2, A4, and A5 in the Additional file 1). Of these, the reported incremental cost-effectiveness ratios (ICERs) suggest that 19% had cost-saving interventions, 71% had main results that can be classified as cost-effective, and 10% were not cost-effective using New Zealand GDP per capita as a threshold. The median value (2017 NZ$) was $5080 per QALY/DALY/LY as per the values in Table 2 and taking the first figure if multiple values are shown.
Table 3 shows additional details relating to the study methods including the variable discount rates (range 0 to 10%). Only 57% of the studies used a 3.0% or 3.5% discount rate (with 3.0% being the internationally recommended rate , but within New Zealand, 3.5% is recommended by the Pharmaceutical Management Agency (PHARMAC; the pharmaceutical and device funding agency for the New Zealand Government)). QALYs were used in 76% of the studies and around half (48%) used a lifetime time horizon. However, 24% used a very short time horizon of only 12 months (i.e., usually the length of the trial upon which the health economic analysis was based). Most studies used a health system perspective (86%), with the rest being a societal one (two studies had both). Only three of the studies included productivity losses, and none include greenhouse gas emissions, even though a number of interventions involved dietary changes and transport costs.
The interventions studied were most commonly for primary prevention (38%), followed by treatment/management (33%) and then secondary prevention/screening (29%) (Table 2). Most of the interventions involved targeting a particular population group (90%) and were voluntary (95%), i.e., not involving regulations or taxes. Disease and condition topics were diverse, but 24% were related to cardiovascular disease and the next most common category was cervical cancer.
The search strategy also identified five CBAs (which did not have a cost-effectiveness component) and 12 other types of CEAs (albeit not using the QALY/DALY/LY metrics) (Table A3 in Additional file 1). Of the CBAs, only one included a cost-effectiveness component (i.e., cost-per-fall prevented ). Of the CEAs, only 17% (2/12) used a cost-per-death prevented, with the rest using other health metrics.
Main findings and interpretation
It seemed possible to produce a reasonably coherent league table for the ICER values from different studies in this high-income country. However, given the method differences, such a league table is probably only useful as a general guide in terms of broad categorisations of “likely cost-saving”, “likely to be cost-effective”, and “unlikely to be cost-effective”. Indeed, relatively small differences between ICER results (e.g., less than $20,000 per QALY/DALY averted, or less than a twofold difference in ICER) may not be particularly critical to informed prioritisation of interventions (given the many other considerations needed in policymaking such as the size of the health gain, the size of the costs/cost-savings, impact on health inequalities, and intervention feasibility). However, differences larger than these, especially if premised on expert consideration of the assumptions and context of each study being compared, probably make the simple league table comparisons outlined in this study potentially useful for policymaking or at least guiding prioritisation around further research efforts.
The high levels of methodological diversity in the identified CEAs suggest the need for researchers to strive harder for methodological compatibility of their work, e.g., with the use of the ISPOR guidelines/CHEERS checklist  and other internationally agreed best practice health economic approaches . This would allow their study results to be more readily integrated into methodologically compatible league tables. Similarly, if researchers perform CBAs, they should ideally also attempt to produce results in terms of cost per QALY gained from both a societal and health sector perspective.
The overall pattern of results of many cost-saving and cost-effective interventions was reasonably compatible with other league table work for this particular high-income country (the league tables in these three publications [11,12,13] and the online league table developed by BODE3 Programme ). They are also broadly compatible with Australian league table work [6, 7] and other international work on league tables—which has also identified both cost-saving and highly cost-effective interventions (see Background ). This work collectively should therefore be considered by policymakers to guide research priorities and to inform at least the cost-effectiveness aspect of future intervention selection. Similarly, the league table work aimed at low and middle-income countries can sometimes identify interventions that could be considered for high-income countries like New Zealand (e.g., polypills for cardiovascular disease prevention, dietary salt reduction policies, and regulation around food advertisements) .
Study strengths and limitations
A strength of this study is that it was the first attempt to produce a league table of interventions covering methodologically diverse health interventions for this particular high-income country. It also included relatively few studies that had industry funding (Table 2) which may reduce bias in the results . Nevertheless, this work has a number of limitations as detailed below.
Firstly, the search strategy might have missed some published studies given that occasionally cost-effectiveness results might only be found in appendicised information and hence are not readily identified in literature searches. Publication bias may have resulted in some studies not appearing in the peer-reviewed literature if authors or journal editors/reviewers considered the results to be of relatively little interest to policymakers (e.g., perhaps for studies which reported very cost-ineffective interventions).
Secondly, there are limitations with the threshold we used for categorising cost-effectiveness (in terms of GDP per capita as per the WHO CHOICE approach ). Such an approach has been criticised in the literature (e.g., in several articles [44,45,46]), with a particular weakness being that it is not linked to the shadow price of a health system’s budget constraint. Nevertheless, its use here can be justified given that it provides some link to real-world resourcing capacity and because of the absence of agreed alternative options proposed for this particular case study country.
Thirdly, this study did not attempt to include the components of the size of the health gain and cost impacts in the league table because this is very challenging due to the need to allow for varying population sizes, discerning total population and target population focus, and standardising the time horizon. There was also still variation in the outcome measures (i.e., QALY/DALY/LY) and, of course, the size of these varies with the different discount rates used (hence differentially impacting the ICER results).
Fourthly, more sophisticated critiques of the identified studies are possible (e.g., applying the CHEERS checklist  or detailed comparisons with state-of-the-art guidelines for CEAs ). However, this was not justified given the results in Table 3 that already show substantive deviation from established guidelines for many of the studies (e.g., in the discount rate).
It was possible to produce a reasonably coherent league table for the ICER values from different studies (within broad groupings) in this high-income country. Most interventions were cost-effective and a fifth were cost-saving. Nevertheless, study methodologies did vary widely and researchers need to pay more attention to using standardised methods that will allow their results to be included in future league tables.
Availability of data and materials
All data used in this report are available on request to the authors upon reasonable request.
Incremental cost-effectiveness ratio;
Chapman RH, Stone PW, Sandberg EA, Bell C, Neumann PJ. A comprehensive league table of cost-utility ratios and a sub-table of “panel-worthy” studies. Med Decis Making. 2000;20(4):451–67.
Cohen JT, Neumann PJ, Weinstein MC. Does preventive care save money? Health economics and the presidential candidates. N Engl J Med. 2008;358(7):661–3.
Neumann PJ, Rosen AB, Greenberg D, Olchanski NV, Pande R, Chapman RH, Stone PW, Ondategui-Parra S, Nadai J, Siegel JE, et al. Can we better prioritize resources for cost-utility research? Med Decis Making. 2005;25(4):429–36.
Horton S, Gelband H, Jamison D, Levin C, Nugent R, Watkins D. Ranking 93 health interventions for low- and middle-income countries by cost-effectiveness. PLoS One. 2017;12(8):e0182951.
Tengs TO, Adams ME, Pliskin JS, Safran DG, Siegel JE, Weinstein MC, Graham JD. Five-hundred life-saving interventions and their cost-effectiveness. Risk Anal. 1995;15(3):369–90.
Vos T, Carter R, Barendregt J, et al. Assessing cost-effectiveness in prevention (ACE-Prevention): final report: University of Queensland and Deakin University, 2010.
Dalziel K, Segal L, Mortimer D. Review of Australian health economic evaluation - 245 interventions: what can we say about cost effectiveness? Cost Eff Resour Alloc. 2008;6:9.
Greenberg D, Earle C, Fang CH, Eldar-Lissai A, Neumann PJ. When is cancer care cost-effective? A systematic overview of cost-utility analyses in oncology. J Natl Cancer Inst. 2010;102(2):82–8.
Schorling E, Niebuhr D, Kroke A. Cost-effectiveness of salt reduction to prevent hypertension and CVD: a systematic review. Public Health Nutr. 2017;20(11):1993–2003.
Mauskopf J, Rutten F, Schonfeld W. Cost-effectiveness league tables: valuable guidance for decision makers? Pharmacoeconomics. 2003;21(14):991–1000.
Metcalfe S, Dougherty S, Brougham M, Moodie P. PHARMAC measures savings elsewhere to the health sector. N Z Med J. 2003;116(1170):U362.
Nghiem N, Blakely T, Cobiac LJ, Cleghorn CL, Wilson N. The health gains and cost savings of dietary salt reduction interventions, with equity and age distributional aspects. BMC Public Health. 2016;16(1):423.
Nghiem N, Cleghorn CL, Leung W, Nair N, van der Deen FS, Blakely T, Wilson N. A national quitline service and its promotion in the mass media: modelling the health gain, health equity and cost-utility. Tob Control. 2018;27:434–41.
University of Otago & University of Melbourne. ANZ-HILT: Australia and New Zealand Health Intervention League Table (Vers 2.0) 2019 [Available from: https://league-table.shinyapps.io/bode3/].
Ministry of Health. Appendix: Background Information: New Zealand’s Tobacco Control Programme. Wellington: Ministry of Health; 2016. https://www.health.govt.nz/system/files/documents/pages/appendix-8-april-background-info-tobacco-control-programme.pdf.
WHO: Choosing interventions that are cost effective (WHO-CHOICE). Geneva: World Health Organization; 2012. Available from: http://www.who.int/choice/costs/en.
Kvizhinadze G, Wilson N, Nair N, McLeod M, Blakely T. How much might a society spend on life-saving interventions at different ages while remaining cost-effective? A case study in a country with detailed data. Popul Health Metr. 2015;13:15.
Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, Kuntz KM, Meltzer DO, Owens DK, Prosser LA, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA. 2016;316(10):1093–103.
Keall MD, Pierse N, Howden-Chapman P, Guria J, Cunningham CW, Baker MG. Cost-benefit analysis of fall injuries prevented by a programme of home modifications: a cluster randomised controlled trial. Inj Prev. 2017;23:22–6.
Leung W, Ashton T, Kolt GS, Schofield GM, Garrett N, Kerse N, Patel A. Cost-effectiveness of pedometer-based versus time-based Green Prescriptions: the Healthy Steps Study. Aust J Prim Health. 2012;18(3):204–11.
O'Keeffe KM, Gander PH, Scott WG, Scott HM. Insomnia treatment in New Zealand. N Z Med J. 2012;125(1349):46–59.
Scott GW, Scott HM, O'Keeffe KM, Gander PH. Insomnia - treatment pathways, costs and quality of life. Cost Eff Resour Alloc. 2011;9:10.
Lew JB, Simms K, Smith M, Lewis H, Neal H, Canfell K. Effectiveness modelling and economic evaluation of primary HPV screening for cervical cancer prevention in New Zealand. PLoS One. 2016;11(5):e0151619.
Friedman J, Reed P, Sharplin P, Kelly P. Primary prevention of pediatric abusive head trauma: a cost audit and cost-utility analysis. Child Abuse Negl. 2012;36(11-12):760–70.
Gander P, Scott G, Mihaere K, Scott H. Societal costs of obstructive sleep apnoea syndrome. N Z Med J. 2010;123(1321):13–23.
Lake RJ, Horn BJ, Dunn AH, Parris R, Green FT, McNickle DC. Cost-effectiveness of interventions to control Campylobacter in the New Zealand poultry meat food supply. J Food Prot. 2013;76(7):1161–7.
Webb M, Fahimi S, Singh GM, Khatibzadeh S, Micha R, Powles J, Mozaffarian D. Cost effectiveness of a government supported policy strategy to decrease sodium intake: global analysis across 183 nations. BMJ. 2017;356:i6699.
Maddison R, Pfaeffli L, Whittaker R, Stewart R, Kerr A, Jiang Y, Kira G, Leung W, Dalleck L, Carter K, et al. A mobile phone intervention increases physical activity in people with cardiovascular disease: results from the HEART randomized controlled trial. Eur J Prev Cardiolog. 2015;22(6):701–9.
Dalziel K, Segal L, Katz R. Cost-effectiveness of mandatory folate fortification v. other options for the prevention of neural tube defects: results from Australia and New Zealand. Public Health Nutr. 2010;13(4):566–78.
Sopina E, Ashton T. Cost-effectiveness of a cervical screening program with human papillomavirus vaccine. Int J Technol Assess Health Care. 2011;27(4):290–7.
Panattoni L, Brown PM, Te Ao B, Webster M, Gladding P. The cost effectiveness of genetic testing for CYP2C19 variants to guide thienopyridine treatment in patients with acute coronary syndromes: a New Zealand evaluation. Pharmacoeconomics. 2012;30(11):1067–84.
Simms KT, Smith MA, Lew JB, Kitchener HC, Castle PE, Canfell K. Will cervical screening remain cost-effective in women offered the next generation nonavalent HPV vaccine? Results for four developed countries. Int J Cancer. 2016;139(12):2771–80.
Te Ao B, Brown P, Fink J, Vivian M, Feigin V. Potential gains and costs from increasing access to thrombolysis for acute ischemic stroke patients in New Zealand hospitals. Int J Stroke. 2015;10(6):903–10.
Te Ao BJ, Brown PM, Feigin VL, Anderson CS. Are stroke units cost effective? Evidence from a New Zealand stroke incidence and population-based study. Int J Stroke. 2012;7(8):623–30.
Keall MD, Pierse N, Howden-Chapman P, Cunningham C, Cunningham M, Guria J, Baker MG. Home modifications to reduce injuries from falls in the Home Injury Prevention Intervention (HIPI) study: a cluster-randomised controlled trial. Lancet. 2015;385(9964):231–8.
Milne RJ, Hockey H, Rea H. Long-term air humidification therapy is cost-effective for patients with moderate or severe chronic obstructive pulmonary disease or bronchiectasis. Value Health. 2014;17(4):320–7.
Rush E, Obolonkin V, McLennan S, Graham D, Harris JD, Mernagh P, Weston AR. Lifetime cost effectiveness of a through-school nutrition and physical programme: Project Energize. Obes Res Clin Pract. 2014;8(2):e115–22.
Pinto D, Robertson MC, Abbott JH, Hansen P, Campbell AJ, Team MOAT. Manual therapy, exercise therapy, or both, in addition to usual care, for osteoarthritis of the hip or knee. 2: economic evaluation alongside a randomized controlled trial. Osteoarthritis Cartilage. 2013;21(10):1504–13.
Carrasco LR, Lee VJ, Chen MI, Matchar DB, Thompson JP, Cook AR. Strategies for antiviral stockpiling for future influenza pandemics: a global epidemic-economic perspective. J R Soc Interface. 2011;8(62):1307–13.
Harris A, Cooper BA, Li JJ, Bulfone L, Branley P, Collins JF, Craig JC, Fraenkel MB, Johnson DW, Kesselhut J, et al. Cost-effectiveness of initiating dialysis early: a randomized controlled trial. Am J Kidney Dis. 2011;57(5):707–15.
Leung W, Roberts V, Gordon LG, Bullen C, McRobbie H, Prapavessis H, Jiang Y, Maddison R. Economic evaluation of an exercise-counselling intervention to enhance smoking cessation outcomes: The Fit2Quit trial. Tob Induc Dis. 2017;15 (1) (no pagination)(21).
Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, Augustovski F, Briggs AH, Mauskopf J, Loder E, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)--explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health. 2013;16(2):231–50.
Bell CM, Urbach DR, Ray JG, Bayoumi A, Rosen AB, Greenberg D, Neumann PJ. Bias in published cost effectiveness studies: systematic review. BMJ. 2006;332(7543):699–703.
Marseille E, Larson B, Kazi DS, Kahn JG, Rosen S. Thresholds for the cost-effectiveness of interventions: alternative approaches. Bull World Health Organ. 2015;93(2):118–24.
Revill P, Walker S, Madan J, Ciaranello A, Mwase T, Gibb DM, Claxton K, Sculpher M. Using cost-effectiveness thresholds to determine value for money in low- and middle-income country healthcare systems: are current international norms fit for purpose? Working Paper. CHE Research Paper. Centre for Health Economics, University of York, York, 98; 2014.
Newall AT, Jit M, Hutubessy R. Are current cost-effectiveness thresholds for low- and middle-income countries useful? Examples from the world of vaccines. Pharmacoeconomics. 2014;32(6):525–31.
Funding of this work by the Health Research Council of New Zealand (16/443).
The authors received a grant from the Health Research Council of New Zealand (16/443).
Ethics approval and consent to participate
Consent for publication
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Table A1. Examples of the types of studies that were excluded from this review and league table development (albeit with some being briefly described in Table A3). Table A2. Additional commentary on the interventions in Table 2 of the main manuscript for the 21 New Zealand cost-effectiveness studies identified and published in the period 1 January 2010 to 8 October 2017 (ordered by decreasing cost-effectiveness, with additional details on each study in Table A4 and Table A5). Table A3. Additional cost-benefit analysis or other types of cost-effectiveness analyses identified for New Zealand in the period 1 January 2010 to 8 October 2017 (i.e., not using the QALY/DALY/LY metrics). Table A4. Summary of the methods characteristics of the 21 studies with cost-effectiveness analyses for New Zealand in the period 1 January 2010 to 8 October 2017 (included studies are ordered alphabetically by first author surname). Table A5. Results of the included studies for New Zealand in the period 1 January 2010 to 8 October 2017 (ordered alphabetically by first author surname). (DOCX 200 kb)