Deriving disability weights for the Netherlands: findings from the Dutch disability weights measurement study

Abstract

Background

The aims of this study were to establish national disability weights based on the health state preferences of a Dutch general population sample, examine the relation between results and respondent’s characteristics, and compare disability weights with those estimated in the European disability weights study.

Methods

In this cross-sectional study, a web-based survey was administered to a general population 18–75 years from the Netherlands. The survey included paired comparison questions. Paired comparison data were analysed using probit regression and located results onto the 0-to-1 disability weight scale using non-parametric regression. Bootstrapping was used to estimate 95% uncertainty intervals (95%UI). Spearman’s correlation was used to investigate the relation of probit regression coefficients between respondent’s characteristics.

Results

3994 respondents completed the questionnaire. The disability weights ranged from 0.007 (95%UI: 0.003–0.012) for mild distance vision impairment to 0.741 (95% UI: 0.498–0.924) for intensive care unit admission. Spearman’s correlation of probit coefficients between sub-groups based on respondent’s characteristics were all above 0.95 (p < 0.001). Comparison of disability weights of 140 health states that were included in the Dutch and European disability weights study showed a high correlation (Spearman’s correlation: 0.942; p < 0.001); however, for 76 (54.3%) health states the point estimate of the Dutch disability weight fell outside of the 95%UI of the European disability weights.

Conclusions

Respondent’s characteristics had no influence on health state valuations with the paired comparison. However, comparison of the Dutch disability weights to the European disability weights indicates that health state preferences of the general population of the Netherlands differ from those of other European countries.

Background

In the early 1990s, the burden of disease concept was introduced. The burden of disease quantifies health loss in a population and has proven indispensable for monitoring population health, identifying major risk factors of ill health and guiding policy decisions in the field of public health [1, 2].

A frequently used metric to capture the burden of disease is the disability-adjusted life year (DALY). The DALY is a health gap measure that summarizes health loss due to premature mortality, expressed in years of life lost (YLL), and health loss due to time spent living in a situation of non-optimal health, expressed in years lived with disability (YLD). Pivotal for the calculation of the YLD are disability weights, which reflect the relative severity of health consequences of a health state on a 0-to-1 scale and which are based on the health state valuations of a panel of judges, such as health experts or the general population [1,2,3,4,5].

The DALY has been used in large international studies, such as the Global Burden of Disease (GBD) study and the Global Health Estimates, and many national studies [6,7,8,9]. The Dutch National Institute for Public Health and the Environment (RIVM) was one of the first national health institutes to apply the DALY metric in a national foresight study [10, 11]. Rather than using the GBD 1990 disability weights, a new set of disability weights was established that could be applied in a national context and that was more refined with regards to disease stages and health state descriptions compared to the GBD 1990 disability weights [12]. The Dutch disability weights were determined using the same methodological approach that was used to establish the GBD 1990 disability weights. This approach consists of asking medical experts to evaluate health states with ranking and person trade-off techniques. Particularly the person trade-off is a complex task that has been criticized as having limited validity and reliability [13,14,15].

After publication of the GBD 1990 and 1997 Dutch disability weights study, views with regards to appropriate disability weight measurement methods changed. One of the most important changes was the shift from incorporating health preferences of medical experts to incorporating health preferences of members of the general population [16, 17]. One of the reasons for this shift is that burden of disease studies are used for priority setting in health and guiding health policy decisions. It is therefore important to incorporate general populations’ perceptions and health state preferences, rather than health experts’ perceptions and health state preferences.

Linked to this shift with regards to study population is the shift towards the use of less complex health state preference elicitation methods, as some members of a study population consisting of persons from the general population may have lower numeracy or literacy levels compared to health experts [16, 17]. This means that complex health state elicitation methods, such as the person trade-off, that were used in earlier disability weights measurement studies targeting health experts cannot be used in general population samples. A frequently used method that has shown to elicit high quality information on health state preferences of a representative sample of the general population is the paired comparison technique. The paired comparison technique is grounded in Thurstone’s Law of Comparative Judgment theoretical framework and it has adequate reliability and good validity [18,19,20,21].

Based on these insights, GBD researchers developed a new methodological approach to determine a set of global disability weights [22]. Subsequently, a slightly refined version of this methodological protocol was used to establish a European set of disability weights and several national sets of disability weights [23,24,25,26]. However, in the Netherlands, the set of disability weights used to quantify YLD in the Dutch national burden of disease studies has not been updated since 1997. This means that the disability weights used in Dutch national forecast studies do not represent the Dutch general populations’ perceptions and health state preferences are based on valuation techniques that have theoretical challenges.

The primary aim of this study was to obtain a set of national disability weights based on the health state preferences of a representative sample of members from the Netherlands. The secondary aims of this study were to examine the relation between results and socio-demographic and health characteristics such age, gender, highest attained level of education, and chronic disease status, as well as to compare the resulting disability weights with those estimated in the European disability weights measurement study.

Methods

Study design

This is a cross-sectional observational study. We administered a web-based survey to a cohort of members of the general population of the Netherlands. Data were collected between 1 and 26 June 2023. Ethical approval was obtained from the Erasmus MC Ethics Review Board (MEC-2023-0239).

Panel participants and eligibility criteria

Participants were recruited by Flycatcher Internet Research, a market research agency. Flycatcher Internet Research invited members from existing panels consisting of members of the general population residing in the Netherlands to complete the online survey. Inclusion criteria were: member of existing market research agency panel, aged 18–75 years, and sufficient command of the Dutch language. Exclusion criteria were age less than 18 years or older than 75 years. Information on age, gender, highest attained level of education, and region of residence of the individual Internet panel members was already known. Based on this information, Dutch panelists were invited to fill out the questionnaire to ensure national representativeness across age, gender, and highest attained level of education.

Health states lay descriptions

A total of 210 health states were evaluated, of which 156 health states were included in the GBD 2010, European, or Japanese disability weights measurement studies [22,23,24]; 47 were new; and seven were included for experimental purposes and were not part of the Dutch disability weights measurement study. For the health states originating from preceding disability weights measurement studies, we either used the same health state descriptions as included in those studies or we modified the health state descriptions based on the advice of disease experts. For the 47 new health states, new health state descriptions were developed under the guidance of medical expert(s), aligning with the design principles employed in the GBD 2013 disability weights study (i.e. brief lay descriptions of up to 70 words were constructed, using simple and non-clinical vocabulary explaining the main symptoms and functional limitations of each health state) [22, 27]. A complete listing of all health states, their origins and lay descriptions used in this study can be found in the Additional file 1 (page 3).

Health state elicitation technique

The paired comparison asks the participant to consider two hypothetical individuals (person A versus person B) with different health states and to indicate which person they regarded as healthier than the other (Additional file 1, page 2). Each respondent performed 18 paired comparison tasks. The health states depicted in these paired comparison tasks were drawn randomly, using a computer-generated algorithm from all available possible comparisons. We repeated the same pair of health states in the 2nd and 15th paired comparison questions, with health states presented in the same order (2nd question) and reverse order (15th question). This allowed us to evaluate the internal consistency and test re-test reliability of paired comparison responses.

Socio-demographic and health characteristics

The survey also included questions about socio-demographic characteristics (e.g. gender, age, highest level of education, and region of residence). The highest level of education achieved was categorized into three groups according to the International Standard Classification of Education (ISCED) 2011: ISCED 0–2 (“Low”), ISCED 3–4 (“Middle”), and ISCED 5–8 (“High”). This categorization is in line with Statistics Netherlands’ adoption of ISCED as the Dutch standard for measuring educational directions. Additionally, the survey included questions about health characteristics (e.g. chronic disease status). Chronic disease status was measured by the presence of up to 11 chronic conditions (i.e. asthma or chronic bronchitis, heart disease, stroke, diabetes, arthritis, severe back complaints, arthrosis, cancer, memory problems, depression or anxiety disorder, and/or other problems). The number of chronic diseases was categorized into three groups: “Zero”, “One”, “Two or more”.

The data capture system did not allow for missing values, meaning that participants were required to respond to all questions. It did not allow participants to adjust their responses (i.e. to go back in the questionnaire).

Statistical analysis

All statistical analyses were performed with R (version 4.1.0) and SPSS (version 28.0.1).

Socio-demographic and health characteristics:

Descriptive analyses were performed for socio-demographic and health characteristics data.

Test re-test analysis:

Paired comparison responses on the deliberate repetition of the 2nd and 15th questions were examined in the form of a test re-test analysis. The probability of choosing the same health state if the two health states were presented in the same order (2nd question) was calculated as the total number of consistent instances divided by the total number of responses. Similarly, the probability of choosing the same health state if the two health states were presented in reverse order (15th question) was calculated as the total number of inconsistent instances dived by the total number of responses. Inter-rater reliability was measured by Cohen’s Kappa (κ) [28, 29]; it allows for the assessment of agreement beyond what would be expected by chance by considering both the observed agreement and the expected agreement that would occur by chance. It ranges from − 1 to + 1 where values ≤ 0.20 indicate slight or no agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial agreement; and 0.81-1.0 almost perfect agreement. The level of agreement was examined by educational groups and regions.

Paired comparison responses:

Paired comparison data were first analysed based on the choice probabilities over all possible health state pairs. Response probabilities were ordered and plotted in a heatmap matrix. Paired comparison data were then analysed using probit regression analysis. A binary response variable was coded as 1 if the first health state in a pair was chosen as the healthier one; as -1 if the second health state in a pair was chosen as the healthier; and as 0 for health states other than the pair being considered. Probit regression of paired comparison responses provides estimates that capture the relative differences in valuation of health states and are on an arbitrary scale rather than on the 0-to-1 disability weight scale. Thus, an additional analytic step to anchor the resulting estimates onto the 0-to-1 disability weight scale was performed.

Anchor results from probit regression analysis:

To predict the resulting estimates of the probit regression on the 0-to-1 disability weight scale, a non-parametric regression model (loess) of the probit regression coefficients against the logit-transformed disability weights from the GBD 2013 disability weights study was ran [27]. Then, a bootstrapping approach with 1000 replicate samples with means defined by the predicted probit coefficients and variance by the standard deviation of the predicted probit coefficients was ran to estimate 95% uncertainty intervals (95%UI). On each bootstrap sample, a non-parametric model of the logit-transformed disability weights from the GBD 2013 disability weights study against the regression coefficients was fitted. An inverse logistic transformation was then applied to the mean predicted disability weights in order to obtain the Dutch disability weights on the 0-to-1 disability weight scale. Finally, the 95%UI were obtained from the corresponding distribution of the sampled disability weights.

Relation between probit coefficients and socio-demographic and health characteristics:

The relation of probit regression coefficients between age, gender, highest attained level of education, and chronic disease status was evaluated using the Spearman’s correlation. A correlation matrix was generated based on the probit coefficients associated with the above socio-demographic and health characteristics.

Comparison of the resulting disability weights to the European disability weights:

To test for differences between the Dutch and European disability weights [23], the Spearman’s correlation and the Wilcoxon signed-rank test was used.

Availability of code

The statistical code used to derive the Dutch disability weights is available on GitHub (https://github.com/periklisch/Dutch_Disability_Weights.git) and included in the Additional file 2.

Results

Study population

Table 1 shows the socio-demographic and health characteristics of the respondents, as compared to the whole population distribution in the Netherlands. A total of 3994 respondents completed the web-based questionnaire. Due to the participant recruitment and survey distribution methods, the response rate could not be calculated. The age-gender distribution of the Dutch disability weight cohort sample versus the national population can be found in the Additional file 1 (page 24). The age-gender-education distribution of the participants by regional level can also be found in the Additional file 1 (page 25).

Table 1 Socio-demographic and health characteristics (n = 3994)

Paired comparison

Figure 1 depicts a heatmap of the paired comparison response probabilities for the possible paired comparisons of 210 health states. Each cell in the heatmap represents the response probability for one pair of health states. Each colour represents the probability that the first health state in a pair comparison is chosen as the healthier outcome. Red cells correspond to probabilities less than 0.25; orange, yellow, and green cells correspond to probabilities between 0.25 and 0.75; and blue cells correspond to probabilities greater than 0.75. Please note that not all possible 210 × 210 pairs were evaluated by paired comparisons, which is indicated by some blanks in the figure. A completely smooth transition in colours from blue (upper left) to red (lower right) indicates a very small amount of measurement error and high internal consistency in paired comparison responses. Figure 1 does not show a completely smooth transition from blue to red.

Fig. 1

Response probabilities for paired comparisons

Of the repeated paired comparison question 48.1% were presented in same order and 51.9% in reverse order. The probability of choosing the same health state was higher when they were presented in same order (0.741, κ = 0.482) compared to when they were presented in reverse order (0.727, κ=-0.453). Test re-test analysis of the paired comparison response probabilities by Dutch-pooled and Dutch-regions and educational level can be found in the Additional file 1 (page 26).

Disability weights

Table 2 shows the estimated disability weights with 95%UI. The estimated disability weights ranged from 0.007 (95%UI: 0.003–0.012) for mild distance vision impairment to 0.741 (95% UI: 0.498–0.924) for intensive care unit admission. Of the 210 health states, 57 had a disability weight lower than 0.05. The lowest disability weights were associated with mild health states (e.g. mild neck pain: 0.055; 95%UI: 0.036–0.081), while the highest disability weights were associated with moderate (e.g. moderate neck pain: 0.146; 95%UI: 0.097–0.210) and severe (e.g. severe neck pain: 0.279; 95%UI: 0.191–0.377) health states.

Table 2 Estimated disability weights with uncertainty intervals

Relation between probit coefficients and socio-demographic and health characteristics

Figure 2 illustrates the Spearman’s correlation of the probit coefficients between sub-groups based on education level, gender, age category, and chronic disease status. Spearman’s correlations were all above 0.95 (education level r_s: 0.953–0.995; age categories r_s: 0.956–0.978; gender r_s: 0.978; and chronic disease status r_s: 0.980, all p < 0.001), Additional file 1 (pages 27–29).

Fig. 2

Correlation of probit coefficients by highest attained level of education, gender, age category, and chronic disease status in the Dutch disability weights cohort sample

Comparison to the European disability weights

Comparison of the disability weights of the 140 health states that were included in both the Dutch and European disability weights study showed a high correlation (Spearman’s correlation: 0.942; p < 0.001). The median disability weight of all 140 health states did not differ significantly between the Dutch and the European disability weight study (median of Dutch disability weights: 0.103, interquartile range (IQR) 0.041–0.291; median of European disability weights: 0.122, IQR 0.044–0.281); p = 0.972). However, for 76 (54.3%) of the 140 health states the point estimate of the Dutch disability weight fell outside of the 95%UI of the European disability weights. For 33 (23.6%) health states, the Dutch disability weights were lower than the lower bound. Approximately one in three of the neurological and injury health states was significantly lower than the European disability weights (Additional file 1, page 30). For 43 (30.7%) health states, the Dutch disability weights were higher than the higher bound of the 95%UI of the European disability weights. All of the diabetes, digestive, and genitourinary disease health states and four in five cardiovascular and circulatory diseases were significantly higher than the European disability weights (Additional file 1, page 30).

Discussion

This study determined disability weights based on health preferences of a Dutch general population sample. The resulting disability weights ranged from 0.007 (95%UI: 0.003–0.012) for mild distance vision impairment to 0.741 (95% UI: 0.498–0.924) for intensive care unit admission. Diseases with multiple stages in terms of severity (e.g. traumatic brain injury and hearing loss) had a logical ranking with lowest disability weights attributed to mild stages and the highest disability weights to severe stages.

However, the ranking of certain health states with multiple severity stages seems counterintuitive. For example, the disability weight for profound intellectual disability (0.076, 95%UI: 0.050–0.108) was estimated to be lower than the disability weight for severe intellectual disability (0.089, 95%UI: 0.060–0.125) and moderate intellectual disability (0.088, 95%UI: 0.061–0.125). A possible explanation for this difference may be that a modified health state description for profound intellectual disability was used. This may elicit differences in disability weights, and it is likely that the major functional consequences and symptoms associated with moderate and severe versus profound intellectual disability were not fully understood by the Dutch general population. Another noteworthy observation was the difference between moderate (0.293, 95% 0.204–0.394) and severe (0.266, 95%UI: 0.183–0.362) dementia. It should be noted that a similar observation was identified in the European NOISE disability weights measurement study [26]. This underscores the importance of adapting brief lay descriptions for dementia in future disability weights measurement studies.

We found high correlations of the probit coefficients between sub-groups based on educational level, age category, gender, and region of residence. This indicates that socio-demographic characteristics had no influence on health state valuations with the paired comparison. These findings are in agreement with those of previous disability weight measurement studies [25, 32, 33].

Comparison of the Dutch disability weights to the European disability weights showed that for slightly more than half of the health states the value of the Dutch and European disability weights differed significantly. Notably many health states in disease categories cancer, neurological disorders, injury and other health states had a significantly lower disability weight compared to the European disability weights. On the other hand, many of the health states in disease categories diabetes, digestive, and genitourinary disease health states and cardiovascular and circulatory diseases and infectious disease had a significantly higher disability weight compared to the European disability weights [23]. These differences in disability weights between the Dutch and European studies may be due to differences in health state preferences across countries. The study population of the European disability weight study consisted of members of the general population of four countries, namely Hungary, Italy, the Netherlands and Sweden [23]. Comparison of the responses on the paired comparison task showed that there was more variation in health states valuation between countries than between other respondent characteristics [32]. Particularly the correlation of the probit coefficients between the Netherlands and Italy, Hungary and Sweden was lower compared to the correlations between Hungary, Italy and Sweden. This may indicate that health state preferences of the general population of the Netherlands differ from those of other European countries.

Another reason for differences in health state preferences between the Dutch and European study could be that health state preferences may have changed over time. The data of the European disability weights study were collected in 2013, ten years before data collection of the Dutch disability weight study. In the meantime, the COVID-19 pandemic occurred and in January 2023, 8.6 million COVID-19 cases had been recorded in the Netherlands (17.6 million inhabitants) [34]. Recent experience with a COVID-19 infection themselves or in a significant other may have had an effect on the respondent’s health state preferences. The relatively high percentage of infectious disease and respiratory disease health states with a significantly higher disability weight compared to the European disability weights may point in that direction. However, the link between experiencing COVID-19 infection and changes in health state preferences with regards to diabetes, digestive, and genitourinary disease and cardiovascular and circulatory diseases health states is less obvious. In addition, the findings of our study showed that having one or more chronic disease(s) had no influence on respondents’ health state valuations.

Comparison of the disability weights derived in this study to those of the 1997 Dutch disability weights study shows that the 1997 Dutch disability weights almost cover the full 0-to-1 disability weight scale and are on average higher [12]. The values of the 1997 Dutch disability weights range from 0 (gingivitis) to 0.93 (terminally ill), whereas the disability weight established this study range from 0.007 to 0.741. This can be explained by the differences in health state valuation techniques and statistical methods that were used to derive the disability weights in the two studies. As a result, the values of the disability weights of the two Dutch disability weight studies are incomparable and the disability weights cannot be used together in the same study [17].

A strength of our study is that the disability weights are tailored to the epidemiological data that are used to determine the burden of disease of diseases, injuries and risk factors in the Netherlands. This means that for several diseases disability weights were established on a more detailed level (e.g. multiple stages for asthma, diabetes, autism) compared to the GBD 2013 disability weights. Furthermore, data were collected via a web-based survey among a panel of respondents that were representative for the adult Dutch population in terms of age, gender, and educational level. Although collecting data via a web-based survey may affect data quality person [34], our study showed substantial test-retest reliability of the paired comparison responses, indicating high quality of responses.

A limitation of this study is that we used the GBD 2013 disability weights to anchor the Dutch disability weights, rather than population health equivalence questions as was done in GBD 2010 and 2013 study and the Chinese disability weight study [22, 25, 27]. As result, the disability weights established in this study are not based on trade-offs between non-fatal and fatal outcomes of the Dutch general population, but on the GBD study population, which covered many countries from different world regions [27]. However, previous studies have shown that the quality of population health equivalence data derived via a web-based survey among a sample of the general population is low and we therefore choice for this alternative anchoring strategy [23, 24].

In conclusion, we observed logical ranking of the disability weights that were established in this study, with lowest disability weights attributed to mild health states and the highest disability weights to severe health states. Socio-demographic and health characteristics had no influence on health state valuations with the paired comparison. However, comparison of the Dutch disability weights to the European disability weights indicates that health state preferences of the general population of the Netherlands differ from those of other European countries.