Error and bias in under-5 mortality estimates derived from birth histories with small sample sizes

Under-5 mortality, the probability of death before age 5 (denoted ₅ q ₀), is an important overall indicator of child health. In countries without functioning systems to continuously register births and deaths, estimates of under-5 mortality are generally derived from survey and/or census data, particularly in the form of birth histories where women are asked for information about the survival of their children.

Birth history data are routinely used for estimating mortality at the national level. It is often of interest, however, to estimate under-5 mortality at a subnational level or to stratify by some other characteristic (e.g., income or maternal education). Subnational or stratified analyses with survey data are complicated by small sample sizes: in the case of surveys in particular, the sample size for a given subnational unit or stratum is often quite small, and it is not apparent if the estimates derived from these limited data are useful. While a number of subnational analyses with birth history data have been undertaken using census data[1–3] where small sample sizes are less of a concern, existing subnational mortality estimates using survey data tend to be at a relatively coarse level (often provinces or regions) to avoid small samples[2, 4].

Two different types of birth histories are routinely collected. In a complete birth history (CBH), women are asked for information about the date of birth and, if applicable, the age at death of each child they have given birth to. Because complete birth histories contain information about dates and ages for individual children they allow for direct calculation of under-5 mortality. In a summary birth history (SBH), women are asked only about the total number of children they have given birth to and the number of these children who are still alive. Summary birth histories lack information about dates and ages for individual children and demographic models must be employed to estimate under-5 mortality from these data. Although complete birth histories are more straightforward to analyze they are less frequently undertaken than summary birth histories, which are far less labor-intensive and time-consuming to collect.

In this paper, we aimed to determine how much error and/or bias can be expected in under-5 mortality estimates derived from both types of birth histories at various small sample sizes. To this end, we carried out a data-based simulation study using Demographic and Health Survey (DHS) data wherein we treated each survey as a population with known mortality and sampled from this population to mimic surveys with small sample sizes. We examined how estimates derived from summary birth history data and complete birth history data (analyzed using several alternative methods) compared in terms of error and bias at increasingly small sample sizes. Further, we performed stratified analyses to explore in more detail how the performance of each method relates to the underlying true level of mortality and the time prior to data collection.

This analysis suggests that all methods of analyzing birth history data perform poorly at sample sizes of fewer than 100 women, with large expected errors and, for some methods, noticeable downward bias. There are large differences in performance between models, however, and even at higher sample sizes (500 and 1000 women), the magnitude of the expected error for many methods is still unacceptably high.

Unfortunately, there is not an obvious ‘best’ method. Overall, summary birth histories and moving window complete birth history methods with very long windows provide estimates with the smallest magnitude error and least bias, especially at the smallest sample sizes. In the case of the former, the better performance may be a result of the models that underlie the method which could, to some extent, constrain more outlying estimates from being generated. In the case of the latter, the better performance, particularly in terms of the expected magnitude of the error, is likely a result of the increased pooling of information across time. These same methods, however, do not perform uniformly across levels of mortality, and in particular, they tend to overestimate in low-mortality settings and underestimate in high-mortality settings. It is likely that the same strengths that underlie the better performance of these models overall are also at least partly responsible for these pitfalls. In the case of the summary birth histories, the models may be constraining final estimates too closely to the mean, biasing unusually low or unusually high estimates toward this mean. In the case of the moving window complete birth history methods, the increased pooling also runs the risk of smoothing out real trends in mortality and biasing the final estimates. Similarly, the moving window complete birth history methods with very long windows do not perform uniformly across time periods prior to the survey: they tend to overestimate in more recent periods and underestimate in more distant periods, and the magnitude of the error increases noticeably the earlier the estimate. Under-5 mortality has generally decreased with time, so it is likely that differences in the level of mortality at different time periods are at least partially driving the differences in performance observed in this analysis at different time periods (the reverse is also possible). Beyond this effect, however, it is also likely that the magnitude of the error is larger in earlier time periods because only the oldest women captured in the survey report children that far in the past and consequently the total number of children observed is smaller in earlier time periods compared to later time periods. The methods with less smoothing (i.e., the complete birth history methods with period or window lengths of no more than five years) are far less problematic with respect to differential bias by level of mortality or time prior to survey, but the magnitude of overall error from these methods is much larger than the other methods.

The results of this analysis suggest that the birth history methods considered are of limited utility for estimating mortality in small samples and, in particular, for making meaningful comparisons among geographic units or strata. Given the value of these types of estimates, however, investment in other data sources may be warranted. In particular, sample registration schemes may be a useful alternative to both surveys, with the problems enumerated here, and full vital registration systems, which are expensive and technically challenging to maintain. Alternatively, research into adapting existing small area methods frequently used in epidemiology and other fields[10, 11] for use with birth histories could prove useful. These models explicitly account for unusually high sampling error in estimates derived from small samples and attempt to overcome this challenge by exploiting spatial and temporal relatedness. Several authors have already used birth history data to inform these models, though the focus of these analyses has generally been on the relationship between other factors and mortality and not on prediction of mortality levels for specific areas or subgroups[12–16].

This analysis has several limitations. The stratified analyses by mortality level and time prior to survey do not control for each other, making it difficult to conclusively disentangle the two effects. Further, birth histories, like all survey data, are subject to a number of data errors, including, among others, recall bias and age misreporting. We treat the reported population in each survey as truth and don’t consider the additional effect on error or bias that any of these errors could introduce. It is well documented that these types of errors can impact the reliability of mortality estimates, but future research could consider specifically how these errors interact with the problems due to sample size explicitly considered here. Microsimulation–where synthetic populations are created by simulating births and deaths given set mortality and fertility schedules–could provide useful mechanisms for more fully exploring the issues described here.

Nonetheless, this study boasts several strengths. The use of empirical data, rather than simulated populations, ensures that the mortality and fertility relationships are realistic and representative of the types of scenarios where birth history data are most likely to be collected. Additionally, in contrast to previous research[17, 18] which has examined errors in birth history estimates and compared different methods of analyzing birth history data, we estimate error by comparing to a true gold standard (in this case the full sample) rather than using statistical techniques such as the Jackknife to estimate error. Finally, this study compares a large number of different methods for analyzing available data and makes explicit the comparison between these methods at different sample sizes, which should prove useful to analysts deciding between different methods given a particular dataset.

Overall, the results of this analysis suggest that birth histories in all but the largest of surveys are of limited utility for making subnational estimates or estimates across many strata. Censuses may be more useful for this purpose, having much larger sample sizes, but generally only include summary birth history information if they include birth history information at all. Given the value of subnational and stratified analyses of under-5 mortality and the limitations of the methods examined here, further research into methods for using existing data sources and investment in alternative data sources is warranted. In particular, small area methods, which address the issue of small sample sizes by borrowing strength across geographic units, may be useful when analyzing birth history data at a subnational level.