Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

The use of a lot quality assurance sampling methodology to assess and manage primary health interventions in conflict-affected West Darfur, Sudan

Population Health MetricsAdvancing innovation in health measurement201614:34

https://doi.org/10.1186/s12963-016-0103-3

Received: 17 February 2016

Accepted: 22 September 2016

Published: 6 October 2016

Abstract

Background

Organizations working in conflict-affected areas have a need to monitor and evaluate their programs, however this is often difficult due to the logistical challenges of conflict areas. Lot quality assurance sampling may be a suitable method of assessing programs in these situations.

Methods

We conducted a secondary data analysis of information collected during Medair’s routine program management functions. Medair’s service area in West Darfur, Sudan was divided into seven supervisory areas. Using the available population information, a sampling frame was developed and interviews were conducted from randomly selected caretakers of children in each supervisory area every six months over 19 months. A survey instrument with questions related to key indicators for immunizations and maternal, newborn, and child health was used for the interviews. Based on Medair’s goals for each indicator, decision rules were calculated for the indicators; these decision rules determined which supervisory areas and indicators performed adequately in each assessment period. Pearson’s chi-squared tests, adjusted for the survey design using STATA “svy: tab” commands, were used to detect overall differences in coverage in this analysis.

Results

The coverage of tetanus toxoid vaccination among pregnant women increased from 47.2 to 69.7 % (p value = 0.046), and births attended by a skilled health professional increased from 35.7 to 52.7 % (p value = 0.025) from the first to last assessment periods. Measles vaccinations declined from 72.0 to 54.1 % (p value = 0.046). The estimated coverage for the proportion of women receiving a postpartum dose of vitamin A (54.7 to 61.3 %, p value = 0.44); pregnant women receiving a clean delivery kit (54.6 to 47.1 %, p value = 0.49); and pentavalent vaccinations (49.7 to 42.1 %, p value = 0.28) did not significantly change.

Conclusions

Lot quality assurance sampling was a feasible method for Medair staff to evaluate and optimize primary health programs in a conflict-affected area. Medair managers were able to collect, analyze, and disseminate data to staff alongside the routine work of the organization. These results suggest LQAS may be used in other complex humanitarian emergencies in which there are logistical challenges and limited resources.

Keywords

Lot quality assurance sampling Monitoring and evaluation Conflict Complex humanitarian emergency

Background

In 2013, the Organisation for Economic Co-operation and Development estimated that industrialized countries donated $134.8 billion USD in aid to developing countries [1]. Donors are increasingly requiring funded organizations to not only deliver high quality services but also conduct rigorous monitoring and evaluation programs to assess the impact of these service [2]. Independent of donor requirements, organizations implementing programs want frequently updated information to help their own managers optimize services to meet organizational targets and make positive changes for beneficiaries.

Since active conflict began in the Darfur region of Sudan in 2003 [3], Sudan has been a major recipient of international humanitarian assistance. In 2012 it received the ninth-largest amount of humanitarian aid, and the total amount of assistance from the United States alone stands at $887 million USD [4]. Despite the amount of aid that is directed towards Sudan, it has been challenging for non-governmental organizations (NGOs) to provide services, particularly in the Darfur region where much violence has occurred [5]. The ongoing, protracted conflict in Darfur has progressively decreased the humanitarian space, making the delivery and monitoring of aid services difficult [6]. After the International Criminal Court charged Sudanese president Omar al-Bashir with war crimes and crimes against humanity in 2009, al-Bashir expelled ten international NGOs working in Darfur [7, 8]. Furthermore, armed militia have specifically directed attacks at aid organizations by looting supplies and kidnapping aid workers [9]. Adding to these difficulties, there are logistical challenges with the delivery of services in Darfur, since large distances separate communities across areas that are subject to violence [10]. In response, the United Nations World Food Programme (WFP) established a helicopter service to ferry aid groups to service sites [11]. The flight schedule set by the WFP, in combination with security measures limiting humanitarian workers’ ability to stay overnight in field locations, regularly restricts project managers’ time in communities to 2–3 h, hampering the ability of aid workers to monitor and maintain programs.

Description of Medair’s programs in West Darfur

From 2001 to 2012, the Switzerland-based NGO Medair worked in West Darfur [12], a state with an estimated population of 1.3 million [13] and one of the three states that comprise the Darfur region. Medair’s humanitarian assistance programs in West Darfur included primary healthcare service provision and maintenance of health clinics; health promotion activities; support for the Sudanese Ministry of Health programs; and water, sanitation, and hygiene service provision and promotion. These programs were conducted in both refugee and internally displaced persons (IDP) camps, as well as in towns with sizeable refugee and IDP populations, serving a total population of approximately 325,000. As part of their primary healthcare programs, Medair provided supportive supervision, quality assurance, and finance and human resources for primary health care workers – including midwives – who worked in West Darfur Ministry of Health clinics and primary health care workers who worked in clinics located in refugee and IDP camps. Medair also provided logistic and human resource support and intensive health promotion campaigns to augment the immunization programs conducted by the West Darfur Ministry of Health.

Description of the Medair/Johns Hopkins Monitoring and Evaluation Program

In 2010, Medair collaborated with the researchers from the Johns Hopkins School of Medicine and Bloomberg School of Public Health to establish a monitoring and evaluation (M&E) program to track indicators focused on primary healthcare coverage in West Darfur. The M&E program was initiated in the context of Medair having to expand its programs after the 2009 expulsion of ten NGOs working in Darfur. One of Medair’s goals was to have population-based methods of measuring the effectiveness of primary health care services in West Darfur. At the same time, Medair wanted to give its program managers on the ground a tool to empirically guide the management of primary health care programs.

Given the multiple challenges of logistics and security in West Darfur, a management method using lot quality assurance sampling (LQAS) was proposed for the monitoring and evaluation of Medair’s services. LQAS was originally developed in the 1920s as a method of assuring a minimum level of quality in industrial production [14]. Using LQAS concepts, a health program manager decides three factors which determine the sampling plan: (1) a target percent coverage for an indicator (the upper threshold); (2) a percent coverage below which a lot is considered unacceptable (the lower threshold); and (3) the tolerable levels of misclassifying failures and successes (the alpha and beta errors, respectively) [15, 16]. The three factors yield a decision rule; if a lot sample has fewer successes than the decision rule, the lot is classified as having unacceptable quality.

Valadez and colleagues have published a protocol for LQAS in health programs, which simplifies LQAS procedures [17]. In their formulation, LQAS is typically carried out in two stages. In the first stage, a random sample of 19 is taken within several local supervision areas (SA), which are usually subsets of a larger program catchment area. Supervision areas are assessed for performance of key health indicators; if the number of successes in an SA is greater than the pre-determined decision rule, then the performance of the relevant health indicator is classified as acceptable. The protocol simplifies LQAS by setting the sample size at 19 and a 30 % difference in the upper and lower thresholds, resulting in less than a 10 % probability of misclassification (i.e., the alpha and beta errors are less than 10 %). In the second stage, information from several supervision areas are combined and weighted, and an estimate for the entire program area is made for each health indicator. In certain sampling designs, the estimates for the program area are comparable in accuracy to estimates made from typical 30-cluster sample surveys that are often carried out by NGOs in low- and middle-income countries [18]. LQAS provides additional value over cluster sampling by providing information about the performance of health indicators within the program area (by supervision area), while cluster sampling provides estimates at the overall program catchment area level only. LQAS therefore allows program managers to identify variation in performance among supervision areas and to prioritize problem-solving in SAs with inadequate performance, rather than assuming performance is the same across the program area. In addition, the survey work for LQAS can be organized at the local level of each SA, and can be carried out concurrent with program activities using the same logistics (transport, personnel, etc.). As a result, programming does not need to stop for a period to allow for data collection activities, and survey costs can be shared with the routine costs of program activities.

LQAS has been used to assess many development programs globally [15] (such as malaria [19], malnutrition [18], and immunization [20]). There are few studies, however, that discuss the feasibility of using LQAS for measuring health indicators and outcomes in conflict-affected settings. Deitchler, et al. compared the precision, time, and cost of different variations of cluster surveys in West Darfur with a secondary aim of using LQAS hypothesis tests with each variation [21]. In another paper, Valadez and colleagues describe the use of LQAS in South Sudan to assess maternal and newborn health indicators [22].

The many advantages of LQAS made it preferable to the cluster survey method for Medair in West Darfur, where there are multiple challenges including the distance between the communities served by Medair and the tenuous security situation. Medair managers and supervisors had limited access to the field, and felt that the ability to conduct LQAS alongside usual programmatic work was a particular advantage of LQAS over cluster surveys. The objective of this paper is to describe the process of using the LQAS method in a conflict-affected region as a process to evaluate the effectiveness of primary health programs, and to guide the management of these programs.

Methods

This is a secondary data analysis of information collected during routing program management, monitoring, and evaluation of the Medair program. The monitoring and evaluation of primary health programs was carried out from May 2010 through December 2011 in West Darfur, Sudan. Four separate LQAS assessments were conducted, allowing measurement of key indicators approximately every six months. Medair and a group from Johns Hopkins had agreed to collaborate during these four assessments with the ultimate goal of training Medair staff such that Medair could independently continue assessments after this period without additional technical assistance.

Development of the sampling plan

As an initial step of random sampling, the Medair service area was divided into seven supervisory areas. The SAs were determined based both on geography and on Medair’s organizational structure, such that one manager could be assigned to each SA (Fig. 1). The SAs also corresponded with Medair’s seven areas for supervised and supported primary health care clinics and health promotion activities.
Fig. 1

Map of Medair’s program areas in West Darfur, Sudan

A sampling plan requires estimates of the population in the geographic area of interest, however true population estimates in Darfur are difficult to obtain [23]. Even without conflict, migration is common among the Darfur population; conflict added further instability to settlements and communities, making the determination of true population numbers a challenge [23]. For Medair’s LQAS program, the Sudanese government ministries and the United Nations organizations operating in the catchment area agreed upon the estimated population figures. Where estimated population information was not available or incomplete, we asked other NGOs for their population estimates of the communities. We then calculated the cumulative population of the serviced community (Table 1). For a given set of questions, nineteen households were selected in each SA, with the probability of a community being selected proportional to the community’s population [24]. We assumed that the proportion of women 15–49 years of age with live births in the 2 years prior to the survey (the target population of Medair’s services) was relatively constant in the SAs. For each round of LQAS, changes to the population listing were made as needed, and a new set of 19 households were selected in each SA.
Table 1

Supervisory areas and populationa

Region

Supervisory area (SA)

Population estimates

Community type

El Geneina

1

34,960

IDP Camps

Northwest

2

49,110

Towns & IDP Camps

East 1

3

29,505

Towns & IDP Camps

East 2

4

55,633

IDP Camps

Southwest

5

60,623

Towns & IDP Camps

Forboranga Town

6

50,885

Towns & IDP Camps

Forboranga/Habila Rural

7

44,626

Towns & IDP Camps

Note: aThese population figures refer to communities Medair serviced, not to West Darfur as a whole. The 2008 population of West Darfur was estimated at 1.3 million by the Sudanese government

Selection of indicators and development of the survey

With input from the Medair staff, portions of the United Nations Children’s Fund’s (UNICEF) Multiple Indicator Cluster Survey 4 (MICS4) questionnaire were adapted for our assessment [25]. Specifically, we adapted questions that provided information on the following indicators: immunization coverage for diphtheria, pertussis, and tetanus (indicator 3.3); measles immunization coverage (indicator 3.4); hepatitis B immunization coverage (indicator 3.5); neonatal tetanus protection (indicator 3.7); and skilled attendant at delivery (indicator 5.7). In 2011, the World Health Organization no longer recommended vitamin A supplementation in postpartum women [26]. However, Medair continued this practice. In order to monitor Medair’s programs, an indicator for vitamin A supplementation in postpartum mothers from the MICS3 was also included [27]. Finally, the number of women who had received a clean delivery kit prior to their last delivery was also measured because Medair managers felt it was important to assess this indicator, even though it is not a part of the MICS3 or MICS4. The 2010 Sudan Household Health Survey (SHHS), a countrywide assessment of public health indicators conducted by the Sudanese government, was based on the MICS4 [28]. The use of questions from MICS4 thus enabled the comparison of key indicators between areas where Medair provided primary health services to areas where Medair did not work.

The adapted questionnaire was translated from English to Sudanese Arabic, the language of study participants. The questionnaire was then back-translated by local translators; back-translations were reviewed by the research team and the local Medair team working in West Darfur. This process was necessary even though an Arabic translation of the MICS4 is available from UNICEF because the target population often did not understand the wording of UNICEF’s Arabic translation. Additional edits were made through an iterative process to ensure that the translation captured the essence of each question. Pilot testing was done with convenience samples in two communities of IDPs. Interviewers made notes of the capacity of the target population to understand and respond to the questionnaire. The results of the pilot testing were reviewed jointly by Johns Hopkins investigators and local Medair team members, and final edits were made based on the feedback from interviewers.

Participant recruitment and data collection

The majority of communities could only be accessed by helicopter, and the time in those communities was often restricted to two to three hours. In the first LQAS round, the Medair staff developed a detailed logistic plan that included delineating which staff members were going to specific communities with exact times for flight manifests. Medair staff worked on the survey separate from their usual work during the first round of data collection so that they could concentrate on learning data collection and management methods without the distraction of their usual work duties. During subsequent LQAS rounds, Medair staff working in each locality conducted interviews with participants under the supervision of research coordinators. To leverage the advantages of the LQAS methodology, survey work in these later rounds was done alongside the routine tasks of Medair staff on days when the staff visited assigned program communities that overlapped with the LQAS sample selection. The completed questionnaires were transported by helicopter or car to the main Medair office in El Geneina, the capital of West Darfur, for data entry and analysis.

Once staff arrived in a community, they conferred with a community leader or other key informant to develop a map of the community. Our team also informed community leaders about our planned activities and the purpose of the interviews, and we asked for any input community leaders had as a means of engaging the population. From these discussions, the community was divided into subsections, from which a subsection was randomly selected using a random number table. If the subsection had more than approximately 30 households, it was further divided and the process of random selection was repeated until an area was chosen with approximately 30 households to facilitate the process of randomly selecting a household [17]. The households were then numbered, and one was randomly chosen using a random number table. For large communities where more than one sampling unit was located, the process of selecting a household was repeated.

A parallel sampling method was used to administer the questionnaire [29]. In each randomly selected household, women between the ages of 15 and 49 who had a live birth in the 2 years prior to the interview were enrolled. If a household did not have a woman who met the inclusion criteria, surveyors went to the next nearest household as measured from the front door of one dwelling to the front door of the next. Questions in the survey were grouped depending on the age group of children to which they referred. For example, immunization questions required children that were at least 1 year of age while questions on MNCH indicators referred to children younger than 1 year. Thus, staff asked appropriate question groups depending on the age of children in a household. If a qualifying household did not have children from the age group required for a group of questions, the next closest households were screened until children from each age group were included in the survey. As a result, more than 19 participant households were required in each SA to complete the different set of questions. Verbal consent was obtained from all participants from a standardized consent script that was read to the participants by the surveyors. Verbal consent was used instead of written consent to ensure the respondents were not placed at an increased risk of harassment from local militias for participating in the study as would be the case if a written document were in the household’s possession.

Medair managers selected staff members to be trained in LQAS survey techniques, which took place over a three-day period of class work and one day of hands-on practice in a nearby community. Staff who showed a particular aptitude for the survey techniques were designated as survey supervisors. During the survey, supervisors oversaw the household selection process and reviewed completed paper questionnaires to assure completeness prior to leaving the community. Medair staff entered responses from the paper questionnaire into a database that was created in Epi Info (version 3.5.4, Centers for Disease Control and Prevention, Atlanta, GA).

Statistical analysis

For each of the randomly selected 19 households in each SA, a number of positive responses to questions about a health indicator (“successes”) were used to classify whether or not a health indicator’s performance in that SA was adequate. The classification was done by comparing the number of positive responses to a pre-determined decision rule value that had been calculated using LQAS sampling rules [17]. If the number of positive responses did not exceed the decision rule value, the performance of the health indicator in an SA was classified as inadequate; otherwise, the performance of the indicator was classified as adequate. Next, the responses from all SAs in the program area were pooled. A population estimate (including a mean and 95 % confidence interval) for the total program area was calculated for each indicator using the population estimates of each SA to weight the data.

For the first round of LQAS, the pooled percentage (unweighted) for each indicator was used along with LQAS sampling rules to determine the decision rule to be used in each SA. Doing so allowed the project to classify SAs as either adequate or inadequate for each indicator, as determined by baseline figures. For subsequent assessments, Medair managers established pre-determined decision rules for each indicator by using data from previous rounds of LQAS as well as Medair’s organizational goals and the managers’ projections of the capacity of the SAs to improve based on programmatic intervention.

The data were then imported into Stata/SE (version 11, StataCorp, College Station, TX) for analysis at the program level using Stata’s ‘svy’ options. Stata allowed us to estimate the performance of each indicator (with mean and 95 % confidence interval) by each round. We weighted the data by the population size of each SA and adjusted the standard errors for clustering at the SA level. To assess the overall change in Medair’s programs during the assessment period, we used Pearson’s chi-squared tests, using STATA “svy: tab” commands to adjust for the survey design to determine if there was a statistically significant difference (p value < .05) for each indicator using the same population weights and clustering adjustments.

Before starting surveillance, approval was obtained from the Johns Hopkins School of Medicine Institutional Review Board and the Sudanese Government Humanitarian Aid Commission at the national level in Khartoum and at the state level in West Darfur.

Results

Demographic characteristics of the sample

A total of 1323 women participated across the four rounds of LQAS assessments from May 2010 to December 2011 (Table 2). The average ages of participants in the four rounds were similar, with a range in mean age of 26.5 to 27.6. The most common level of education was no schooling, with approximately two-thirds of women reporting no formal education. The second most common level of education was a primary education, which 21.1–24.4 % of women reported throughout the assessment period. Almost all of the women were married (a range of 93.1–99.4 %), and the average number of children the participants reported was just over two (2.1–2.2).
Table 2

Demographic Characteristics of Study Participants

 

Assessment 1 (May 2010)

Assessment 2 (Oct. 2010)

Assessment 3 (May 2011)

Assessment 4 (Dec. 2011)

Total interviewed

315

338

348

322

Age (Years)

 Mean

27.6

26.5

27.4

26.5

 Standard Deviation

7.9

5.8

7.4

6.6

Education level (%)

 No schooling

64.5

68.6

66.1

68.9

 Primary

21.7

22.9

24.4

21.1

 Intermediate

0.3

0.0

1.4

0.6

 Secondary

2.2

2.0

1.2

1.9

 University/Higher Institute

0.6

0.0

0.3

0.0

 Khalwa (Religious School)

11.0

7.5

4.0

5.6

 Adult Education

(Not recorded)

(Not recorded)

2.6

1.9

Married (%)

93.1

97.6

98.3

99.4

Number of children

 Mean

2.2

2.2

2.2

2.1

 Standard Deviation

1.3

0.9

1.1

1.0

LQAS evaluation results

Several indicators for maternal, newborn, and child health showed improvement from the first to last LQAS rounds when calculated at the program level (Tables 3 and 4). Tetanus toxoid vaccination coverage for neonates increased throughout the assessment period from 47.2 to 69.7 % (p value = 0.046). In the last two rounds of LQAS, all the SAs were classified as adequate for this indicator. The only time an SA was classified as inadequate for tetanus toxoid coverage was SA7 during the first round. The coverage of births attended by a skilled health professional also improved significantly from 35.7 % in the first round to 52.7 % in the final round of LQAS (p value = 0.025, Table 3). At most, one SA was classified as inadequate in each assessment period, and in the final round of LQAS, all SAs were classified as adequate.
Table 3

Maternal and Child Health Indicators

Indicators

Protection against neonatal tetanus

Births attended by a skilled health professional

Assessment periods

May 2010

Oct. 2010

May 2011

Dec. 2011

May 2010

Oct. 2010

May 2011

Dec. 2011

Target Coverage

50 %

50 %

60 %

60 %

35 %

30 %

45 %

45 %

2010 Sudan Household Health Survey Result for West Darfur [28]

46.5 %

33.4 %

Decision Rule

7

7

9

9

4

3

6

6

Program Coverage (95 % CI)

47.2 % (28.4, 66.8)

53.8 % (48.1, 59.3)

76.9 % (59.3, 88.4)

69.7 % (54.9, 69.5)

35.7 % (21.0, 53.6)

34.5 % (22.2, 49.4)

40.2 % (24.1, 58.6)

52.7 % (39.0, 66.0)

P value for difference between 1st & 4th Rounds

0.046

0.025

Number of Successes

 SA1

13

11

17

16c

8

12

10

15c

 SA2

a

9

14

14

a

5

8

12

 SA3

11

12

15

15

6

2*

6

11

 SA4

7

11

16

14

3*

4

6

7

 SA5

10

9

18

12

10

9

13

12

 SA6

12

11

11

11

9

6

6

8

 SA7

2*

9b

11

13

4

7d

3*

7

Notes: *SAs that do not reach the benchmark. aDue to insecurity, samples were not taken from SA2 in the May 2010 round of LQAS. b18 samples instead of the usual 19 were obtained; the decision rule is unchanged. c20 samples instead of the usual 19 were obtained; the decision rule is unchanged. d18 samples instead of the usual 19 were obtained; the new decision rule is 2

Table 4

Maternal and Child Health Indicators

Indicators

Women who received a clean delivery kit

Mothers receiving 1 dose of postpartum vitamin A

Assessment periods

May 2010

Oct. 2010

May 2011

Dec. 2011

May 2010

Oct. 2010

May 2011

Dec. 2011

Target Coverage

50 %

35 %

55 %

55 %

50 %

45 %

60 %

60 %

2010 Sudan Household Health Survey Result for West Darfur [28]

Not Reported

15.3 %

Decision Rule

7

4

8

8

7

6

9

9

Program Coverage (95 % CI)

54.6 % (30.5, 76.7)

63.8 % (50.3, 75.4)

56.8 % (38.9, 73.1)

47.1 % (33.7, 60.9)

54.7 % (39.2, 69.3)

54.7 % (40.0, 68.6)

45.5 % (30.6, 61.3)

61.3 % (48.9, 72.4)

P value for difference between 1st & 4th Rounds

0.49

0.44

Number of Successes

 SA1

12

13b

15

16d

8

15

10

7*d

 SA2

a

7

9

12

a

8

6*

11

 SA3

9

8

7*

10

14

9

15

11

 SA4

7

5

10c

11

10

9

13

6*

 SA5

14

7

9

8

9

7

13

6*c

 SA6

2*

4

3*

6*

9

5*

9

4*c

 SA7

8

6b

5*

10

3*

7c

7*

8*c

Notes: *SAs that did not reach the benchmark. aDue to insecurity, samples were not taken from SA2 in the May 2010 round of LQAS. b18 samples instead of the usual 19 were obtained; the new decision rule is 3. c18 samples instead of the usual 19 were obtained; the decision rule is unchanged. d20 samples instead of the usual 19 were obtained; the decision rule is unchanged

Two other MNCH indicators did not improve (Table 4). The coverage of women who received a clean delivery kit did not significantly change between the initial and final rounds of LQAS (from 54.6 to 47.1 %, p value = 0.49). For this indicator, results during the May 2011 round of LQAS were particularly poor as three of the seven SAs were classified as inadequate. SA6 was the poorest performing among all the SAs since it was classified as inadequate in three of the four rounds of LQAS. The percentage of mothers receiving at least one dose of postpartum vitamin A changed from 54.7 to 61.3 % throughout the assessment period, however this difference was not statistically significant (p value = 0.44, Table 4). Although the December 2011 round of LQAS yielded the highest coverage value for post-partum vitamin A, five SAs were classified as inadequate.

Other than tetanus coverage, the findings for other vaccinations were disappointing (Table 5). From the start of the monitoring and evaluation program to the end, the coverage of pentavalent vaccine (a combination of the Haemophilus influenzae type B, Pertussis, Tetanus, Hepatitis B, and Diphtheria vaccines) did not significantly change (from 49.7 % in Round 1 to 42.1 % in Round 4, p value = 0.28). For pentavalent vaccine coverage, five of the seven SAs were classified as inadequate in the last assessment period. The proportion of infants receiving the measles vaccination by the first birthday also failed to meet goals set forth by Medair; the coverage significantly decreased from 72.0 to 54.1 % from the first to last survey periods (p value = 0.046). Notably, five SAs in May 2011, and six SAs in December 2011 were classified as inadequate. Although we collected information on the coverage of the polio vaccine, the results were difficult to interpret because the definition of the indicator was changed between the second and third rounds of LQAS to be aligned with the indicator used in the Sudan Household Health Survey. As such, the results are not included here.
Table 5

Immunization Indicators

Indicators

Pentavalent Vaccination by First Birthday

Measles Vaccination by First Birthday

Assessment periods

May 2010

Oct. 2010

May 2011

Dec. 2011

May 2010

Oct. 2010

May 2011

Dec. 2011

Target Coverage

50 %

60 %

70 %

70 %

75 %

68 %

90 %

90 %

2010 Sudan Household Health Survey Result for West Darfur [28]

45.8 %

54.4 %

Decision Rule

7

9

10

10

11

10

14

14

Average Coverage (95 % CI)

49.7 % (31.7, 67.8)

74.8 % (62.5, 84.1)

53.7 % (43.8, 63.4)

42.1 % (30.6, 54.6)

72.0 % (60.0, 81.5)

72.9 % (56.9, 84.6)

69.1 % (57.7, 78.5)

54.1 % (38.4, 69.0)

P value for difference between 1st & 4th Rounds

0.28

0.046

Number of Successes

 SA1

10a

16

13

6*

16a

17

14

8*

 SA2

b

14c

13

12

b

14f

17

13*

 SA3

14a

6*

7*

11

13a

9*

8*

15

 SA4

8

12

11

7*

15

13

13*

12*

 SA5

12

14

9*

9*d

14

17

13*

11*g

 SA6

10

12

10

7*

13

11

13*

5*

 SA7

3*

10

8*

3*e

10*

9*

12*

6*h

Notes: *SAs that did not reach the benchmark. (a) 18 samples instead of the usual 19 were obtained; the decision rule is unchanged. (b): Due to insecurity, samples were not taken from SA2 in the May 2010 round of LQAS. (c) 17 samples instead of the usual 19 were obtained; the new decisiosn rule is 8. (d) 20 samples instead of the usual 19 were obtained; the new decision rule is 11. (e) 12 instead of the usual 19 were obtained; the new decision rule is 7. (f) 17 samples instead of the usual 19 were obtained; the new decision rule is 9. (g) 20 samples instead of the usual 19 were obtained; the new decision rule is 20. (h) 12 samples instead of the usual 19 were obtained; the new decision rule is 9

Discussion

This paper has described the process of using LQAS to monitor and evaluate the primary health programs of Medair’s operations in West Darfur. Medair used the baseline LQAS findings as an integral part of strategic planning for targeted MNCH interventions and immunization efforts. Based on the results of the MNCH indicators from the first round of LQAS, Medair implemented two interventions to improve maternal and newborn care. First, Medair focused on in-service training for midwives to improve care in the immediate postpartum period for both mothers and infants. The care provided by midwives and supported by Medair was augmented with supervisory visits from advanced-practice midwives, doctors specifically trained in obstetrics and gynecology, and with biannual five to ten day in-service trainings. New strategic topics were introduced during in-service trainings, and the supervisory midwives and doctors visited various clinics to mentor midwives in the implementation of these skills and for quality improvement and quality assurance of the clinical care delivered to mothers and infants in the project area.

Second, Medair trained health promotion volunteers in communities and camps to deliver key messages about the importance of postpartum care. These messages were delivered to groups of mothers in the waiting areas at antenatal clinics and through existing women’s groups (e.g., women’s trade groups). The messages emphasized the importance of delivery with a skilled health worker; the importance of tetanus vaccines, iron supplements, and Vitamin A for the mothers; danger signs for newborns that warrant immediate medical attention; and the importance of a postpartum visit with a midwife or other health worker 2 to 3 days after giving birth and 6 weeks after the birth. Medair used data on key indicators from each round of LQAS to adjust and maximize the interventions after each round. The LQAS-based M&E system would periodically give Medair managers empirical feedback, allowing Medair to improve its programs.

Two out of the four MNCH indicators measured by the survey (protection against neonatal tetanus and births attended by a skilled health worker) showed improvement throughout the period that LQAS was used for M&E. Medair’s coverage of the two indicators was also greater than the coverage reported in the 2010 Sudan Household Health Survey for West Darfur State (Table 3). Medair’s coverage of protection against neonatal tetanus ranged from 47.2–69.7 %, while the SHHS coverage was 46.5 %. For the percentage of births attended by a skilled health professional, Medair’s coverage ranged from 34.5–52.7 %, compared with the SHHS coverage of 33.4 %. Medair’s coverage of mothers receiving a dose of postpartum vitamin A did not significantly improve. However, Medair’s coverage ranged from 45.5–61.3 % during the survey period and was higher than the 15.3 % coverage reported in the SHHS (Table 4).

The improvement in protection against neonatal tetanus and births attended by a skilled health worker is associated with the Medair managers’ decision to focus on enhancing MNCH programs in West Darfur during the 19 months that LQAS was conducted. However, the improvements were not uniform across the program areas, and Medair faced obstacles in improving MNCH indicators. For example, supervisory midwives and health promotion advisors had limited access to remote communities in SA2 during all assessment rounds due to continued armed conflict and a lack of security. Another program challenge in SAs 4, 6, and 7 was related to two human resource challenges: (1) the number of available trained midwives serving the population was insufficient for the population size, according to Sphere standards [30]; and (2) local labor laws – which necessitated specific work schedules and pay for on-call hours – resulted in interruptions in services.

Throughout the 19-month program implementation period, immunization indicators other than tetanus did not statistically improve, and in contrast to the MNCH indicators, the program coverage in the last assessment period was lower than figures reported in the 2010 SHHS (Table 5). The coverage of the Pentavalent vaccination was 42.1 % during the last assessment period, compared with a coverage of 45.8 % from the SHHS in West Darfur while Medair’s coverage of the measles vaccination was 54.1 % in the last assessment period compared with a coverage of 54.4 % from the SHHS in West Darfur. This may be due to the fact that Medair’s responsibilities were primarily supportive rather than active in West Darfur immunization activities and, although extensive, appear not to have been sufficient. Immunizations were led by the West Darfur State Ministry of Health through routine immunization programs and “mop-up” campaigns, which are campaigns in communities that go house-to-house, often as a result of a confirmed outbreak or in a push to ensure stronger coverage of a specific vaccine (e.g., polio) to immunize children who did not receive vaccines in the initial round. Medair’s supportive interventions included providing salary increases for the Expanded Program on Immunization (EPI) workers; intensifying health promotion about immunizations; providing performance-based incentives for community vaccinators; supporting the logistics for vaccines from the state capital to health facilities throughout the state (e.g., cold chain and storage); and strengthening linkages between antenatal care and immunizations areas within health facilities in the region. Changes were made to the physical layout of some clinics (e.g., placing the EPI office next to the rooms for midwifery care) to facilitate the vaccination of newborns when women came for postpartum visits. However, even with periodic data on the status of the immunization programs from LQAS, Medair’s supportive role did not allow Medair the programmatic control over administration of routine immunizations or the conduct of mop-up campaigns as was the case in other Medair programs.

Medair also faced challenges that were anticipated given working conditions in West Darfur. Supervisory Areas 6 and 7 were absorbed into Medair’s existing programs when other international NGOs were expelled in the first quarter of 2009. Medair started programs in these communities in late 2009. In the interim period, the West Darfur Ministry of Health worked to maintain the health facilities in this area, but a number of health workers resigned and several health facilities closed for a period of time. Medair’s program implementation included reopening these facilities, filling vacant health worker positions, initiating health promotion activities in the communities, and ensuring that clinical care met standards set by both Medair and the State Ministry of Health. There were also logistical challenges in the area, particularly in the rainy season, when some of the health facilities and trade centers were inaccessible due to flooding.

With these challenges, the data reflect poor performance in SAs 6 and 7. SA6 was classified as inadequate for the receipt of clean delivery kits in three of the four assessment periods, including the final round of LQAS. It was also classified as inadequate for two of the three immunization indicators in the final round. Similarly, SA7 was classified as inadequate for the coverage of measles immunization in every LQAS round and for two of the three immunization indicators in the December 2012 assessment.

Active armed conflict and the lack of security also hampered the efforts of Medair to deliver services to the catchment area and to assess the effectiveness of these services. Medair planned monthly quality assurance, supervision, and mentoring visits for clinical staff and health education volunteers, but frequently these visits were delayed due to security restrictions. When on-site visits were possible, Medair staff had limited time for these visits with local clinic staff and volunteers. At times, field visits of 3–4 h each quarter were the only times allowed for these interactions, compounding the challenges of conducting LQAS. Thus, while Medair was able to feasibly train and conduct the LQAS M&E program in a conflict setting, LQAS was a necessary but not sufficient element in translating the M&E system into improvements for Medair’s beneficiaries. LQAS by itself cannot overcome the barriers to healthcare access created by conflict and large distances between towns. If such barriers are eliminated, however, LQAS helps managers to efficiently monitor and adjust their programs.

There are limitations to the data we present. As mentioned above, accurate population estimates in Darfur are difficult to obtain; it is possible that our population estimates were inaccurate, which would affect the formulation of the sampling plan and introduce bias into our results. We did not record the number of individuals who declined to participate in the survey. Women who did participate may be systematically different than those who declined, which may also bias our results. Furthermore, during our first assessment, Medair staff was unable to enter SA2 due to armed conflict. The absence of this data in addition to the small sample sizes in LQAS in general decreases the precision of the estimates of indicators and makes drawing conclusions more difficult. As already described, access to clinics was hampered during the rainy season, and differences in indicators from one assessment period to the next six-month assessment round may have been affected by these seasonal factors. While there are other types of complex humanitarian emergencies for which LQAS may be optimal, we only examined the use of LQAS in a conflict-affected setting. Our conclusions may not apply in other types of humanitarian emergencies.

Another limitation of the data is that Medair restricted its use of LQAS to assess the population coverage of specific services related to the organization’s programs. The monitoring and evaluation system for this project focused on coverage, but did not include measurements of quality of care. Medair utilized other quality assurance measures in their supportive supervision of clinics and community health promotion volunteers. LQAS was specifically utilized to help determine the coverage of specific interventions and education initiatives that impact key health issues. Additionally, Medair needed to limit the length and time for the survey questionnaire and the limited available time of surveyors in communities. However, LQAS can be used as a tool to assess quality of care. There are multiple examples of LQAS being used for quality management [15], such as assessing the quality of care in malaria control in Nigeria [19], and the quality of health facility services in South Sudan [22]. While quality of care was not specifically measured in the presented study in West Darfur, we recommend using LQAS for this purpose in the future so that the full range of the program’s expected results (coverage and quality) can be monitored on a regular basis in a conflict-affected environment.

The next steps from the findings we present are to implement LQAS in a different complex humanitarian emergency to examine if our experience using LQAS in Darfur is consistent in other contexts. While there are published papers describing the combination of LQAS with aspects of the cluster survey design [21, 31], further investigations should also focus on comparing LQAS with the traditional cluster survey method in a complex emergency, with an emphasis on comparing the feasibility of the two survey methods to understand more clearly which method is preferable in various settings. In addition, applications of LQAS as a quality management tool in humanitarian emergencies should be further tested, including more robust efforts to support use of periodic or mid-program data to redirect activities as issues are identified. Until such studies are conducted, this paper has taken a step in illustrating how a novel method of quality assurance was adapted to provide a tool for management of a health program in an insecure, dynamic environment.

Conclusions

The LQAS management method was beneficial for Medair’s services in West Darfur, despite the many obstacles to conducting population-level assessments of primary health programs. The surveying technique required up front coordination, time to train managers and establishment of survey and data collection techniques. After the initial training, however, local managers were able to autonomously conduct LQAS alongside their usual work, gather and analyze data, and feed the collected information back to the programs to improve service delivery in the catchment area. Program managers gained skills not only in data collection and analysis, but also in using data to make program decisions and to influence program strategy. One key learned lesson from Medair’s experience is the importance of the ability to change interventions based on results from LQAS; because Medair did not control all aspects of immunization programs in West Darfur, Medair could not fully implement changes based on the LQAS results. Nevertheless, the ability of LQAS to be easily taught to local managers and the decentralized nature of data collection and analysis in LQAS enhances its prospects for sustainability, which is vital in low-resource settings. The experience of Medair and Johns Hopkins in Sudan demonstrates that LQAS can be used in a challenging environment that is plagued by armed conflict.

Abbreviations

EPI: 

Expanded Program on Immunization

IDP: 

Internally-displaced persons

LQAS: 

Lot quality assurance sampling

M&E: 

Monitoring and evaluation

MICS4: 

Multiple Indicator Cluster Survey 4

MNCH: 

Maternal newborn, child health

NGO: 

Non-governmental organization

SA: 

Supervisory area

SHHS: 

Sudan Household Health Survey

UNICEF: 

United Nations Children’s Fund

WFP: 

(United Nations) World Food Programme

Declarations

Acknowledgements

The authors would like to recognize both the national and international staff of Medair in Sudan, who had the drive to critically and scientifically assess their programs and who carried out much of the data collection presented in this paper. Specifically, we would like to thank Adam Gelaladin, Hatim Ali, Ali Mohamedain, Mohammed Ahemed Wadeljed, and Imam Abdallah. We would also like to thank Tina Yu, Medair’s Country Director of Sudan at the time of the study for her support in implementing the LQAS-based M&E project and Wendy Dyment, MD MPH, Senior Health and Nutrition Advisor at Medair in Geneva for her input throughout the study design, study implementation, data reporting, and manuscript review. Finally, the authors thank the Center for Refugee and Disaster Response in the Department of International Health at the Johns Hopkins Bloomberg School of Public Health for assistance with publishing the paper.

Funding

Medair funded the collection of data. Their managers consulted on the design of the study. Medair had no role in the analysis or interpretation of data. One of the authors, Emily Sharpe, previously worked for Medair but does not currently work for the organization.

Availability of data and materials

The data will not be shared. The data was originally collected in a collaboration with Medair, and due to Medair’s organizational policies, the data will not be publically shared.

Authors’ contributions

KP participated in the design of the study, participated in the collection of data, analyzed data, and drafted the manuscript. ES participated in the design of the study, participated in the collection of the data, and helped to draft the manuscript. WMW participated in the analysis of data and helped to draft the manuscript. AV designed the study, participated in the collection and analysis of data, and helped to draft the manuscript.

Authors’ information

KP is an Assistant Professor in the Department of Emergency Medicine at the Johns Hopkins School of Medicine and at the Center for Refugee and Disaster Response in the Health Systems Program in the Department of International Health at the Johns Hopkins Bloomberg School of Public Health.

ES is a consultant for global health, HIV/AIDS, and nutrition programs. At the time of this study she worked with Medair as the Health Coordinator for the West Darfur Primary Health Care program.

WMW is an Associate Scientist with the Center for Refugee and Disaster Response in the Health Systems Program in the Department of International Health at the Johns Hopkins Bloomberg School of Public Health.

AV is the Director of the Johns Hopkins International Emergency and Public Health Fellowship and an Assistant Professor in the Department of Emergency Medicine at the Johns Hopkins School of Medicine and at the Center for Refugee and Disaster Response in the Health Systems Program in the Department of International Health at the Johns Hopkins Bloomberg School of Public Health.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Approval was obtained from the Johns Hopkins School of Medicine Institutional Review Board (NA_00037665) and the Sudanese Government Humanitarian Aid Commission at the national level in Khartoum and at the state level in West Darfur.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Emergency Medicine, Johns Hopkins School of Medicine
(2)
Department of International Health, Center for Refugee and Disaster Response, Johns Hopkins Bloomberg School of Public Health
(3)
Global Health, HIV/AIDS, and Nutrition Consultant

References

  1. Aid to developing countries rebounds in 2013 to reach an all-time high. Organisation for Economic Co-operation and Development. 2014. http://www.oecd.org/newsroom/aid-to-developing-countries-rebounds-in-2013-to-reach-an-all-time-high.htm. Accessed 25 Mar 2015.
  2. Crawford P, Bryce P. Project monitoring and evaluation: a method for enhancing the efficiency and effectiveness of aid project implementation. Int J Project Manage. 2003;21:363–73.View ArticleGoogle Scholar
  3. Copnall J. Darfur conflict: Sudan’s bloody stalemate. 2013. http://www.bbc.com/news/world-africa-22336600. Accessed 10 Dec 2014.Google Scholar
  4. Sudan. Country profiles. Global Humanitarian Assistance. 2013. http://www.globalhumanitarianassistance.org/countryprofile/sudan. Accessed 10 Oct 2014.
  5. Darfur: new humanitarian needs and aid delivery fact sheet. UN Office for the Coordination of Humanitarian Affairs. 2014. http://reliefweb.int/report/sudan/darfur-new-humanitarian-needs-and-aid-delivery-fact-sheet-25-may-2014. Accessed 25 May 2014.
  6. Young H, Maxwell D. Participation, political economy and protection: food aid governance in Darfur. Sudan Disasters. 2013;37(4):555–78.View ArticlePubMedGoogle Scholar
  7. Rice X, Branigan T. Sudanese president expels aid agencies. The Guardian. 2009. http://www.theguardian.com/world/2009/mar/05/sudan-aid-agencies-expelled. Accessed 27 Apr 2015.Google Scholar
  8. Rice X. Sudanese president Bashir charged with Darfur war crimes. The Guardian. 2009. http://www.theguardian.com/world/2009/mar/04/omar-bashir-sudan-president-arrest. Accessed 27 Apr 2015.Google Scholar
  9. Usmani M. Restrictions on humanitarian aid in Darfur: the role of the International Criminal Court. Georgia J Int Compar Law. 2008;36(257):257–89.Google Scholar
  10. Young H. Looking beyond food aid to livelihoods, protection and partnerships: strategies for WFP in the Darfur states. Disasters. 2007;31 Suppl 1:S40–56.View ArticlePubMedGoogle Scholar
  11. UNHAS. Current operations. World Food Programme. 2015. http://www.wfp.org/countries/sudan/operations/unhas-current-operations. Accessed 25 Mar 2015.
  12. Sudan: Medair concludes relief programmes in Sudan. Medair. 2012. http://relief.medair.org/en/stories/sudan-medair-concludes-relief-programmes-in-sudan/. Accessed 10 Oct 2014.
  13. Census 2008 results - West Darfur State. Republic of Sudan Central Bureau of Statistics. 2013. catalog.ihsn.org/index.php/catalog/4216/download/55706. Accessed 3 Oct 2016.
  14. Dodge HF, Romig RH. A method of sampling inspection. Bell System Technical J. 1929;8:613–31.View ArticleGoogle Scholar
  15. Robertson SE, Valadez JJ. Global review of health care surveys using lot quality assurance sampling (LQAS), 1984–2004. Soc Sci Med. 2006;63(6):1648–60.View ArticlePubMedGoogle Scholar
  16. Pagano M, Valadez JJ. Commentary: understanding practical lot quality assurance sampling. Int J Epidemiol. 2010;39(1):69–71.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Valadez JJ, Weiss W, Leburg C, Davis R. Asessing Community Health Programs. A Trainer’s Guide. Using LQAS for Baseline Survey and Regular Monitoring. St Albans, UK: TALC; 2003.Google Scholar
  18. Deitchler M, et al. A field test of three LQAS designs to assess the prevalence of acute malnutrition. Int J Epidemiol. 2007;36(4):858–64.View ArticlePubMedGoogle Scholar
  19. Biedron C, et al. An assessment of Lot Quality Assurance Sampling to evaluate malaria outcome indicators: extending malaria indicator surveys. Int J Epidemiol. 2010;39(1):72–9.View ArticlePubMedPubMed CentralGoogle Scholar
  20. Alberti KP, et al. Use of Lot Quality Assurance Sampling (LQAS) to estimate vaccination coverage helps guide future vaccination efforts. Trans R Soc Trop Med Hyg. 2008;102(3):251–4.View ArticlePubMedGoogle Scholar
  21. Deitchler M, Deconinck H, Bergeron G. Precision, time, and cost: a comparison of three sampling designs in an emergency setting. Emerg Themes Epidemiol. 2008;5:6.View ArticlePubMedPubMed CentralGoogle Scholar
  22. Valadez JJ, et al. Finding the gap: revealing local disparities in coverage of maternal, newborn and child health services in South Sudan using lot quality assurance sampling. Trop Med Int Health. 2015;20(12):1711–21.View ArticlePubMedGoogle Scholar
  23. Darfur crisis: death estimates demonstrate severity of crisis, but their accuracy and credibility could be enhanced. U.S. Government Accountability Office. 2006. http://www.gao.gov/products/GAO-07-24. Accessed 23 Sept 2014.
  24. Armitage P, Colton T. Encyclopedia of biostatistics. 2nd ed. Hoboken: Wiley; 2005.View ArticleGoogle Scholar
  25. MICS4 Tools. UNICEF. 2009. http://mics.unicef.org/tools?round=mics4. Accessed 5 June 2016.
  26. Guideline: Vitamin A supplmentation in postpartum women. WHO. 2011. http://apps.who.int/iris/bitstream/10665/44623/1/9789241501774_eng.pdf. Accessed 5 June 2016.
  27. MICS3 Tools. UNICEF. 2006. http://mics.unicef.org/tools?round=mics3. Accessed 25 Mar 2015.
  28. Sudan household and health survey second round 2010: summary report. Republic of Sudan Central Bureau of Statistics. 2011. http://reliefweb.int/sites/reliefweb.int/files/resources/MICS4_Sudan_2010.pdf. Accessed 9 June 2016.
  29. Protocol for parallel sampling: using lot quality assurance sampling to collect rapid CATCH information. CORE Group. 2008. http://www.coregroup.org/resources/550-lot-quality-assurance-sampling-lqas-protocol-for-parallel-sampling-a-guidance-faqs. Accessed 5 June 2016.
  30. The Sphere Handbook. The Sphere Project. http://www.spherehandbook.org/. Accessed 28 Apr 2015.
  31. Hund L, Pagano M. Extending cluster lot quality assurance sampling designs for surveillance programs. Stat Med. 2014;33(16):2746–57.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© The Author(s). 2016

Advertisement