Data collection, quality and analysis

1Data collection methods

• Quantitative: household surveys, counts, distribution registers, health data. They tell us “how many”.
• Qualitative: focus group discussions (6 to 10 people from a homogeneous group, by sex and age), key informant interviews, direct observation, case studies. They tell us “why” and “how”.
• Mixed methods: combining the two, together with triangulation (comparing several sources), strengthens the reliability of conclusions.

Common digital tools: KoboToolbox and ODK (forms on mobile phones that work offline), then Excel, Power BI or R for analysis.

2Sampling

We rarely survey the entire population: we draw a representative sample.

Sample size for a proportion: n = z² × p × (1 − p) / e². With a 95% confidence level (z = 1.96), an unknown proportion (p = 0.5) and a 5% margin of error (e = 0.05): n = 3.8416 × 0.25 / 0.0025 ≈ 384 households.

• If the population is small, apply the finite population correction: n' = n / (1 + (n − 1) / N). For N = 1,000 households, n' ≈ 278.
• In cluster sampling (randomly selected villages), multiply by the design effect, often 1.5 to 2.
• Add 5 to 10% for refusals and absences.
• Methods: simple random (complete list), systematic (one household in every k), stratified (by area or group), cluster. For qualitative work, participants are selected purposively.

3Data quality

Five criteria: validity (we really measure what we intend to), reliability (the same result if repeated), precision, timeliness and integrity (no manipulation).

• Train and test enumerators, pre-test the questionnaire.
• Constraints and checks in the digital form (impossible values, required fields).
• Supervision and revisits to a sample of households (back-checks).
• Data quality assessments (DQAs) for key indicators.

4Analysing and using data

Useful analysis answers a decision-making question. Compare with the baseline and the target, disaggregate (women, men, girls, boys, people with disabilities, areas), and look for gaps and their causes using qualitative data.

Post-distribution monitoring (PDM) checks, a few weeks after a distribution, whether the assistance arrived, how it was used and whether it met needs. Present the results on one page, with simple charts and 3 recommendations, at a meeting where the team decides on corrective action.

5Designing a questionnaire and a digital form: a step-by-step method

A questionnaire is built from the indicators and the analysis questions, never from a blank page or an old questionnaire copied over. Most quality problems found at the analysis stage (ambiguous questions, inconsistent units, modules that cannot be used) originate at this step.

The steps

1. Draw up the data analysis plan: for each indicator and each learning question, list the variables needed, the disaggregation and the final table expected. Any question in the form that does not feed a row of this plan is removed.
2. Reuse standard modules: Food Consumption Score, Reduced Coping Strategies Index, Washington Group questions, water and sanitation access modules from the WHO/UNICEF Joint Monitoring Programme. Reuse them word for word, with their recall periods (7 days for the FCS and the rCSI, for example).
3. Order the questions: consent, minimal identification, household composition, least sensitive modules first, sensitive modules at the end, thanks and information on the complaints mechanism.
4. Limit the duration: beyond 45 minutes, respondent and enumerator fatigue severely degrades quality. For a phone survey, aim for 15 to 20 minutes at most.
5. Translate and back-translate: one person translates into the local language, another, who has not seen the original, translates it back into the source language; the discrepancies reveal ambiguities. Key terms (household, income, improved latrine) are defined in a glossary given to enumerators.
6. Program the form (XLSForm format, used by KoboToolbox and ODK): constraints (age between 0 and 120, household size between 1 and 30), relevance conditions (the breastfeeding module only appears if there is a child under 2), required fields, choice lists rather than free text, automatic recording of start and end time, device ID and, if it is safe to do so, GPS location.
7. Test internally, then pre-test with 10 to 20 households outside the sample, in a comparable area. Measure the duration, note questions that are misunderstood, refusals and frequent "don't know" answers.
8. Freeze the version, number it and archive it with the data analysis plan. Any later change is logged.

Question wording: classic mistakes

Data on gender-based violence is not collected in a general household survey. IASC guidance and the GBV Area of Responsibility point out that you do not need to know prevalence in order to act: assume that violence exists and check the availability and accessibility of services.

Enumerator training

Allow 3 to 5 days for a full household survey: presentation of objectives, ethics and consent, code of conduct and protection from sexual exploitation and abuse, question-by-question review, role plays, handling of phones, household selection procedure, a pilot day in the field followed by a debriefing. A final test makes it possible to retain only the best candidates; plan to train 10 to 20% more people than needed.

6Advanced sampling: clusters, probability proportional to size and comparing two surveys

The basic formula seen in the core course gives the sample size for estimating a proportion at a given point in time. In practice, two situations require going further: cluster sampling, almost always necessary in rural areas for lack of a complete list of households, and comparison between a project baseline and endline.

Calculating the design effect

Rather than applying a flat design effect, it can be estimated: design effect = 1 + (m − 1) × ICC, where m is the number of households surveyed per cluster and ICC the intra-cluster correlation coefficient (how similar households in the same village are on the indicator). The ICC often ranges from 0.01 to 0.1 for household indicators; it is high for access to water (the whole village shares the same water point) and lower for individual practices. With m = 12 and an ICC of 0.05: design effect = 1 + 11 × 0.05 = 1.55. With m = 30: 2.45. Hence the rule of thumb: more clusters with fewer households per cluster gives better precision for the same number of interviews, but costs more in travel.

Sample size for measuring change

To check that an indicator has really changed between baseline and endline, the formula becomes:

n per survey = (zα + zβ)² × [p1 (1 − p1) + p2 (1 − p2)] / (p2 − p1)², with zα = 1.96 for a 95% confidence level and zβ = 0.84 for 80% power.

Example: the indicator is 34% at baseline and 50% is expected at endline. n = (2.80)² × [0.34 × 0.66 + 0.50 × 0.50] / (0.16)² = 7.84 × 0.4744 / 0.0256 ≈ 146 households per survey. With a design effect of 1.55: 227. Allowing for 10% non-response (divide by 0.9): 253. With 12 households per cluster: 21.1, rounded up to 22 clusters of 12 households, i.e. 264 households at baseline and the same number at endline. Note: the smaller the expected change, the faster the sample grows; to detect a shift from 34% to 40%, more than 1,000 households per survey would be needed before the design effect.

Selecting clusters with probability proportional to size

When villages differ in size, clusters are selected with probability proportional to size (PPS), then the same number of households is selected in each cluster. Each household in the area thus has the same probability of being surveyed.

1. List the villages with their population (or number of households) and calculate the cumulative population.
2. Calculate the sampling interval: total population / number of clusters. For 6,000 households and 22 clusters: 272.7.
3. Draw a random number between 1 and 272.7, for example 118. The clusters correspond to the villages whose cumulative population contains 118; 118 + 272.7 = 390.7; 663.4; and so on.
4. A large village may be selected twice: it then contains two clusters.
5. In each cluster, select households by random draw from a recent list if one exists, otherwise by segmenting the village or using the random walk method, bearing in mind that the latter introduces bias and must be described in the limitations.

Stratified sampling and weighting

If reliable results are wanted separately for displaced and host households, the sample is stratified and a sample size calculated for each stratum. The sample is then no longer proportional to the population and the overall results must be weighted. Example: 1,800 displaced households (30%) and 4,200 host households (70%), 132 households surveyed in each stratum. Result: 44% acceptable FCS among displaced households, 58% among host households. The unweighted average (51%) is wrong; the weighted average is 0.30 × 44 + 0.70 × 58 = 53.8%. The household weights equal the stratum's share of the population divided by its share of the sample: 0.6 for displaced households and 1.4 for host households.

Qualitative sampling

Qualitative methods do not seek statistical representativeness but diversity of viewpoints and saturation (the point at which new interviews no longer bring new information). Practical benchmarks: 2 to 4 focus group discussions per major population subgroup (displaced women, displaced men, host women, young people, persons with disabilities…), 8 to 15 key informant interviews with people chosen for their position (authorities, religious leaders, health staff, traders, teachers), and a few case studies to illustrate individual trajectories.

7Full example: organising a project endline survey

Fictional context: the endline survey of the NGO "Terres d'Avenir" food security project must survey 264 households in 22 clusters in the Moussoro-Nord region, with a 40-minute questionnaire, and conduct 8 focus group discussions and 10 key informant interviews.

Calculating productivity and duration

• An enumerator completes 5 interviews per day in a rural area (40 minutes each, plus time for household selection, travel between households and breaks).
• With 12 enumerators, capacity is 60 interviews per day, i.e. 264 / 60 = 4.4 days, rounded up to 5 days of data collection, plus 1 reserve day for revisits and absent households.
• Organisation in teams of 4 enumerators and 1 supervisor; each team covers about 1.5 clusters per day.
• Composition: 6 female and 6 male enumerators, so that women can be interviewed by women; all of them speak the two main languages of the area.

Timeline

Survey budget

The cost per household surveyed (about EUR 25 for the quantitative part) is a useful benchmark for budgeting subsequent surveys; it doubles or triples in hard-to-reach areas or when escorts or flights are needed.

Result and interpretation

The endline measures 52% of households with an acceptable FCS (baseline 34%). Calculating the confidence interval: simple standard error = square root of (0.52 × 0.48 / 264) = 0.0307; multiplied by the square root of the design effect (square root of 1.55 = 1.245): 0.0383; 95% interval = 52% ± 1.96 × 3.83 = 44.5% to 59.5%. The baseline (34%, interval of about 27% to 41%) and the endline do not overlap: the change is statistically significant. However, the 50% target lies within the interval: it can be said that the target is met, but the interval must be stated, along with a reminder that the endline took place after the harvest, which contributes to the improvement. Data from the regional monitoring system show an 8-point improvement in areas not covered by the project over the same period, suggesting that part of the change is seasonal: being honest on this point strengthens the report's credibility.

8Checking quality during and after data collection

Data quality is checked continuously during data collection, not once the survey is over, when it is too late to return to the field.

Daily high-frequency checks

Every evening, the MEAL officer downloads the day's data and applies a series of checks, ideally automated in a script or a workbook prepared in advance:

Revisits

A supervisor or an independent team revisits about 10% of households (at least a few households per enumerator), chosen at random and, as a priority, among interviews flagged by the checks. They ask 5 to 10 questions whose answers should not change (household size, water source, receipt of assistance). A high discrepancy rate for an enumerator leads to their interviews being redone and, if falsification is confirmed, to termination of their contract.

Data cleaning

1. Keep an intact copy of the raw data, never modified.
2. Keep a cleaning log: each change (ID, variable, old value, new value, reason, author, date) is recorded. An evaluator or auditor must be able to retrace the steps.
3. Handle missing values, inconsistencies and outliers according to rules written in advance, not case by case according to the desired result.
4. Recode frequent "other, specify" answers.
5. Calculate composite indicators (FCS, rCSI) according to standard methods, and check their distributions.

The data quality assessment

For indicators reported to the donor, a data quality assessment checks, on a sample of sites and periods, that the reported figure matches the sources. The data are recounted in the source documents (registers, signed lists) and compared with the reported figure: the verification factor equals the recounted figure divided by the reported figure. A factor of 0.85 means a 15% overestimate; beyond a 5 to 10% discrepancy, the cause must be identified (data entry error, double counting, different definition) and the reports corrected. The assessment also examines the five dimensions of quality: validity, integrity, precision, reliability and timeliness.

9Analysing to decide: the example of post-distribution monitoring

Fictional context: third post-distribution monitoring round of a cash transfer programme of EUR 60 per month for 1,800 households. 210 randomly selected households were interviewed by phone two to three weeks after payment, by MEAL team operators who are not involved in the distributions.

Main results

From analysis to decision

• Problematic payment point: reports of incomplete amounts and "contributions" demanded at a single point are a signal of fraud or abuse of power. This is not dealt with in a general meeting: it is immediately passed on to the compliance officer in accordance with the anti-fraud procedure, and any complaints are handled confidentially. Verification calls to all beneficiaries at that point are organised.
• Accessibility: 63% of persons with disabilities wait more than 2 hours. Decision: priority queue, option to designate a proxy, study of mobile money payment for these households.
• Information: awareness of complaints channels is low, especially among women. Decision: messages at every payment, sessions in women's groups, number printed on the beneficiary card.
• Transfer value: the rising share of rent and prices justifies sharing the analysis with the Cash Working Group to review the minimum expenditure basket.

A useful post-distribution monitoring note fits on two pages: the 5 essential findings, differences between groups, warning signs, and 3 to 5 recommendations with a proposed person responsible. The full tables go in an annex.

Analysing qualitative data

Notes from focus group discussions and interviews are transcribed the same day, then coded using a framework built from the analysis questions and enriched with emerging themes. The frequency of themes is counted by type of group (without turning them into percentages, which would give a false impression of representativeness), anonymised illustrative quotes are noted and dissenting views are actively sought. Qualitative analysis serves to explain the figures (why the share of rent is rising) and to identify what the questionnaire could not anticipate.

10Collecting data in difficult contexts

Phone surveys

They have become common, particularly for post-distribution monitoring and in remote management. They are quick and inexpensive, but they exclude people without a phone or network coverage, often the most vulnerable, and frequently under-represent women when the phone belongs to the head of household. Good practice: draw the sample from the beneficiary list and not from those who answer easily, plan at least three call attempts at different times, ask to speak to the registered person, limit the duration to 15 to 20 minutes, compare the respondents' profile with that of the target population and describe the bias in the report.

Common mistakes to avoid

• Enumerators drawn from the distribution team or from the community with interests in the selection: courtesy bias and risk of pressure.
• Freely replacing absent households with neighbours: bias towards households present that day, often less vulnerable. Replacements must follow a written rule and be counted.
• Reporting a result without a confidence interval or sample size, particularly for subgroups (a percentage calculated on 18 households with a person with a disability does not carry the same weight as a percentage based on 264 households).
• Presenting unweighted averages when the sample is stratified.
• Comparing a baseline and an endline conducted in different seasons without flagging it.

11Interview questions and key references

Interview questions

1. "Calculate the sample size needed for an endline survey in 30 villages. What information do you need?"

A good answer asks for the objective (estimating a proportion or measuring a change), the expected value of the indicator, the desired precision, the confidence level and power, the population size, the sampling design (clusters, strata), the design effect or ICC, the non-response rate and the need for results by subgroup. It can do the calculation and explains the trade-off between the number of clusters and the number of households per cluster.

2. "You discover that an enumerator has probably fabricated interviews. What do you do?"

It describes the signals (short durations, repetitive answers, improbable times), verification through revisits, removal and replacement of the interviews concerned, the disciplinary procedure, and preventive measures for the future (daily checks, supervision, rotation). It documents the incident and its effect on the sample.

3. "What is the difference between a statistically significant result and a result that matters for the programme?"

It explains that a small difference can be significant with a large sample without having any practical relevance, and that a large difference may not be significant with a small sample. It links the answer to the decision to be taken and to confidence intervals.

4. "How do you conduct post-distribution monitoring in an inaccessible area?"

It proposes phone calls to a random sample from the list, third-party verification, payment service provider data, key informants, and acknowledges the biases of each method. It mentions respondent safety and data protection.

5. "How do you combine quantitative and qualitative data in a report?"

It explains the sequencing (qualitative work explains or prepares quantitative work), triangulation, and gives an example where qualitative data explained an unexpected result.

Key references

• KoboToolbox and ODK documentation, and the XLSForm standard.
• SMART methodology manual for nutrition and mortality surveys.
• WFP technical guidance on food security modules and post-distribution monitoring.
• Cash Learning Partnership (CALP) tools and guidance on cash transfer monitoring, and harmonised post-distribution monitoring templates from Cash Working Groups.
• Questionnaires and methodological notes from the WHO/UNICEF Joint Monitoring Programme for Water Supply, Sanitation and Hygiene.
• USAID guidance on data quality standards and data quality assessments.
• WHO Ethical and Safety Recommendations for Researching, Documenting and Monitoring Sexual Violence in Emergencies (2007).
• ACAPS documents on humanitarian needs analysis and the use of secondary data.