Accountability, evaluation and learning

1Accountability to affected populations

Accountability to affected populations (AAP) rests on three pillars: informing (who we are, the selection criteria, what people can expect), involving (consulting communities at every stage) and listening and responding (complaints and feedback).

2The complaints and feedback mechanism

• Multiple channels suited to the context: toll-free hotline, suggestion boxes, help desks, committees, field staff. They must be accessible to women, children, older people and people with disabilities.
• Categories: request for information, suggestion, complaint about the quality of assistance, sensitive complaint.
• Sensitive complaints (sexual exploitation and abuse, fraud, corruption): a separate confidential channel, passed directly to the designated focal point, with referral of the survivor to support services. Never handled by the local team implicated.
• Response times known to everyone, and a closed loop: the person receives a response.
• Regular analysis of trends, presented to managers so that the programme can be adjusted.

3Evaluation

Types of evaluation: real-time (at the start of an emergency), mid-term, final, ex post (after the end), impact. It may be internal, external or mixed.

The six OECD DAC criteria (revised in 2019):

1. Relevance: does the intervention respond to people's needs and priorities?
2. Coherence: does it fit with other interventions and with norms?
3. Effectiveness: is it achieving its objectives?
4. Efficiency: are resources used economically and in a timely way?
5. Impact: what long-term changes, positive or negative, intended or unintended?
6. Sustainability: will the benefits last after funding ends?

The terms of reference (ToR) set out the context, purpose and use of the evaluation, the evaluation questions by criterion, the methodology, the timeline, the deliverables, the evaluator's profile and the budget. After the evaluation, a management response states for each recommendation whether it is accepted, by whom and when it will be implemented.

4Keeping learning alive

• After-action reviews (30 to 60 minutes, just after an activity): what did we plan? what happened? why? what will we do differently?
• Quarterly reviews of the tracking table with the whole team, including partners.
• Knowledge capture: short lessons-learned notes, shared with the cluster and other NGOs.
• Feedback to communities: present the results to the people who responded to the surveys.

5Designing a complaints and feedback mechanism: the complete method

A complaints and feedback mechanism (CFM or CFRM) cannot simply be decreed: it is designed with communities, equipped with written procedures, staff and a budget, and judged by its ability to produce responses and changes. Commitment 5 of the Core Humanitarian Standard and the IASC Commitments on Accountability to Affected People (revised in 2017) are the main references.

Step 1: consult before designing

Organise separate focus group discussions (women, men, adolescent girls, adolescent boys, older people, persons with disabilities, minorities) with four questions: through which means do you prefer to receive information? whom do you trust to report a problem to? what would stop you from complaining? how would you like to receive a response? The answers vary widely: in many contexts, women have limited access to phones and prefer to speak to a woman in person; young people prefer messaging apps; older people trust leaders, whom women or minorities may on the contrary fear.

Step 2: choose and combine channels

Step 3: categories and timeframes

Step 4: the database and the feedback loop

Each piece of feedback is logged in a database with: a unique code, the date, the channel, the location (at site or village level only), the sex and age group of the person, disability if they wish to state it, the category, the sector, a summary, the person assigned, the actions taken, the date and means of response, satisfaction with the response and the closure date. The person's contact details are stored separately and deleted after closure. Sensitive complaints appear in the general database only as a code with the note "referred", without any details.

A complaint is only closed when the person has received a response, even if that response is "we cannot include you, and here is why". A high rate of complaints marked "handled" with no response communicated is the most common weakness found by accountability evaluations.

6Sensitive complaints and protection from sexual exploitation and abuse

Complaints of sexual exploitation and abuse (SEA) by humanitarian staff are the most serious a mechanism can receive. Their handling follows strict rules, derived in particular from the IASC Six Core Principles Relating to Sexual Exploitation and Abuse (adopted in 2002, revised in 2019), the 2003 UN Secretary-General's Bulletin on special measures for protection from sexual exploitation and sexual abuse, and the IASC Best Practice Guide on Inter-Agency Community-Based Complaint Mechanisms.

The principles to apply

• Survivor-centred approach: safety, confidentiality, respect for the survivor's choices and non-discrimination. The survivor is not asked to tell their story several times.
• Assistance first: the person is referred immediately, with their consent, to available services (health, psychosocial support, legal aid, protection) through the GBV referral pathway established in the area, whether or not the complaint is subsequently substantiated.
• Strict confidentiality: only the designated focal point and the investigation function know the identity of the people involved. No information circulates through the usual channels.
• No investigation by the MEAL or programme team: staff receiving the complaint record the facts reported without investigating them, then refer it. The investigation is carried out by trained investigators, in accordance with the organisation's policy.
• Duty to report: humanitarian staff are obliged to report any suspicion through the established channels; keeping silent is itself misconduct.
• Children: any sexual activity with a person under 18 is prohibited regardless of the local age of majority or consent; the best interests of the child guide all decisions, with the support of specialised child protection actors.

The standard pathway for a sensitive complaint

1. Receipt through any channel or staff member; listening without intrusive questions; informing the person about next steps.
2. Immediate referral to assistance, with consent.
3. Referral to the PSEA focal point within 24 hours, through a secure channel (never through group messaging or by email with multiple people copied).
4. Assessment by the focal point and management: protection measures (including temporary removal of the subject of the complaint if necessary), informing headquarters, notifying the donor in line with contractual obligations (most donors require prompt notification of SEA allegations, often within a few days; check the agreement).
5. Investigation by a trained independent function.
6. Decision and disciplinary measures; informing the complainant of the outcome, while respecting confidentiality.
7. Anonymised trend analysis to strengthen prevention.

Serious mistakes never to make

• Handing the complaint to the local team leader, who may be connected to the subject of the complaint.
• Organising a confrontation or mediation between the survivor and the subject of the complaint.
• Discussing the case in a community or team meeting.
• Requiring evidence before referring the person to assistance.
• Dismissing a complaint because it is anonymous: it can trigger discreet verification and stronger prevention.

A mechanism that receives no complaints of sexual exploitation or abuse is not a sign of good health: in most contexts, it is a sign that people do not know the channels or do not trust them.

7Full example: analysing six months of feedback data

Fictional context: the multi-sector project of the NGO "Horizons Partagés" in Haut-Kessi province received 1,146 pieces of feedback in six months, for about 46,000 people concerned.

Breakdown by category and by channel

The analysis

• Requests for inclusion (28%) exceed the 14-day target timeframe. Content analysis shows that 70% come from host families, who do not understand why they are not eligible for cash transfers. This is primarily a problem of information on the criteria, not of complaint handling: the mechanism reveals a weakness in the programme's communication.
• Women make little use of the hotline (29%) but are the majority at help desks (61%). Closing the help desks to cut costs, as had been proposed, would exclude a large proportion of women.
• One sensitive complaint was referred in 4 days instead of 24 hours, because the CFM assistant did not know how to reach the focal point over a weekend. Measure: an on-call number for the focal point and a designated deputy, known to all staff.
• The boxes receive few messages and responses cannot be communicated. Measure: keep them in two sites where the committees request them, and remove them elsewhere.
• The total volume of feedback (about 2.5 per 100 people concerned in six months) indicates a mechanism that is known and used; however, the monitoring survey finds that only 49% of women know at least one channel, compared with 70% of men.

Decisions taken at the review committee

The loop is then closed at collective level: at the following month's community meetings, the team presents what was heard and what is changing ("you told us… we did…"). This feedback builds trust in the mechanism and encourages its use.

8Managing an evaluation: from terms of reference to the final report

The MEAL professional is rarely the evaluator; they are most often the evaluation manager. It is the evaluation manager who ensures that the evaluation will be useful, credible and ethical. The ALNAP Evaluation of Humanitarian Action Guide (2016) and the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation (2016), together with its Ethical Guidelines (2020), are the basic references.

Criteria specific to humanitarian action

In addition to the six OECD DAC criteria, humanitarian evaluation often uses adapted criteria, drawn from ALNAP's work:

• Coverage: did the assistance reach the groups that needed it, in proportion to their needs? Who was excluded, and why?
• Connectedness: were emergency activities carried out taking into account longer-term problems and their links with what follows?
• Coordination: was the intervention coordinated with other actors, and did this coordination improve the response?
• Appropriateness (appropriateness): was the assistance tailored to local needs and preferences, thereby strengthening accountability?

Standard timeline for an external final evaluation

The evaluation matrix: an example

Budget and quality

An external final evaluation of a medium-sized project commonly involves 25 to 40 consultant days, plus fieldwork, translation and workshop costs; budgets of between 1 and 3% of the total project budget are common, more for impact evaluations. Quality can be checked against a few simple elements: do the conclusions clearly follow from the findings, and the findings from the data? Are the limitations explicit? Are the recommendations few in number, prioritised, addressed to a specific recipient and feasible? Are the voices of affected people, including marginalised groups, present?

The evaluation manager protects the evaluator's independence: they may ask for factual errors and methodological weaknesses to be corrected, but never for a conclusion to be changed because it is unwelcome.

9From evaluation to learning: management response and adaptation

The management response: a template

A recommendation may be rejected, provided the rejection is reasoned. Following up the management response, at 6 and 12 months, is what distinguishes a learning organisation from an organisation that commissions reports.

Building a learning agenda

• Learning questions: 3 to 5 questions defined at start-up, drawn from the most uncertain assumptions in the theory of change (for example "do support groups that include grandmothers change feeding practices more than groups of mothers alone?").
• Planned learning moments: after-action reviews after each major distribution, quarterly reviews with partners, a mid-term reflection workshop, a lessons-learned workshop at the end of the project.
• Adaptive management: each review leads to decisions recorded in a log; major changes are discussed with the donor, backed by data.
• External sharing: presentation of results to the cluster, publication of evaluations (most major donors and networks encourage publication, for example in the ALNAP library), two-page lessons-learned briefs.
• A culture that accepts mistakes: a team only learns if people can say "that did not work" without being penalised. Management leads by example by presenting its own mistakes.

Variations by context

10Interview questions and key references

Interview questions

1. "How do you know that a complaints mechanism is working?"

A good answer cites specific indicators: awareness of channels measured by survey and disaggregated, number and profile of users, response times, rate of closed loops, satisfaction with the response, and above all documented programme changes. It stresses that a mechanism without sensitive complaints is not necessarily healthy.

2. "A beneficiary tells you that a distributor is asking for sexual favours in exchange for registration. What do you do within the next hour?"

It listens without investigating, ensures safety and confidentiality, offers referral to services with the person's consent, refers the case to the PSEA focal point through a secure channel within 24 hours at the latest, tells no one else, and does not contact the subject of the complaint. It cites the IASC principles and the survivor-centred approach.

3. "Write three evaluation questions for a water and sanitation project."

It formulates evaluative questions ("to what extent…", "for whom…", "why…"), linked to criteria (effectiveness, sustainability, coverage), and specifies sources and methods. It avoids closed or purely descriptive questions.

4. "The evaluator concludes that your project was not very effective. Management wants to change the report. What do you do?"

It distinguishes factual and methodological corrections (legitimate) from changes to conclusions (unacceptable), recalls the UNEG and ALNAP independence standards, and proposes using the management response to express the organisation's reasoned disagreement.

5. "Give an example of a lesson learned that changed a practice."

It describes a specific case, how the lesson was identified (after-action review, evaluation, complaints data), how it was disseminated and what concretely changed, with evidence.

Key references

• Core Humanitarian Standard (CHS), 2024 edition, Commitments 1, 5 and 7.
• IASC Commitments on Accountability to Affected People and Protection from Sexual Exploitation and Abuse (revised 2017) and the Collective AAP Framework.
• IASC Six Core Principles Relating to Sexual Exploitation and Abuse (revised 2019); UN Secretary-General's Bulletin ST/SGB/2003/13.
• Best Practice Guide: Inter-Agency Community-Based Complaint Mechanisms: IASC (2016).
• Evaluation of Humanitarian Action Guide: ALNAP (2016).
• Norms and Standards for Evaluation (2016) and Ethical Guidelines (2020): UNEG.
• Better Criteria for Better Evaluation: OECD DAC (2019).
• Inter-Agency Humanitarian Evaluation (IAHE) Guidelines: IASC.