Six Challenges Organisations Face During Impact Assessments
Even a well-planned impact assessment can face practical challenges once the assessment begins. Programme records may be incomplete, baseline data may be missing, beneficiaries may be difficult to reach, and the available evidence may not fully answer the evaluation questions.
These challenges and limitations do not prevent an impact assessment from being conducted. However, they can influence the methodology, the strength of the evidence and the conclusions that can reasonably be drawn from the findings.
Consider Project Unnati, an illustrative financial inclusion programme for adolescent girls that we have used in our earlier articles. The programme has ended, and the organisation now wants to understand whether the girls continued using digital financial services and became more financially included.
As the assessment team begins its work, it encounters several practical challenges. Some beneficiary contact details are outdated. Baseline information on financial behaviour is limited. Not all girls can be reached, and some beneficiaries may find it difficult to remember their financial behaviour before the programme.
How should the assessment team respond to these challenges without compromising the credibility of the assessment?
In this article, we’ll examine six common challenges organisations face during impact assessments and practical ways to address them.
Challenge 1: Unclear Evaluation Questions and Scope
Organisations often begin an impact assessment with a broad objective such as “We want to understand the impact of our programme.” While this expresses the overall objective, it does not specify what the assessment actually needs to answer. Without clear evaluation questions, the assessment can become unfocused and collect information that may not be useful.
How to Overcome It
Begin by translating the broad objective into specific questions linked to the programme objectives and Theory of Change. These questions can then guide the scope, indicators, methodology and evidence required.
Project Unnati in Practice
An organisation may begin with a broad question such as:
“What was the impact of Project Unnati?”
This needs to be translated into more specific questions. In Project Unnati, the organisation was clear that it wanted to understand:
- Did the girls continue using digital banking services independently?
- Did their financial confidence improve?
- What changes in financial inclusion occurred, and what contributed to those changes?
These questions provide a clearer basis for defining the scope, indicators and methodology for the assessment.
Challenge 2: Weak or Missing Baseline Data
One of the common challenges in impact assessment is assessing change when little or no baseline data is available. Without information about beneficiaries’ situation before the programme began, it can be difficult to determine how much change occurred and what role the programme may have played in that change.
How to Overcome It
Start by reviewing available monitoring data, programme records and other existing evidence. Where baseline data is unavailable, the assessment can use retrospective questions, qualitative methods and relevant secondary data to understand the situation before the programme. However, these approaches have limitations, which should be clearly acknowledged when interpreting the findings.
Project Unnati in Practice
Suppose Project Unnati did not collect baseline data on how girls made payments before joining the programme. Three years later, the impact assessment finds that 65% of the girls surveyed now use digital payments independently.
The finding tells us their current behaviour, but without baseline data, we cannot directly determine how much this changed because of the programme.
The assessment team could use retrospective questions such as:
“Before joining Project Unnati, how did you usually make payments?”
and explore the change further through FGDs and available programme records. However, because beneficiaries are recalling behaviour from several years earlier, the findings should be interpreted carefully due to the possibility of recall bias.
Challenge 3: Difficulty Reaching Beneficiaries
Impact assessments are often conducted months or years after a programme has ended. By then, beneficiaries may have changed their contact details, moved for education or employment, or migrated after marriage, making them difficult to reach. This can reduce participation and affect how representative the findings are.
How to Overcome It
Use multiple ways to reach beneficiaries, including alternate contact details, programme staff, community networks and different survey modes such as phone, digital or in-person surveys. Where beneficiaries remain unreachable, compare available information about respondents and non-respondents to identify possible non-response bias.
Maintaining updated beneficiary and alternate contact information throughout the programme can also make future impact assessments easier.
Project Unnati in Practice
Suppose the team can reach only 650 of the 1,000 girls. Some may have migrated due to marriage, education or employment. The team could use programme and community networks to trace them and offer different ways to participate. If a substantial number still cannot be reached, the assessment should examine who is missing and consider this limitation when interpreting the findings.This helps the assessment team consider whether the experiences of those who could not be reached may differ systematically from those who participated.
Challenge 4: Choosing the Right Methodology and Indicators
Choosing appropriate methodologies and indicators is one of the most common challenges in impact assessment. Evaluation questions that are not aligned with the programme’s Theory of Change, or indicators that do not adequately measure the intended outcomes, can result in evidence that does not answer the questions that matter.
How to Overcome It
Begin with the Theory of Change and programme objectives to define the evaluation questions. Then define the scope by identifying the beneficiaries, geographical coverage, assessment period and outcomes to be examined. Based on this, identify relevant and measurable indicators and select quantitative, qualitative or mixed methods that can generate the required evidence.
The methodology should also consider the availability and quality of existing data, as some analytical approaches require sufficient historical or longitudinal data and may not be appropriate where programme data is limited. Where the assessment seeks to understand the programme’s contribution to observed changes, the methodology should be appropriate to that question and should consider other factors that may have influenced the results.
Project Unnati in Practice
If Project Unnati’s Theory of Change expects digital financial literacy to lead to greater independent use of financial services, the assessment could measure the percentage of girls independently using digital payment methods. FGDs could then explore what enabled or prevented girls from continuing to use these services.
Challenge 5: Ensuring Data Quality and Minimising Response Bias
Poor-quality data can affect the credibility of an impact assessment. Incomplete responses, inconsistent data collection, poorly framed questions and response bias can all influence the findings.
How to Overcome It
Data collection tools should be pilot tested, field investigators properly trained and data quality regularly monitored. Questions should be simple, neutral and aligned with the indicators being measured. Findings should also be compared across different sources and methods wherever possible.
Project Unnati in Practice
If girls are asked, “Did Project Unnati help you become confident in using digital payments?”, they may give a positive response because they feel it is expected.
Instead, the assessment could ask about actual behaviour:
“When you buy something, how do you usually pay?”
“When you make a digital payment, do you do it yourself or does someone help you?”
These responses could be compared with insights from FGDs to better understand whether girls are independently using digital financial services.
Challenge 6: Turning Findings into Decisions
An impact assessment may generate valuable evidence, but its value is limited if the findings remain within a report. Organisations often struggle to turn findings into decisions about future programmes, strategies and funding priorities.
How to Overcome It
Recommendations should be practical, prioritised and clearly linked to the evidence. Findings should be discussed with programme teams, management, funders and other relevant stakeholders to identify what can realistically be acted upon and what should inform future programme design.
Project Unnati in Practice
Suppose the assessment finds that many girls can use digital payment services independently, but continued use is lower among girls who have limited access to smartphones or family support.
Rather than simply reporting this finding, the organisation could use it when designing future financial inclusion programmes by considering device access, family engagement and post-training support as part of the programme design.
Project Unnati In Practice
Suppose the assessment found that girls had improved their financial knowledge but continued to face barriers due to limited family support. The organisation could use this evidence to include family engagement and post-programme support when designing future financial inclusion programmes.
Conclusion
Challenges are a normal part of conducting an impact assessment. Many impact assessment problems, including missing baseline data, difficulty reaching beneficiaries and methodological limitations, do not necessarily make an assessment invalid. What matters is how these challenges are identified, addressed and reflected in the interpretation of the findings.
A well-designed impact assessment should be realistic about the available evidence and transparent about its limitations. By defining clear evaluation questions, selecting appropriate methods, strengthening data quality and using findings for decision-making, organisations can generate evidence that is both credible and useful.
The goal is not to produce a perfect assessment, but to generate the best possible evidence within the programme context and available resources, while being clear about what the evidence can and cannot tell us.
Continue the Conversation
Every impact assessment comes with its own practical challenges. The right approach depends on the programme context, available evidence and the questions the assessment needs to answer.
If you are navigating a challenge in your impact assessment or have found a different way to address one, we’d be interested to hear your experience. Let’s continue the conversation and learn from what works in practice.
#ImpactAssessment #MonitoringAndEvaluation #ProgrammeEvaluation #EvidenceBasedDecisionMaking #SocialImpact #CSR #Nonprofits #MonitoringAndEvaluation #SocialImpact #TheoryOfChange #SocialImpact #ImpactMeasurement