Lessons about baseline studies and dealing with high variability and external forces in evaluation work on climate change vulnerability

Based on the experience of conducting an initial evaluation study for climate change adaptation programming, CARE advocates now more than ever for integrating the principles of “inclusivity”, “triangulation” and “adaptive learning” into the design and implementation of studies.

 CARE’s programme, PRESENCES (Projet de Résilience face aux Chocs Environnementaux et Sociaux au Niger), part of BRACED , is a critical intervention addressing the dire consequences of climate change in a vulnerable region. The project aims to tackle identified forms of vulnerability by providing a combination of inputs that aims to reduce negative strategies for coping with climate shocks.

 How did we begin to address such a complex challenge? We conducted a baseline study, though which we explored the current severity and gravity of negative coping behaviour when facing a climate shock. Our baseline study has informed our entire programme.

This blog advocates for the importance of baseline studies and presents some key observations that apply to most development projects dealing with heightened variability and uncontrollable external forces.

In our view, a baseline study is more than an assessment of reference values against future progress and an expected impact. It represents the narrative about the context of the project, the stakeholders and the key challenges in delivery. It should be seen as the starting point – the first milestone – in a journey of learning, adapting, improving, and delivering impact.

A baseline study also outlines the starting point of the project and it sets the foundation for the whole M&E framework and its tools, methodologies and sources of information for both tracking inputs delivery and large-scale changes.

What did we learn from conducting our baseline study?

We learned three key lessons:  1) significant multi-stakeholder engagement is necessary for any evaluation, 2) an integrated and IT-enabled M&E approach linked with management response reduces delay between gathering evidence and making adjustments to a project or programme and 3) in climate change resilience, behavioural and livelihood changes matter a great deal and should be frequently monitored.

Learning 1: Choose a conceptual framework in which to generate evidence

In view of the complexity in appraising climate change resiliency, the 3As+T framework (Absorption, Anticipation and Adaptation plus tracking) was adopted because it was already validated within the BRACED programme and is conceptually aligned with CARE’s approach.

Each element in the framework is linked with the other: the ability to adapt to multiple, long-term and future climate change risks, and also to learn and adjust after a disaster; the ability to anticipate and reduce the impact of climate variability and extremes through preparedness and planning; the ability to absorb and cope with the impacts of climate variability and extremes.

Given the variety of contexts in which BRACED operates, the need to generate evidence is compelling both at the local level – to verify what works in that context – and at the international level – to outline comparable trends. For this reason, PRESENCES baseline adapted an impact measurement methodology that was previously utilised across selected CARE projects to reflect both contextual forces and common challenges vulnerable groups face when coping with extreme shocks.

The 3As continuum is well-embedded in PRESENCES impact assessment tool as it probes target groups around immediate food insecurity and long-term strategies on the way assets are managed at the household level. Addressing both areas of inquiry (food and assets) during monitoring and evaluation should shed critical information on the kind of impact facilitated by PRESENCES’ delivery model.

Learning 2: Consider methodological challenges and opportunities when measuring resilience

The wide range of variability that a climate change phenomenon manifests means that any project focusing on resilience must use a broad lens of analysis. In fact, from the landscape analysis of Tilabery region in Niger, many trends were identified (rainfall, temperature etc) – with no single one decisive. Because of seasonal variances every year, it becomes difficult to forecast where the shock is going to take place and its intensity with a high degree of geographical specificity.

PRESENCES’ baseline involved a broad analysis of livelihood, access to climate information, rural practices, savings, financial inclusion and conflict over natural resources management. This wide array of variables was selected and analysed as they all represent possible dimensions of vulnerability when local communities get affected by climate shocks.

The methodological challenge to appraise the degree of shock and response cannot be captured solely from evaluation studies; it also requires significant monitoring. When it is feasible depends on available resources (make sure to budget for those!) and how information generated at baseline enables the project’s team to identify the most significant vulnerabilities and issues to be closely monitored.

In view of this fundamental barrier between large-scale studies and key vulnerabilities monitoring, a sample of target groups will be appraised regularly. When possible, frequent representative and research studies with adapted tools for rapid appraisals should look at both quality of input delivery and outcome-level behavioural changes.

In the context of PRESENCES, the analysis of both dimensions ensures a continuum between short-term results from quick responses to shocks, and longer-term aggregated shifts in the ability to cope.

Learning 3: Consider the ‘reality of the field’ when sampling for gender and complexity

The ability to collect livelihood data at the domestic level usually rests on the head of the household. In Niger, it has been a real challenge to disaggregate the sample by gender in equal proportion, since power is not distributed equally at the domestic level.

In response to this structural bias, PRESENCES baseline sample was 67% men and 33% women – the latter value being 5% above what would be derived through secondary sources. The team conducting the baseline study thought such variation to be a good practice as it prioritised each community to ensure all women-led households would be available during data collection.

The reality of the field also did not allow for a random strategy to engage communities during data collection. As in many countries, in Niger it is good practice for community leaders to be solicited so that as many respondents as possible can be informed and find time to share information on an agreed day.

CARE Niger drew on its strong reputation at the community level across all targeted sites of intervention, based on several years of development work in the region, to ensure respondents’ active participation in the process.

Seasonality is another important variable that the PRESENCES team considered before initiating a large-scale study. As much as the context determines certain activities more than others, the same thinking applies when approaching a significant number of people at the same time.

The year is constellated by various religious, social and natural events that constrain availability of communities; therefore the data collection took place before Ramadan and harvest, when time becomes an issue for all.

In synthesis, from PRESENCES experience, the major operational concerns that ensured an appropriate level of data collection quality were: distribution of power within the household, access to respondents, time and seasonality.

Learning 4: Generate recommendations that are strategic and operational

PRESENCES findings generated two levels of recommendations with methodological and management applications.

For this specific case, we used the coping strategy index (CSI), which was developed by CARE, the World Food Program and TANGO, to measure degrees of vulnerability. CSI provides an indicator of household food security that is simple and quick to use and provides a score that indicates whether the status is declining or improving. Its use seems essential if analysed in its gender, geographic, and livelihood specifications. The baseline results showed great differences in these scores between men and women and geographical areas.

As CSI information gets collected during large-scale studies and various research strands, specific assumptions about the socio-economic and climatic circumstances will also be tracked, and assessed within the MEL system in order to identify their influence on the project’s impact.

Baseline recommendations also generate useful insights into the way the project’s inputs should be managed. To provide an example, an in-depth analysis on access to improved seeds was carried out. Evidence showed a correlation between their utilisation and food security but it also highlighted limited access to the input. This information brought the risk of a bottleneck in delivery to the surface and prompted the project team to consider alternative delivery avenues for the improved seeds. All future appraisals are expected to generate recommendations aimed at improvements in project delivery.

Learning 5: Identify the approach for each evaluation, and the links between them

The sequence of evaluation studies (baseline, midline and endline) will progressively shift focus from analysis of implementation to analysis of how the various inputs led to the expected results. The context of high variability in which the project is delivered means there may be a significant degree of variation in how results are distributed. Therefore, a real-time monitoring system will provide context-specific evidence based on key studies and relevant information during the onset of a crisis/shock. It is worth mentioning that all future data collection will be done digitally (as shown in the photo at the top of this blog) in order to increase efficiency in accessing and processing information.

The baseline study in PRESENCES enabled a broader reflection on how to proceed with future large-scale appraisals for the project. It was a learning experience in terms of understanding the context where the project operates, information accessibility and the validity of selected hypothesis.

The mid-term review will intersect monitoring information with additional data collected from the target communities. Combining information and analysing gaps in delivery at this stage requires particular attention as the appraisal will also explore responses by project managers to evidence produced during implementation.

The endline study will include a final appraisal of the process, although its main focus will be on results. The magnitude of contribution of selected inputs to the project’s impact will be measured, along with the project’s capacity to absorb and share M&E-generated feedback. Methodological consistency between baseline and endline will be a priority in order to better aggregate and triangulate relevant information. The endline will also leverage on a very large amount of monitoring information, produced during implementation, which will help when measuring to what extent the expected results have been reached.

Conclusion: Evaluation needs to be participatory

The baseline established a framework wherein PRESENCES future evaluation studies will progressively focus on complex behavioural changes in a context with uncontrollable forces and varying degrees of crisis severity.

The proposed evaluation approach attempted to strike a balance between rigour, triangulation and participation of different stakeholders. The fulfilment of these principles requires a diversified and validated set of tools and methodologies which CARE has developed and shared in an iterative and step-by-step manner with local communities, implementing partners and field supervisors.

An environment that is conducive to multi-stakeholder interactions is a necessary condition to produce and share evidence/learnings across the consortium. To this end, the PRESENCES team will continue to promote dialogue on tools, methodologies and evidence, both downward with communities to deepen quality of information, and upward with national-level institutions to influence policy change and government response to climate shocks.