We Tried to Capture Rich but Hard-To-Measure & Unanticipated Changes. And This Is What We Learnt

FROM: Marcus Jenal, Steff Deprez, Zenebe B. Uraguchi – 18. January 2020

Three months ago, we started a regional initiative to capture and describe some of the effects of three youth employment projects of the Swiss Agency for Development and Cooperation (SDC). The RisiAlbania, EYE Kosovo and MarketMakers Bosnia & Herzegovina projects are implemented by Helvetas and its consortium partners in Eastern Europe (Partners Albania, Management Development Associates in Kosovo and KOLEKTIV in Bosnia & Herzegovina).  

The objective was twofold: to learn about and improve the projects’ contributions to more and better changes in the selected sectors, and to document and share the learning with other stakeholders. In order to keep the study manageable, we focused on the Information and Communications Technology (ICT) and Business Process Outsourcing (BPO) sectors.

Assessing and knowing about changes (including impacts) in development cooperation remains complex and nuanced. We therefore recognize the importance of understanding concretely what kind of changes happen.

A simple but profound question is: how do we get to know about changes in systems that happen but are not identified or expected by projects?

Tracing changes through Outcome Harvesting

We used Outcome Harvesting, a method that enables evaluators or projects to identify, formulate, verify, and make sense of outcomes of their initiatives. The method was inspired by the definition and is grounded in the same principles and practice of an outcome in Outcome Mapping: a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution (that are within the sphere of your influence).  

The method doesn’t measure progress towards predetermined outcomes or objectives, but rather collects evidence of what has been achieved, and works backward to determine whether and how the projects contributed to the change.

Outcomes were harvested in a two-day workshop with each project in October 2019. Besides the project teams, representatives of the consortium partners, co-facilitators and partly also sector partners participated in the workshops. A small number of key informant interviews to verify the outcomes harvested by the project team and to potentially find additional outcomes.

Based on the outcomes, the teams and consultant strung together a coherent narrative, that gives an easily accessible but comprehensive picture of what the respective project has achieved and how it has contributed to these changes.

A second two-day workshop was held with each team in November to complete the descriptions of the Outcomes, refine some of the categories to tag outcomes, and discuss the criteria to assess whether a change is systemic or not. Furthermore, the work of the teams during the second workshop led to a better understanding on how to rate the projects’ contribution in a more objective way.

For the substantiation of the outcomes, we didn’t choose to validate a selected group of individual outcomes and contribution statements by external sources, which is usually the case in outcome harvesting. Instead, we preferred to work with a selected group of experts in each country providing feedback on the credibility of the entire narrative that was developed for each project. Our idea was to minimize the likelihood that the team could in the worst case intentionally overstate the effect of the project but might in any case introduce an unintended bias in favor of the project.

As is common in Outcome Harvesting method, the prime informants of the study were the project teams themselves. This is one of the major limitations of our attempt. From an evaluation perspective, this can be problematic.  

Changes occur at different levels and in different time frames

Changes in the form of “impacts” are often defined as the subsequent effects on the target population or group, be it in the form of income, job, resilience or empowerment. For this to happen, changes at different levels need to occur.

However, what we learnt (and confirmed) from capturing and describing changes in the three youth employment projects is that changes don’t happen in a linear way. This is the one of the downsides of existing “impact logic” thinking in most projects, including those that apply a market systems development approach.

As Michael Kleinman forcefully argues, “our logframes are fictions – necessary fictions, but fictions nonetheless.” They ignore that development cooperation is becoming more complex not because of its size, cost and duration. It’s mainly because the systems in which development work takes place are complex adaptive systems.

Outcome Harvesting allowed the teams to collect and organize all changes they could observe, whether they were part of their plan or unexpected. Like this, the teams found that they have generated several unexpected (positive) changes. These changes had not been identified/recognized and documented before. One key lesson from this is the need for the readiness of staff to embrace learning and the requirement for skills and an inquisitive nature to detect patterns and understand power and influence.  

All the three youth employment projects are roughly half-way through their second phases. The focus of the study was on the effects of interventions from this second phase. Yet some changes that have only become visible now were contributed to by interventions from the first phase. An example is the job portal (duapune.com) in Albania which upgraded the features of its website (like filtering of candidates) and introduced payable online recruitment services. It took time for the services to become profitable and become financially sustainable. At the same time, some outcomes that are targeted for the second phase might not yet be visible as they take more time to materialize.

The way forward

After consolidating all outcomes and further revising the outcome narrative, we will organize a sense-making workshop, in which all three teams can work with the results of the study and discuss how these results can improve their understanding of what works and what does not in the project context.

Topics that we’ll touch upon are how the actual outcomes differ from the original plan, how a counterfactual narrative could look like, what the effect of the outcomes on gender and social inclusion is and how we can better understand the systemic effects of the projects as well as their contributions to the effects. We’ll also look into the future and try to understand what this all means for the future strategies of the projects.

An interesting discussion will be around the question of whether outcome harvesting can be integrated as a complementary method into the monitoring and results measurement efforts of the individual projects.

Additional sources

 

Cover picture: Helvetas

 

Marcus Jenal has been working in economic development for more than ten years. He started consulting in 2011 and is now a partner of the international consultancy Mesopartner. Marcus’ work has a particular focus on gaining a better understanding of economic systems and options to shape their evolution in a positive way. He also developed approaches to monitor and evaluate programmes that are facing uncertainties and complexity. He provides strategic advice to organisations and programmes on developing strategies and processes of discovery and sensemaking in complex and challenging contexts and is also an active researcher aiming to apply new economic theories to development.
Steff Deprez provides technical and strategic guidance for the design and roll-out of learning-oriented monitoring and evaluation systems for international development initiatives, particularly in the areas of value chain development, inclusive business, education, gender equality and women’s empowerment. He worked as a development practitioner for over 15 years and currently he works as an independent consultant. In 2017, he founded Voices That Count, a collaborative network of experts focusing on narrative-based research, monitoring and evaluation approaches. His work is inspired by the principles of complexity thinking and practical actor-centred approaches. The last 15 years, he has specialized in Outcome Mapping, Outcome Harvesting and SenseMaker. He is a steward and board member of the Outcome Mapping Learning Community and is an accredited and licensed SenseMaker Practitioner
Programme Coordinator, East and Southeast Europe, Senior Advisor, Sustainable & Inclusive Economies