The Alliance for Community Action (ACA) has since October 2015, been implementing the EU-funded Speak Up Zambia project with the aim of empowering Zambian citizens to have a voice and enabling media to act as watchdogs for society. The ACA jointly implements the project with Free Press Unlimited (FPU) and House of Consciousness (HoC)
This project consists of specifically three result areas: 1) increased capacity of citizens and media to monitor public resource management, 2) increased ability of media to exercise watchdog role, and 3) increased media output that empowers women and amplifies their voices. The main activities under those areas are the training of young people with no journalistic training and sometimes without a full secondary education to be citizen journalists who report on service delivery weaknesses in their own communities. The project also trains fact checkers to verify claims made by duty bearers and other actors on public resource issues and service delivery.
Over time, we have been trying to find and adopt an evaluation method which would give us a fairly accurate understanding of the impact of the project over and above the measurable, quantitative outcomes. We were looking for an evaluation system that would enable us to have an understanding of how the project has impacted direct beneficiaries who are part of the implementation process.
As part of the interim evaluation of this project, the Most Significant Change (MSC) method has been used to evaluate what specific changes the project contributes to in regards to these result areas. MSC is a participatory, democratic and systematic, story-based method to facilitate learning.
Our MSC evaluation was led by a consultant from Free Press Unlimited. It consisted of conducting 32 interviews with a cross section of young women, journalists in radio stations, fact-checkers and citizen journalists who had been trained by ACA and HoC.
Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Thus project beneficiaries report in their own words what had changed most in their lives and work as a result of the Speak up Zambia Project.
Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact. As MSC focuses on impact form beneficiary perspective, It is important that it is paired with another evaluation methodology which would help to measure the predetermined numerical or other measurable outcomes.
Initially a little unsure how asking beneficiaries of project activities what they though was the most significant change in their lives which was a result of their interaction of the project would produce valid answers the team went out to test MSC as much as evaluate the project.
The results were most fascinating. The interviewed Citizen journalist and radio station practitioners brought out impact of the project in their personal lives and work that the project would not have known. For example a major change in the citizen journalist came out as the development of confidence and self-assurance they previously did not possess. The girls especially said they had gone from being “nobodies” to individuals recognised by their communities. Another change coming from the Citizen journalists was the new found ability to understand national issues and to see themselves as having the ability to not only engage with them but bring about change for their communities.
The ACA would like to highly recommend MSC as a very helpful means of identifying unexpected changes in a project. Unlike traditional monitoring techniques that focus largely on monitoring activities and outputs, MSC focuses on monitoring intermediate outcomes and impact, MSC does not make use of pre-defined indicators, especially ones that have to be counted and measured; it therefore affords us the opportunity to learn the impact of our own activities beyond the defined outcomes.
End.