ILO

Back to index

Sustaining Competitive and Responsible Enterprises (SCORE) programme, Phase III - Final evaluation

eval_number:

3164

eval_url:

https://webapps.ilo.org/ievaldiscovery/eval/3164

lessons_learned:

themes:

theme:

Monitoring and evaluation

category:

Organizational issues

---

comments:

Revising the M&E system to be more outcome / impact oriented has been also recommended in the previous evaluation (i.e. the Mid-term evaluation of SCORE Phase III : “Take steps to develop a more “impact-oriented” M&E system”). The Mid-term evaluation of Phase II also mentions flaws in outcome reporting: “The M&E system of SCORE is rather complex and although it produces good and up-to-date data on activities and outputs, it does not generate sufficient quality and reliable outcome level data.” Because SMEs and sectors are unique in size and situation, it is very difficult to aggregate outcome data in a useful way at the global level.” This recommendation was partially applied through impact assessments (IA). However, as mentioned above, IAs cover a limited number of companies and do not allow systematic corrective action. Applying this recommendation would avoid making “repeated mistakes” as mentioned with kindness and benevolence by an ILO Official during another ILO evaluation - speaking about lessons learned.

challenges:

A high number of output-oriented indicators and weak monitoring of outcome-driven data can jeopardize sustainability. A simple approach of asking the question "And so what" when evaluating the quality of an indicator can help assess more critically the extent to which a logframe is truly instrumental to achieve a sustainable programme. For example, based on existing indicators, one will know if a business plan per partner organization was developed and the number of people trained. "And so what?" How does this contribute to knowing to what extent implementing partners market, sell and organize training effectively? What other indicators could be more relevant to monitor the intended outcome?

success:

It is key to involve tripartite constituents, and change management and local experts in setting indicators that are also aligned with country monitoring needs. This will allow continuity between data collected during and after the project ends. Improving outcome and impact data design and monitoring contributes to facilitate change more effectively and in a more sustainable way. It also ensures that the programme can take relevant corrective action. This is not possible based on impact assessments conducted on punctual basis and after activities took place. Furthermore, as demonstrated for example in the IA conducted in India, experience shows that it is challenging to have companies participate in IAs if they have not been in recent contact with SCORE trainers in IAs. This explains the low number of companies covered (i.e. 10 SMEs).

context:

The SCORE Programme operated in 11 countries during its Phase III. Moreover it face serious challenges with regards to the COVID-19 Pandemic, making change even more challenging to occur. While the programme set up robust output-based M&E systems that was well managed, the programme design entails a linear approach in the Theory of Change and does not track relevant outcome / impact indicators. Experience show that the programme developed a complex M&E system that did not include monitoring needs at the country level (e.g concrete business figures to demonstrate training impact to prospective clients - rather than the percentage of companies reporting productivity and working conditions improvement).

description:

The programme developed a robust output-oriented M&E system rather than a more impact-oriented system. As a consequence, significant resources (in terms of time, human resources and efforts) have been allocated without being able to generate outcome- and impact-data. Collecting the latter is crucial in order to monitor change most effectively throughout the intervention, test causal steps and verify assumptions. Monitoring outcome-driven data allows to ultimately generate more impact through corrective action. The latter is to be undertaken on a regular basis and/or at an earlier stage. Pursuing quantitative targets that have weak relationships to outcomes affect the programme’s cost effectiveness and sustainability. There is the risk that output indicators may suggest a more positive picture about sustainability than it perhaps is the case. For such a complex programme requiring significant costs, it is particularly essential that a robust change management strategy developed by experts in the subject matter underlies the Theory of Change and the M&E system - taking into consideration that change is a non-linear process. The Theory of change should take into consideration the causal mechanism that underlies sustainable systems change. It should also integrate individual behavioural change, which is crucial when creating an enabling environment for SMEs to develop export and domestic industrial sectors.

administrative_issues:

Designing a more rigorous M&E system requires more time, more robust management, business and change management expertise often lacking in development programmes. However planning more time and resources to design more impactful interventions ultimately tackles better risks, and allows investing resources in more impactful programmes.

url:

https://webapps.ilo.org/ievaldiscovery/lessons/247158

---

location:

country:

Inter-Regional

region:

Inter-Regional

---

eval_title:

Sustaining Competitive and Responsible Enterprises (SCORE) programme, Phase III - Final evaluation