Report on the Event 'Evaluation of Capacity Development'

In 2009 the Dutch Ministry of Foreign Affairs (Policy and Operations Evaluation Department (IOB)) initiated an evaluation with the aim to gain a better understanding of how and under what circumstances capacity development support can be effective, and what new insights can be gained from the recent capacity development programmes. The outcome of this joint evaluation will be presented by IOB in the autumn of 2010.   

 

In view of the emergence of new coalitions for MFS 2 and considering the importance attached to sustainable capacity development, the joint framework and the way the various evaluation partners are applying it to their own realities merit being shared with the development sector. In October 2009 it was agreed that PSO would take the lead by convening an event to sharing the insights of the ongoing process.  

Why this event?

The event had the following objectives

• Share the methodological insights from the IOB exercise to this date;
• Reflect on the meaning and usefulness of the framework: its theoretical roots, its underlying paradigm, its application.

The event coincided with the International Advisory Board meeting of PSO and thus could benefit from its members’ observations and input as well.

Highlights

Piet de Lange, coordinator of the IOB evaluation provided the fundament for discussion by presenting the framework of the study. This was followed by presentations by Jan Ubels (SNV) and by Jur Schuurman (Agriterra), who illustrated the position of the 'evaluees' in this process. Geert Phlix (Ace- Europe) presented the insights gained so far from her perspective as one of the evaluating parties (for PSO).

Presentation Piet de Lange (IOB)

This presentation elaborated on the background, the conceptual framework and some of the methodological issues related to the evaluation. Central in the evaluation are five capabilities (referred to as 5C’s). Click here for his PowerPoint presentation.

Main points:

• Mainstream evaluation reports do not give insight in what actually changes in terms of capacity. The effectiveness of Dutch support in contributing to those changes remains unclear.
• The starting point of the evaluation is not to have a single focus on accountability but also strongly zoom in on lessons learned.
• UNDP and the World Bank are struggling with the same issues. Drawing from the experiences of IOB, the UNDP is planning to have an evaluation using the same framework.
• The study is based on a general Terms of Reference compiled with the organizations participating in the study: Agriterra, Ministry of Health-Ghana, NCEA, (Netherlands Commission for Environmental Assessment) NiMD (Netherlands Institute for Multiparty Democracy, Partos (comprising Cordaid, Hivos, ICCO, Oxfam-Novib), PSO, and SNV, Netherlands Development Organisation. The study includes 25 case studies, 10 quick scans, 7 team leaders, 30 national consultants, 25 external referents.  
• The four core evaluation questions are:
  • What has changed in the capacity of the organisation?
  • How did changes in capacity affect output and outcome?
  • How effective was external (Dutch) support?
  • What can we learn?

The strength of the method applied in the evaluation is that it starts from a Southern perspective, acknowledging the relative position of the Dutch support. In the evaluation process space is given to Southern organizations to set their own indicators; at the same time it has to be ensured that there is consistency between the different cases.

In conclusion Piet shares his learning points:

-           Evaluating capacity development (CD) support is not an easy task, but some progress is being made. We have to be modest what we can achieve in this evaluation and not expect to get the final answer on issues related to CD

-           The shift from theory based to evidence based is demanding: routines do not apply and a different set of skills is needed.

-           At the level of Southern organisations there is need for a clear definition of system and outcome statement; this is often a struggle

-           Evaluating in collaboration with other parties asks a lot of communication, facilitation, management, with a lot of added value

-           But .... after all: the framework is a tool under development

• Who controls it, for what purpose?
• Who has the skills to use it, does it contribute to CD?

Piet concludes with a warning: “The worst thing that can happen that this tool becomes another instrument for accountability purposes among donors”

Discussion:

How do you deal with the fact that in the evaluation process different indicators might be formulated than when the interventions started?  
In most cases indicators were not set in terms of capabilities at the level of organizational development. Therefore monitoring information does not feed into the perspective IOB  uses in the evaluation. This is an important concern. SNV confirms that in their practice the 5C logic has not been used before. The fact that capacity development is now evaluated at the level of three SNV partners using a different framework makes this a learning evaluation and not one for accountability.

The focus of this evaluation seems to be on the effectiveness of capacity development interventions at the level of organizations. How are these effects on the context taken into account?
The final beneficiaries are the target groups of the organizations: the poor. In the study at village level we use the MSC (Most Significant Changes) approach. It may reveal the effects on the context, but still poses the question to what extent you can attribute changes to the support of the southern organization. A challenge is to relate this to the process of organizational strengthening.

Is there a difference between capacity development support and general development support? Because outcome/impact of CD processes and the outcome/impact of general development support are related.
IOB agrees this is an important issue. All case studies show specific objectives on CD as well as objectives in terms of mainstream development issues, which impact on CD as well.

What is the relation between this evaluation and the policy of the Minister?
We (IOB) are in a process of policy reconstruction and reviewing the theory of change of general budget support. This is a complex and complicated issue; the policy of budget support is not evaluated but debated. With the shift to general budget support it was assumed that partners would take their own responsibility for capacity development. It was thought that donors were needed to create conditions for development by making budget available; rather than countries needing strong organizations to be able to develop. We are now in a shift.

Presentation by Jur Schuurman (Agriterra)

Agriterra aims at strengthening farmer organizations in the South, so the core business is capacity development. Monitoring tools and instruments are designed to see how effective they are in this respect. Agriterra does not use the 5 capabilities but has a different set of indicators. They consider the framework that is used in the IOB study as interesting, and fruitful for partners to reflect on the capacity development process. Agriterra hopes that the evaluation will lead to evidence based analyses and reflection on the basis of which the models used by Agriterra and the 5 C model can be compared.

They observe that the membership character of organizations, in terms of participation and accountability, might not be well reflected in the core capability system. MDF, the evaluator of the Agriterra cases, will take this into account. (Geert Phlix, evaluator for  PSO remarks that she does not experience the problem of including the membership character in the 5C model. The model leaves sufficient space to make them relevant for these organizations) 
On the other hand Agriterra observe that the capability to adapt and self renew is not well reflected in the Agriterra Participatory Planning Monitoring & Evaluation (PME) system. In general Agriterra experiences the whole process as very positive.

The main learning point for Jur is that this evaluation process asks a lot of organizations, here and in the south. We have to communicate more than we think.

Discussion

Will MDF only look at capabilities of the farmer organizations or also at the impact they have on other stakeholders, like research institutes?

Agriterra: our beneficiaries are the members. We look at the impact of the membership organizations on the members only.

In evaluating effectiveness of capacity development interventions of a membership organization, how to differentiate between capacity of the core organization and of each of the members?
Agriterra: we want to strengthen the capacity of the membership organization to do what the members want. As someone adds: but still you have to differentiate between third party serving and member serving.

We also have to differentiate between capabilities and competencies.
Capabilities are made up of competencies, that lead to capabilities, that collectively lead to capacity. On the competency side, members are part and parcel of the organization. It is about competencies of farmers themselves to make their voice heard. Competencies go before capabilities, and capacity development interventions focus on competencies. We hear little about that in relation to actual practices.

Presentation by Jan Ubels (SNV)

The cases of SNV look at capacity development in multi actor systems; in the agricultural value chain in East Africa. Context is the main evaluator in the case of SNV. What is special about the working method of SNV is that the results of the three cases are discussed in an exchange workshop. In the workshop a diverse group of people is represented. Click here for the PowerPoint presentation.

SNV sees the capabilities as the ability to actually do things, it is about performance.  

The 5C framework was received more easily and positive than expected. Jan observes that the 5Cs help to better understand the dynamics that drive organizations than the Logical Framework Analysis (LFA). However, the trap is that the capability categories are going to guide the reality and are used as a ticking box. To prevent it becoming a rigid model we have to force ourselves to always start from the storytelling.

Discussion

It is important not to put the 5C glasses on when you start in the field.
If not, you miss beautiful surprised. You have to start to work with indicators that people use and can recognize. To start from the story; the interpretation game comes afterwards.

How do we deal with the balance between the core capabilities?
The cases show that the balance is very different in the different cases. We can see organizations survive in different ways.

What to do if ‘important indicators’ do not come up in the workshop?
IOB: Imagine the case that all information is gathered but we observe that leadership is not being mentioned. This has to be discussed. We should not take for granted what is generated by the workshop. This implies that the evaluator plays also a role to signal. We did not realize this implication of starting in the South. We loose the guidance of one set of indicators.

 

 

Presentation Geert Phlix, (Ace-Europe) evaluator for PSO

Geert shared the experiences evaluating the case of St Martin in Kenya, a partner of Dutch NGO Mensen met een Missie. This evaluation took place as part of the PSO evaluation. Click here for the PowerPoint presentation. Geert elaborated on the way the assessment workshops are being organized. First, the development of the organization is analyzed from a historical perspectives. Special attention is paid to factors that influence growth and the different phases of growth. The indicators were taken and scored along the different phases of growth. It was observed that new elements came in the discussion because of the 5Cs. The working process made people aware of their own processes of growth in relation to the 5C, with attention not only for explicit but also for implicit processes of CD.

Discussion

Where are the mainstreaming issues such as gender, HIV/Aids, environment in the 5C model?
Geert observes that an issue as gender will not come up if organizations have not mainstreamed gender in their functioning. So how to ensure that these issues are discussed?
As someone remarked: is it really a problem if these issues do not come up, if they are apparently not relevant for an organization?

Theme Group Discussions             

During the presentations and discussions various learning issues emerged. These were quickly categorized in three clusters as referring to (1) Planning Monitoring and Evaluation, (2) the evaluation as demanding exercise in need for strong Communication, (3) the 5C model.
Discussions continued in smaller groups chaired by the three international members of PSO’s IAB.

Summary of theme group Planning, Monitoring and Evaluation

-           We must explore how to combine PM&E of capacity development support with a ‘good professional’ theory of change.

-           We must explore if/how Northern policy issues receive sufficient attention in the negotiation of indicators (Gender, HIV/Aids, environment)

-           How does 5C relate to alternative models (E.g. European Foundation for Quality Management (EFQM) model , the INK model (Instituut Nederlandse Kwaliteit)  ?

-           There is a need to marry thematic expertise with change expertise

-           Let us please share Outcome Mapping experiences

-           We must explore the relationship between this evaluation of capacity development, and possible implication for the monitoring protocol that is needed under the new subsidy system of the Netherlands Government (MFS II) .

Conclusions

-           M&E should be power and culturally sensitive

-           Evaluation is about relevance, not about summing up monitoring results

-           M&E needs to cover multiple stakeholders

-           Monitoring should be more relevant, be MORE than accountability

-           Look at Outcome Mapping and MSC for M&E

-          Start with content & vision, than look at 5Cs

-          Development context is important in M&E

Summary of theme group: Communication  

-          Evaluating CD and CD support is not easy but progress is made

-          Shift from theory based to evidence based is demanding -> routines do not apply

-          Operationalisation of and risks of 5Cs

-          SNV uses original logic of 5 Cs and not final framework; why, what is learning behind this and what difference?

-          Tension/ dilemma: a top down model (gender) versus bottom up indicators

-          Communication and sharing experiences

-          Demanding exercise -> communication is accordingly demanding

Conclusion

Together we have responsibility:

-          don’t be just a teller, just a listener

-          Trust me or……leave me!

Summary of theme group on the 5C model

-          Explore the added value of the 5Cs in relation to other existing models?

-          Explore the link between the 5Cs and performance?

-          Study the usefulness of the 5Cs at network and institutional level?

-          Attention: The 5C model carries the risk (like all models) of becoming a box filling exercise.

-          Start with indicators people use before moving to the theoretical framework!

-          What framework is used to assess the effectiveness of CD interventions?

The meeting was closed by handing over the group conclusions to Director PSO Margo Kooijman and IOB coordinator Piet de Lange. Time constraints did not allow for further discussions and reactions.

Outstanding issues not addressed in the groups 

-          How to mainstream thematic issues like gender, HIV/Aids, environment?  How important is this for the evaluation questions Dominique ?

-          Who controls the capacity development evaluation instrument? For what purpose? (PdL)

-          Who has the skills to use it? (PdL)

-          Does it contribute to Capacity development (PdL)

-          Clear definition needed of system and outcome statement (PdL)

-          Sustain and self renew Dictatorship or selfish?

-          Why not used other models or additional models (e.g INK)