Reflections from Practice – Learning, Monitoring, and Evaluation in Transitional Justice Programmes

Undine Whande,Transitional Justice Programme, Centre for the Study of Violence and Reconciliation

Introduction

What makes any social-living-system work is not how cleverly it is conceived and mapped, but how wisely and mutually it is understood and valued, enabling those who have and take leadership to see and work with what is there and what is possible, and with each other (Doug Reeler, CDRA, 2010)

In the past ten years, the development sector has seen a widening polarity in the evolution of Monitoring and Evaluation (M&E) methodologies. On the one side of the polarity is an increasing awareness that monitoring and evaluation practices need to be context-, actor- and organisation-specific, tailormade and rooted in human relationships. There is a realization that if M&E is to be experienced as use- and meaningful, it requires regular quality time for reflection and conversation, in organizations, among peers and between grantmakers and grantees. This has led to the honing of emergent, conversational approaches, now known broadly under the heading ‘developmental’ or ‘emergent’ evaluation practices (Quinn Patton 2011; Westley, Zimmerman, Quinn Patton 2007, Kaplan and Davidoff 2011; Wheatley and Frieze 2006). On the other side of the polarity, however, the still more established, linear logframe-based approaches, whose origins date back to World War II, remain the norm and base of many a development programme.

The logframe became prominent in the late 1970s and was further developed by aid agencies, in particular German Technical Cooperation (GTZ), as a participatory planning tool. It was then evolved into a widely applied methodology known in English under the heading GOPP – Goal Oriented Project Planning. This method, to my experience, often resulted in vast amounts of little note cards with many ideas; it presented a useful brainstorming tool. However, a sense of success in generating a participatory mode was often dampened when the emerging complexity, the many issues and strands that local development actors would raise and then want to discuss, were in the end once more condensed into the confines of cause-consequence thinking. In the attempt to reduce complexity in order to make the social processes observed ‘manageable’, the GOPP process often flattened the experiential realities of local actors back into linear outcomes that could be ‘managed’. The ‘problem tree’ for instance, was one such tool often used to fixate a cause-consequential chain of envisioned results. Rather than generate conversation about the different layers and levels of interdependency between the phenomena observed in a given situation, I witnessed people spending a lot of time debating whether poverty was a root cause or a symptom of violence. The value of the tool, however, rested not in deciding on a uni-directional cause-consequential link but in lifting out interdependencies and generating reflection on what could be observed. It was precisely in the disagreements about causality and linearity that the value of the conversation generated by the problem tree could be found.

An increasing emphasis has since been placed by development aid funders on refining results-based monitoring through the use of such technical tools, in particular the now popular so-called ‘results-chains’, arguably at the expense of the initial participatory intent, and definitely at the expense of valuing and prioritizing the messy nature of conversation grappling with complexity. For the majority of grantees this has meant working with tight templates to gather statistical data; to measure what they feel can hardly be usefully measured though it can and be experienced, described and made sense of. Meanwhile, there is an emphasis on an M&E orientation that espouses notions of ‘evidence’, ‘proof of effect’ and ‘efficiency’. In particular, the terminology of effectiveness and efficiency govern proposal language. The hunt for visible effects and cost-benefit analyses are seen to be markers of both the quantitative and qualitative success of an intervention, but the terms are rarely defined or explored further in their meaning. However, social transformation efforts resist being reduced to economical equations that could be measured through an input-output paradigm. The less visible part of outcomes in social change work, the temporality in results showing up in ways that can be observed, traced and captured, goes missing (see also Kaplan and Davidoff, 2011). Yet it is precisely this capacity to observe and trace that needs nurturing and in monitoring and evaluating social transformation, and hence also transitional justice initiatives (see also Duggan 2010a, pp. 319-327; Duggan 2010b).

1.        Monitoring and Evaluation in Transitional Justice Interventions

In many of the scenarios described by Transitional Justice organizations in the African Transitional Justice Research Network (ATJRN), practitioners currently experience an over-emphasis by funders on accountability in the technical sense: reporting against preconceived plans and templates, often in terms of numbers of workshops conducted, brochures produced, products and trainings delivered, etc. A major shift was the move from this merely output-driven dimension (what was done or produced) to an outcomes orientation (what did beneficiaries do with the experiences, products and learnings). The outcomes orientation, however, remains often tied to preconceived indicators that are rarely reflected upon and revised continuously in the process of project/programme emergence. Rather, in response to current M&E frameworks, actors ‘tailor’ their thoughts on what has actually transpired during the activities to fit the indicators. The loss is not only valuable information that is not deemed relevant, not asked for and, hence, not reported, but also the potential for learning from what has actually emerged during the process of implementation. In response, TJ practice becomes activity-driven, so as to produce countable results.

This means that the supposed ‘second leg’ of M&E – if accountability is one and learning the other – is not fostered in the current grantmaking and related M&E practices. TJ practice becomes activity-driven, so as to produce countable results. The critical dimension of monitoring as learning, monitoring as a constant reflective reading of the situation, is not evolved as a conscious practice. There is not yet an M&E approach that is specifically tailored to the needs of practitioners working in the sensitive arena of dealing with the past after and often within the continuities of violent conflict.

TJ practitioners engaging with the Learning, Monitoring and Evaluation in Transitional Justice work commented at the Institute for African Transitional Justice on a lack of learning spaces inside their organizations, where practice is often activity- and context-response-driven and characterised by a near-furious busy-ness, particularly in activist-minded organisations and coalitions. M&E is thus often dreaded as an ‘external eye’, with many of the experiences sketched representing the parachute-in-a-consultant-at-the-end-of-the-project-approach.

Summative evaluations are favoured over formative approaches that accompany a project process over time. External consultants then conduct arms-length, often rushed appraisals according to the indicators and needs of the funder. Terms of Reference are neither jointly defined nor discussed in-depth with the practitioners as the process of implementation unfolds. More rare (but existing too) are reports of evaluations where terms of reference are negotiated, where time is taken for pre-conversations, where reflective practice, engaged in with some insiders and some outsiders to the project process generates an evaluative exchange dimension that is experienced as meaningful and productive by all. The later examples left practitioners with a sense of satisfaction, based on the feeling of respect, being taken seriously and able to share processes, results, learning and expertise with the review teams. In particular, the peer reviews that were conducted by the African Transitional Justice Network were experienced as productive. The ATJRN developed a peer review methodology and has run peer reviews between TJ organizations in Uganda-Sierra Leone, South Africa-Kenya and Kenya-Zimbabwe (Mncwabe 2010).

The least developed (or perhaps conscious) part of M&E practice in TJ organisations, however, seems to be the dimension of ongoing Monitoring. This is an area where M&E, in-house knowledge management and organization development practices meet, at best, in an organizational culture that encourages reflective practice, learning, revealing mistakes and learning from failures. Monitoring is much more than the technical exercise of ‘counting’ deliverables that it is sometimes made out to be. It is intimately linked with questions of organizational management and culture as sources for innovation and development. Many organisations struggle with developing systems that make sense for their specific organisational context, staff, origin and orientation of the organization and its work because they feel they have to adhere to M&E conventions and standards, even if they do not suit them. The impulse to institutionalise an M&E system often comes from the outside. It is not driven by an internal intent to learn and improve one’s work. Hence, M&E is seen as a ‘necessary evil’, a donor requirement to release funds rather than a means to generate learning and improve one’s work. This reflects pragmatism in that donors might indeed refuse funding if their requirements for a stringent M&E are not met. Yet often, a conversation about TJ organisation’s needs and finding more constructive ways of M&E-ing is not even attempted though it could serve both sides and enhance the learning dimension of the work. Instead, M&E processes are resented. Reports are written grudgingly, producing flat boring narratives that are then not read and result in little response beyond ‘received with thanks’. The end result is an emptying of energy and meaning in the exchange between grantmakers and grantees.

At the same time the current drive to develop detailed ‘frames’ for M&E is also being recognized as holding the potential and granting the opportunity to evolve more ongoing reflective practices in TJ-oriented organizations. Since there is funding available for M&E systems development, these could also be used to develop and institutionalise learning from experience in ways that suit practitioners and further the evolution of organizations and contexts. Practitioners in transitional justice work thus face several critical challenges with regard to M&E work and the trends sketched above.

2.        Showing Impact – Key Challenges

Attempting to make sense of these challenges in monitoring and evaluation, three key polarities emerged that TJ actors move between:

1. Linear – Nonlinear: linear images of envisioned outcomes are confronted with the non-linear nature of social transformation processes unfolding,
2. Visibility – Invisibility: the attempt to create visibility (of victims concerns, of truths about the past, of perpetrators and their crimes) and needing to operate in the invisible, at times even facilitating concealment, and
3. Ownership – Letting others Claim Results: the tension between claiming ownership of a result and holding silent joy in knowing one had impact but has to let others claim the limelight and credit for it (i.e. policy-influence, advisors of political entities, etc).

2.1.     Moving between high visibility and operating in the invisible

Due to the sensitive nature of processes of dealing with the past, some of the TJ organisations’ practice takes place behind-the-scenes or in the realm of the invisible. Advocacy strategies for TJ processes are usually two (or more) pronged: partly geared at taking a confrontative, public ‘naming and shaming’ approach, partly focused on more associative, dialogue-oriented, ‘winning-over’-key-actors approaches that takes place outside of the public eye. While the former can be documented, traced and ‘exposed’, for the later strategy it is more difficult to ‘prove impact’, for in revealing the same impact is lost. An example here is the process of advising policymakers who cannot be seen as ‘unknowing’ in public but who appreciate quality information, reflection and engagement, and smaller-scale learning processes in trusted environments. These can be offered by civil society actors who manage to build trusting relationships and keep the engagement highly confidential.

Thus, effective TJ work rests on a dance between revelation and concealment, that limits the possibilities to claim certain ‘successes’ publicly, even in a donor report, while sensitive processes are still ongoing. Some of these success stories might be told only years later and have to be kept confidential at the time so as to not jeopardize their impact in the moment. As for the failures, which usually yield the richest learning experience if explored, there seems little scope to admit openly one’s sense of having failed in a particular TJ-related intervention. Still, in trusted conversations among peers, such failures are explored, pondered and learned from. The results of such, often informal, conversations are mostly what evolves into future strategies of handling similar challenges or inspire a changing of course in an intervention. The best ideas are reported to be birthed in the informal sphere of peer exchange (see also Mncwabe 2010).

2.2.     Taking Ownership – Letting others Claim Results

The sensitive nature of the engagement in TJ process work described above, between civil society actors, policy-makers and implementers of TJ, means that if advice is given or any critical input is taken on board by decision-makers and if learning does take place in the interaction with civics working on TJ, this cannot be claimed directly by the practitioners. Then ownership would no longer rest (publicly at least) with those who were quietly advised and who might feel their trust betrayed. Concealment and strategic revelation are then both strategy and source of success in the TJ field, as is the case in many conflict-related interventions. In addition, seeing an immediate effect does not yet reveal impact of an intervention over time. Most times impact reported was achieved by a collaborative effort of a range of actors and through the interactions of several projects, individuals, groups and institutions over time. Money, flowing from various sources, enabled interaction of varied activities and actors in an inter-connected manner that cannot easily be traced back to a single intervention or organization in a TJ process.

Hence, it is at times difficult to prove later which process in this entangled and interconnected web of actions and interdependencies has produced a particular visible effect that can then be claimed as a result. On occasion, positive results and stories then tend to be claimed, told and ‘sold’ as success by several actors. There are as yet few efforts to read projects, programmes and interdependent results in a context together in a systemic way (see Berghof Dialogue Series 2008). Instead the effort is often to take individual strands that fit a project proposal and a set of indicators that are often more than two years old by the time this is done. The struggle is then to try squeeze out a report against these criteria that somehow fits and can be submitted.

Most often the report is never engaged with beyond the ‘accountability’ dimension of the work (was it done, did it have effect, was it visible). The learning dimension which could possibly be part of ongoing reflections prior to report writing, or even ongoing writing out of practice, is most often not followed up on due to a perceived lack of time and the pervasive sense of being overwhelmed among practitioners in implementing organisations as well as desk officers in donor agencies. This general overwhelmed-ness results from the sheer amount of activities implemented and an inherent assumption that ‘more is better’ that is often not queried. In addition, the mass of information dealt with on a daily basis leads to a general sense of being saturated and ‘overfull’ most of the time with meetings, engagements, interactions and data consumption. Small corridor and lunchtime conversations begin to appear as a relief from the bustle. This is where much of the ‘real’ learning seems to be taking place – in those snippets of debriefing standing in the doorway of a colleague’s office, in the aftermath of an intense workshop venue where some are left behind packing up and get talking, in the car after a policy meeting while driving back, in the conversations at the office in the afterglow of an inspiring event and when spontaneously reflecting on a news headline or a significant contextual happening that requires new strategic thinking. These in-between spaces are the very spaces where practitioners are momentarily not driven by the pressures of performance, formality and hierarchy in their organizations. Such spaces produce moments of freedom from anxiety and expectation of what ‘must be done’ and an inner movement to possibility and exploration of what could be brought to life in the future. Such moments were described by most practitioners as one source of inspiration that drives them, that keeps them going with their work. They describe a sudden energy rush and connection to self in such moments when ‘I know why I am doing what I do. It suddenly makes sense again and I know, we are having an impact. I can see it even if others can’t or if I cannot talk about it.’

2.3.     Non-linearity as a Gift

Another character trait that TJ work shares more broadly with peace and development work is the unpredictable, non-linear nature of how processes unfold, often characterised by advances and setbacks, detours and surprises. Often, practitioners reflect, the best ideas come out of crises and setbacks as supposed failures are processed and reflected upon. What seemed like an unnecessary and burdensome obstacle, when a process does not go ‘to plan’ turns out to alert practitioners to an important dimension that has been overlooked or a group of actors that needs inclusion. How do TJ practitioners deal with the nonlinearity of the work of social transformation? How do they engage with social change processes which require stepping out of the binaries of right-or- wrong, success-or-failure thinking?

One valued insight that emerged from conversations with TJ practitioners about Monitoring and Evaluation was that, in fact, all are, consciously or unconsciously, monitoring and evaluating their work all the time. Every practitioner is constantly pondering what she does, wondering if it makes sense, if it is having impact, if there is a need to shift course, to do something differently? This is almost like an inherent navigation system that we steer ourselves by, an impact navigator that traces and ‘reads’ the situation, one’s own intervention, the responses of others, one’s own internal resonances to their response, and this happens all the time without a break. There is an individual intuitive and natural sense-making process that takes place. This covers all eventualities and situations and is particularly alert when things unfold in ways different to how we imagined they would turn out (which is most often the case in social transformation work).

This internal navigator serves an inherent M&E function; it evaluates continuously how we impact on the world and how it impacts back on us. One could see this as an internal observational eye that takes a multifold reading of the situation and absorbs detail meticulously (a particular policymaker’s reaction in a meeting, the response of a group participant in a workshop, the vibe in the crowd at a village gathering). This internal eye traces the process dimension of the work and at the same time is also able to see larger Wholes in the situation emerging (see also Kaplan 2002, pp. 3-14). Among groups of peers and colleagues, such observations and readings are then put to the test – how does what I saw resonate (or not) with another colleague’s observations? How do we make sense of what happened collectively? As the informal conversations described above ensue, such joint interpretation processes prove invaluable, not only to read a situation in the moment but to make meaning of it and arrive at a shared understanding of the situation. It is this sense of a shared understanding emerging that confirms to the internal navigator a sense of having ‘read’ the moment, process or situation accurately and comprehensively.

Such ongoing internal assessment skill serves practitioners in the TJ field well who are often facing fast-paced and unpredictable changes in how their work, their contexts and even their own lives as activists unfold. M&E, seen in this way, is therefore an already existing skill and faculty of ‘reading emergence in a social situation’ that can be built upon (ibid.). This means that an essential asset to a strategic and successful practitioner most often is already well-developed; it is just not consciously seen as a strategic capacity for M&E. This links directly to the often under-developed capacity in organizations for reflective writing as an ongoing practice of documentation that can serve good M&E, knowledge management, organizational development and the personal and professional evolution of practitioners.

3. Conclusion: opening M&E up for questioning 

Tim Murithi suggested at the Institute for African Transitional Justice in Kampala in 2010 that one of the critical strategies to develop the TJ field in Africa is for young and coming practitioners to ‘witness and write’. My experience in building M&E practices that serves African TJ practitioners and that enhance impact resonates strongly with this (re)quest for witnessing and writing. The questions that arise for M&E practice in TJ are:

• How can confidence and skill be built further to use this inherent knowing of tracing emergence of social transformation processes be used for learning and improving one’s work?
• How does one achieve this skill and confidence and ensure that it is witnessed and written without it being “managed” by a technical take on M&E alone?
• Rather than transferring technical skills to implement a pre-conceived template-based M&E, how could practitioners make their reflective practices conscious and visible?
• How could they use their inherent skill of observation as a ‘reading’ of situations as they develop in order to sharpen their perception skills, in a sense develop their intuition further into visibility?
• How could the learning dimension of M&E be strengthened so as to build reflective practice more consciously as a key M&E skill?
• What enables practitioners to build their writing practice so that the inherent stream-of-consciousness M&E becomes a conscious reflection process on the page?
• In the moment practitioners know when they are having an impact, desired or undesired. How can confidence and skill be built further to use this inherent knowing for learning and improving one’s work?

The current realities of many activist-practitioners in the TJ field are characterized by being frantically busy, a mode of working that seems to be high-performance at face value, but that is described as draining and distracting from achieving meaning and substance through the work. At the same time there is frustration around practitioners wanting to write a define their experiential realities, rather than producing data that is used for ‘measuring’ only or that serves researchers and scholars (who often have more distance and writing practice) to ‘mine’ the experiences as raw data. One element of innovating further around M&E currently includes a group of TJ practitioners observing themselves and their actions consciously while asking:

When do our best ideas emerge?

What feeds our energy and sense of purpose?

How can we strengthen our resourcefulness through M&E rather than just looking for gaps and weaknesses in the work?

References

Duggan, Colleen. 2010a. Editorial note. The International Journal of Transitional Justice. 4:315-328.
Duggan,Colleen. 2010b. ‘‘Show me your impact’’: Evaluating transitional justice in contested spaces. Evaluation and Program Planning. doi:10.1016/j.evalprogplan.2010.11.001.
Kaplan, Allan; Davidoff, Sue. 2011. The Singer not the Song. The Vexed Questions of Impact Monitoring and Social Change. Paper by the Proteus Initiative (www.proteusinitiative.org).  May. Accessed 12 August 2011.
Kaplan, Allan. 2002. Development and Social Process. Artists of the Invisible. London, Sterling, Virginia: Pluto Press.
Koerppen, Daniela; Schmelzle, Beatrix; Wils, Oliver. 2008. A Systemic Approach to Conflict Transformation. Exploring Strengths and Limitations. Berghof Handbook Dialogue Series. 6.
Mncwabe, Nokukhanya. 2010. African Transitional Justice Research Network: Critical Reflections on a Peer Learning Process. The International Journal of Transitional Justice. 4:497-508.
Quinn Patton, Michael. 2011. Developmental Evaluation. Applying Complexity Concepts to Enhance Innovation and Use. New York, London: The Guildford Press.
Westley, Frances; Zimmerman, Brenda; Quinn Patton, Michael. 2007. Getting to Maybe. How the World is Changed. Toronto: Vintage Canada.
Wheatley, Margaret; Frieze, Deborah. 2006. Using Emergence to take Social Innovations to Scale. https://margaretwheatley.com/articles/emergence.html Accessed 8 August 2011.

[pdf-embedder url=”https://undine.co.za/wp-content/uploads/2019/12/Intercultural-Transcultural-Transhuman_Connectivity_HailuWhande.pdf” title=”Intercultural Transcultural Transhuman_Connectivity_HailuWhande”]