How can Governments benefit from visual thinking in evaluation? Takeaways from India

Governments can strategically introduce visual thinking as part of monitoring and evaluation to support information management, enhance understanding, ensure retention, ensure effective stakeholder engagement, and most importantly, use evaluative evidence. In this evidence story we share how EvaluVision, a visual thinking technique, increased evaluation use during a workshop hosted by the World Food Programme and the Development Monitoring and Evaluation Office (DMEO) in India.

By Yumiko Kanemitsu, WFP and Alok Mishra, DMEO, NITI Aayog

Monitoring and evaluation as a tool to support effective decision-making continues to gain national priority. Governments around the world face the pressure of delivering more and better services under limited resources due to tightening fiscal environments. India, facing a similar challenge, has turned towards increased monitoring and evaluation (M&E) of its programmes and policies to enhance the effectiveness of public expenditure.

The year 2016 saw a rejuvenation of M&E in India, with an increased focus on yearly reporting of outputs and outcomes for each programme, and the continuation of a programme made contingent upon independent evaluation findings. As a result, with each passing year, India produces huge amount of data and information through its M&E systems. But the question remains: how much information are the stakeholders actually digesting? And what role can visual-thinking and facilitation play in transforming evidence generated through M&E into a learning tool?

 

Michael Quinn Patton in his book “Facilitating Evaluations” observes that there is a high price for poor facilitation and communication of M&E insights such as ignored participant needs, submerged stakeholder voices, and hiding underlying causes that may finally undermine impact of a programme.

Effective facilitation and communication increase the relevance, meaningfulness, credibility, and utility of evaluation insights.

EvaluVision an innovative approach to visual thinking and group facilitation

EvaluVision, a combination of Evaluation and Visualization, aims to improve the utility of evaluation insights by organizing technical information into comprehensible graphics. It encourages engagement across all stakeholders and has been designed and tested at workshops in Asia and the Pacific Region.

A typical public programme generates huge volumes of data in a year. Insights are presented as narrative reports which are then summarized based on the needs of the audience.

At the centre of this process in India is the Development Monitoring and Evaluation Office (DMEO), NITI Aayog, which is the nodal M&E institution. Together with DMEO’s technical support, the central ministries and departments in India generate a large volume of evidence periodically – annual monitoring of key indicators for programmes every year (160 programmes in 2020-21 and 139 in 2021-22), followed by an independent evaluation every two-to-three years.

As a result, non-performing programmes are either discontinued, merged with others, or revised in consultation with key stakeholders. Use of visual-thinking tools and techniques can benefit the entire national M&E value chain, from planning and interpretation of results to its dissemination and uptake.

Planning is at the core of all M&E exercises and involves multiple stakeholders with varying capacity of M&E.

Visual thinking techniques, such as EvaluVision, combine simple yet compelling graphics and group facilitation exercises to help articulate the perspectives of each stakeholder more effectively. An immediate benefit of using EvaluVision is the development of comprehensive theory of change(s) built consultatively with all stakeholders/ministries, clearly describing the interlinkages, underlying assumptions, and timeline of change for a programme.

For instance, India implements one of the world’s largest public food security and nutrition schemes involving multiple ministries in charge of production, supply, and final distribution of food commodities to the most vulnerable population. Using today’s visual-thinking tools, a theory-of-change linking the food safety net with the final goal of Zero Hunger by 2030 can emerge, that describes the pathway of change envisioned by each implementing ministry. As a result, a ministry can ramp-up its M&E activities to generate critical evidence on challenging issues that may have gone undetected and clearly define their timeline of change and efforts required from a medium to long-term perspective.

As ministries move from solving simple and complicated to complex problems, the verticalized intervention or linear thinking would deliver limited impact and therefore a system-based understanding becomes more important. EvaluVision is best suited to facilitate better understanding of complex problems along with inter-relations between various parts for systemic approaches. Moreover, the complex problems based on systemic approaches require a flatter and matrix organization to address them, something which can be explained better through visual thinking, facilitation, and communication techniques.

Interpretation of results is currently a fragmented process, with each stakeholder zooming-in on parts of evidence rather than the larger picture.

For evaluators, the need of the hour is to include visual thinking techniques while preparing a data analysis plan. In usual practice, data analysis is given more emphasis than dissemination. While narrative reports and charts are comprehensive, in practice stakeholders have limited time to digest this voluminous information and arrive at a larger picture of the sector.

This leads to prioritization of some findings above others while missing out on the larger context of the scheme/programme. The space for engaging all stakeholders also gets constrained, and some stakeholders may feel left out. A “diagnostic map” can be developed using EvaluVision tools, that summarizes the performance of contributing activities and their key successes and bottlenecks. This map can act as a catalyst in reducing the verticalized approach of service delivery, increasing the emphasis on system-level understanding and helping to build consensus on the larger situation.

Insights from visual thinking products “stick” longer compared to narrative reports and stakeholders develop a consensus on what needs to be done going forward.

There are a number of reasons why visual communication of M&E findings is especially important in an administrative setup like India:

  1. The senior government officials have short tenures and therefore limited time before transitioning into another ministry/department
  2. The findings from M&E have to percolate to the lowest rung in the government, where the understanding of M&E concepts may be limited
  3. In a country as diverse as India, visual products transcend linguistic, geographical, and cultural barriers
  4. Visual products create better engagement with audience especially for a dry subject like M&E

Products from EvaluVision can help retain institutional knowledge around the larger picture in a constantly changing senior leadership. In practice, the “diagnostic maps” will be very useful for new officers joining a ministry/department to get a quick self-explanatory overview of their implementation landscape and challenges that need tackling.

How can technology facilitate a smooth transition to visual thinking in evaluation?

Today, users can choose from a wide range of collaboration tools and facilitation techniques to help implement visual thinking in evaluations, such as Mural, Miro, Google Jamboard, and Stormboard. WFP has developed a comprehensive guidance e-book on EvaluVision, which is a step-by-step introduction to implementing these techniques. This e-book comes with short introduction video.

What are the possible challenges in using EvaluVision in a government M&E setup?

To answer this, the World Food Programme (WFP) held a virtual workshop with the DMEO, NITI Aayog. Objectives of the workshop were to introduce the concepts of EvaluVision to DMEO, as well as understand the challenges and opportunities to implement EvaluVision in government settings.

 

The “EvaluVision Triangle” was used to visualize the preferred future where visual-thinking, facilitation and communication is used in the M&E process to improve quality and use of insights, and the factors that prevent in implementing such a methodology. Participants were asked to write their thoughts in three separate boxes titled – pull from the future, push from the present, and weight of the past. Mural platform was used for this exercise.

The pull from the future include better interpretation and utilization of M&E results, bridging the gap between evaluators and programme implementers, and the potential to co-create effective programmes with various stakeholders. The push from the present comprise of the need for quicker results by decision makers, need for innovative techniques to engage various stakeholders, and availability of new and online visual-thinking and facilitation tools and platforms.

The critical challenge or weight of the past that may slow down the use of visual thinking techniques is the fast-paced nature of policy cycle in India that leaves little time and space for innovative approaches. There is a gap in capacities among evaluators in terms of using visual-thinking techniques (“fear of getting it wrong”), and less space for experimenting with new dissemination tools due to the tight timelines. Stakeholder’s exposure to visual-thinking and group facilitation also remains limited.

Next steps

There are three opportunities available for institutionalizing EvaluVision approaches within the government M&E system in India. First, as the conversation around use of visual-thinking tools and techniques gain prominence in government evaluations, the next challenge is to start embedding it into the evaluation cycle. Present opportunities include the joint studies planned by WFP and DMEO on food security and nutrition issues in India. This can then be expanded to other evaluation studies, which are proposed to be undertaken by the DMEO.

Second, the ongoing capacity building initiatives for officials at DMEO and other government stakeholders along with introduction of short-term and long-term courses in Universities. In both these cases, visual thinking will have to be embedded in the content, format, and communication aspects of M&E capacity building. The EvaluVision techniques will have to be taught as part of the M&E training or course. Moreover, Massive Open Online Courses (MOOCs) on M&E for explaining the theory and concept can be created using EvaluVision tools.

Third, in a federal set-up like India, where states have their independent M&E systems, an effort will have to be made to reach across to states like Karnataka and Tamil Nadu with more matured M&E systems, to start introducing EvaluVision concepts.

Visual thinking and visual communication were innate to the human civilization before the language was created. Hence, it is only a matter of time and a consistent effort that visual thinking and communication will become a reality in evaluation.

 

Leave a Reply

Your email address will not be published. Required fields are marked *