Tackling the Impact Measurement Challenge
Tackling the Impact Measurement Challenge
By Patrick Regan, 22nd July 2020
In early June 2020, a number of DFF grantees came together virtually for DFF’s first ever “outcomes harvesting” workshop.
This marked the next step in DFF’s plans to pilot a new framework for monitoring the impact of strategic litigation to advance digital rights. But what is “outcomes harvesting”? What is this new framework? Why is it important?
Measuring the impact of strategic litigation is no easy feat.
Litigation can take years, the context is often complex, and there are many factors outside our control which could influence the effectiveness of the litigation. Often, we have only limited (if any) access to key decision makers involved to gain insight into how law, policy or practice has actually changed.
This makes it challenging not only to identify impact, but also to assess your contribution when there is impact. This is compounded by the lack of readily available evaluation tools and methodologies tailored to strategic litigation.
Many NGOs and litigators do not have the time or resources to reflect, evaluate and collect robust evidence to learn from their advocacy activities. However, measuring and demonstrating the impact of litigation is imperative for the digital rights field to work more efficiently and effectively, and attract support for this type of work. So how can we overcome these challenges?
I would be embarrassed to admit the number of times I lay awake pondering the different ways to solve this problem. Thankfully, my pondering was not in vain. In early 2019, I found out that DFF had noticed the same issues related to measuring the impact in strategic litigation, and decided to try and do something about it. This served as a jumping off point to develop an exciting (if you are an evaluation nerd like me) new framework with DFF to help measure the impact of strategic litigation in digital rights.
Developing a Pilot Framework
DFF wanted to develop something that was specific to strategic litigation in the digital rights field, but which could be easily implemented and adapted for different organisations. It was imperative to strike the right balance between something that had enough rigour to be reliable, but also light enough that it would not demand too many (already scarce) resources to use.
Building on my own previous experiences of adapting tools for strategic litigation impact assessment, reviewing available literature and case studies, and talking to other organisations in the field to check relevance and draw inspiration, we developed a framework based on three key components:
- A thematic framework of outcome themes and impact types to help guide the process of monitoring, identifying and analysing outcomes.
- A methodology for capturing “outcome statements” (which takes inspiration from a methodology called “outcomes harvesting”).
- A set of evidence principles to add a lawyer of rigour and quality control.
The framework was published in December 2019 under a creative commons license so it can be easily shared and adapted.
Putting the Framework into Action
On 10 and 15 June 2020, eight DFF grantees came together for a workshop to learn about the framework, marking the next step in DFF’s plans to pilot the framework and put it into practice.
During the first part of the workshop, we discussed some common challenges in measuring impact in strategic litigation and digital rights. This was followed by a short introductory training on the outcome harvesting methodology.
The group raised some important challenges and questions. How can you measure and understand how different groups and communities are impacted by the litigation? What impact does the litigation have when the state in question does not implement a judgment? How can you attribute your work to higher level impacts when so many others are involved? How can we communicate the impact of strategic litigation to a wider audience when both the process and the issues litigated are complex and technical?
How can we communicate the impact of strategic litigation to a wider audience when both the process and the issues litigated are complex and technical?
The framework seeks to respond to some of these challenges. It provides a tool to help document and identify key outcomes and changes, and helps to identify your contribution to these changes. It also gives a structured way to capture learnings, unexpected outcomes and to seek out negative outcomes so we can better mitigate against them in the future.
It’s Not All About the Judgment
The direct legal outcomes of strategic litigation are often significant and important. For example, obtaining a positive judgment that establishes precedent and new case law, or redress for the claimants.
The framework also encourages you to think about the other ways in which litigation may be generating positive results and changes
However, the framework we developed also encourages you to think about the other ways in which litigation may be generating positive results and changes, at all stages of the litigation process. This could be by increasing public awareness or changing public perception on an issue; changing the way a certain topic is presented in the media; or even the impact that the act of taking the case might have on your own organisation, network or the wider field.
“Sometimes we can focus too much on the bigger, longer term aims” commented one participant. This was an important point of discussion during the workshop. Another workshop participant commented on the approach that “it’s a good way to start thinking outside of the box in terms of impacts, to think about all of the things and areas your litigation might effect”.
The framework encourages anyone using it to consider these different layers of outcomes and to appreciate their value and contribution to other, broader outcomes and impacts.
Evidence Standards
During the workshop we also discussed data and evidence quality, considering what was both realistic to collect and credible enough to be used in measuring outcomes. The group discussed some key questions to help think through evidence quality, such as:
- Is the outcome you identified, and your contribution towards it, proportional to your evidence? (i.e. is your contribution claim realistic based on the evidence you have?)
- Is the contribution you are claiming proportional to the size, scale and timing of your litigation/advocacy?
- Has the evidence been peer reviewed or independently verified by anyone else?
- Are the voices of those we claim to have had an impact on represented in the evidence?
The pilot framework has been designed so that organisations can use the data they might already capture, their existing monitoring or evaluation systems, and also anecdotal information that they have access to. It proposes a set of evidence principles to help assess the reliability of the evidence and outcomes you have identified.
Putting the Framework to the Test
After the first workshop, the group then put the methodology into practice by identifying and preparing a number of outcomes concerning their litigation projects. During the second part of the workshop, the group shared some of their outcomes as well as their reflections on using the methodology. We also shared and reflected on some of the challenges, successes and learnings when engaging in digital rights litigation.
The group “harvested” over 30 different outcomes which ranged from establishing important and novel legal precedents and influencing the way in which certain topics are portrayed in the media, to more organisational and field-related outcomes, such as prompting others to engage in legal action or developing and applying knowledge gained through their pre-litigation research to improve the quality and reach of other advocacy campaigns.
They thought about what “impact” would actually look like in reality and what changes they might be able to observe
Some organisations who were at an earlier stage of their litigation used the method to help identify and articulate their desired outcomes. They thought about what “impact” would actually look like in reality and what changes they might be able to observe. This helped them to plan what kind of evidence or data they would need to collect to know if this change has happened.
The framework and methodology is of course only one potential solution. It makes use of a number of different research and evaluation principles that can be used and adapted flexibly in a variety of situations and contexts. As one participant commented, “it seems like a useful way to understand and develop an idea of what your role is in a bigger change, and what part the litigation played”.
DFF will continue to pilot the framework through its grantmaking and by holding more outcomes workshops. If you would like to know more about the framework, or attend a future DFF outcomes workshop, please email DFF’s Programme Officer.
Patrick Regan is an independent evaluation consultant specialising in the evaluation of projects concerning human rights and which use litigation/the law to drive social change. His consultancy, the Rights Evaluation Studio, provides a range of evaluation and project design services.