The external evaluation of DFF’s pilot phase 2018-2020 shows its relevance and effectiveness while identifying ways to further enhance its functionality to the digital rights community.
The Digital Freedom Fund (DFF) has existed since 2017, with its first 3-year strategy going into effect in 2018. During this period, DFF has attracted funding, built a team and organisation, has put in place systems to process grant requests, some of which have already led to high-profile rulings, and has organised a variety of meetings with stakeholders in the field of digital human rights. As its strategy 2018-2020 is approaching its end, DFF – supported by its seed funders – commissioned an independent external evaluation of what has been referred to as DFF‘s “pilot phase”.
The evaluation was conducted by a team of senior consultants from the non-profit consultancy ODS, between February and June 2020 through a combination of desk research, meetings with over 50 stakeholders including DFF’s leadership, grantees, meeting participants and external experts, and round tables with the DFF team and its donors.
The evaluation was conducted by a team of senior consultants from the non-profit consultancy ODS
The evaluation focused on whether DFF’s Theory of Change (ToC) was (still) valid and relevant to the digital rights community. In order to assess that, the evaluators tested the theory and reviewed how far DFF has progressed towards the impact it aims to achieve, and tracked how effective and efficient DFF has been in putting this theory into practice. This resulted in an analysis of the current situation as well as a set of strategic and operational recommendations for DFF’s next phase and strategic period.
In its ToC, DFF has as its overarching objective to “further human rights in digital and networked spaces by increasing the number of successful strategic litigation cases and by supporting the contribution of such cases to wider public debates, policies and practices”. In addition, the ToC holds that a condition for success in achieving that goal is that actors involved in strategic litigation should be sustainable and stronger.
This leads DFF to work towards these goals through two broad workstreams: litigation support, consisting of providing grants to strategic litigation projects and the promotion of pro-bono support, and field building, which involves capacity building, networking, knowledge sharing and strategy development.
The evaluation looked at DFF’s positioning in the field of digital rights, the consistency and balance within its ToC, and the strength of the ToC. The assessment of the evaluators is that DFF is very strong in all these areas.
First, the combination of grantmaking and fieldbuilding with strategic litigation is unique in the field of digital rights and DFF is seen as an important added value to the field. This can be attributed to the strength of the ToC but also to DFF’s practice of continuous horizon scanning and consultation with a diverse group of actors, which allows it to adapt to changing circumstances.
…the combination of grantmaking and fieldbuilding with strategic litigation is unique in the field of digital rights
Secondly, the connection between DFF‘s interventions and goals is convincing and strong. Strategic litigation is important for furthering digital rights and was underutilised in the digital rights field, partly due to a lack of capacity and partly due to a lack of resources. DFF has worked to remedy these limitations by offering actors in the field who are interested in or embarking on strategic litigation access to funds, advice and support in the course of the grant applications, as well as opportunities to learn and network in strategic, thematic and capacity building meetings. Finally, the activities and outcomes in the ToC are also credible and sufficient to achieve the outcomes DFF works towards.
DFF has also been highly successful if measured against its own quantitative goals of providing at least 20 grants, promoting pro-bono support towards at least 75% of applicants and a satisfaction of at least 75% of participants in field building activities. While more difficult to measure, the evaluation also found that the more qualitative metrics around increases in the quality of applications, more collaboration on these applications and more strategic alignment of actors in the digital rights field, have been met as well.
These metrics themselves however, are not sufficient to understand how far DFF has progressed towards its outcomes in the ToC or towards actual societal impact. This is not a problem for now, as two to three years is too short to measure impact in most cases, and this evaluation focused on the conditions for success and progress towards impact. A next strategy, however, would need to bring the metrics in line with the ToC, which is something DFF is in fact working on together with a Monitoring & Evaluation expert.
In providing strategic litigation support, and in the grant application process in particular, DFF has encountered some pushback from the field which was confirmed during the interviews. While all interviewees appreciated the professionalism of and support from DFF’s team and the clarity of its communications, they also indicated that the application process was too intensive and came at too high a cost in relation to the size of the eventual grant. To understand this view, the evaluators traced DFF’s grantmaking process from:
- communication of the aims and conditions of DFF’s support,
- the application requirements
- application assessment
- the conditions of the grant itself.
In doing so, it became clear that while minor improvements could be made to the way DFF communicates and to the application requirements and forms, these were in fact already quite streamlined, clear and, in the case of the application, reasonable in length and detail.
The way DFF assessed the applications and the limited flexibility in the size and conditions of the eventual grant, did lead to some applicants questioning whether they would apply for a grant again in the future. The application of DFF’s own grantmaking guidelines was reviewed at two levels.
One area for improvement would be to reduce the many iterations through which DFF assesses applications, with the team, a Panel of Experts, and the Board all offering their views, sometimes in multiple rounds. The other is more substantive in that the existing rules are applied quite strictly and in the same way for all applicants regardless of the type or organisation or case. This is understandable in a pilot phase as DFF needed to put in place mechanisms to ensure accountability towards its donors, and thus does not lead to a negative assessment of that pilot phase. The evaluators do think that going forward a more tailored approach would be beneficial.
One area for improvement would be to reduce the many iterations through which DFF assesses applications
On the conditions of the grant itself, the challenge has been that during the pilot phase, DFF was unable to commit to the type of longer-term grant that a case spanning multiple instances would require. In addition, a number of costs were excluded (adverse costs in particular) or limited (advocacy costs, legal fees and operational costs).
In addition, the possibility to combine more than one type of grant – single instance, research, emergency – was not widely known with grantees. Increasing the length and scope of the grant would change the cost-benefit analysis of the application and DFF is already developing procedures to do this. It is planning to add the possibility to include adverse costs under certain conditions, and will start offering grants over multiple instances (known as ‘track support’). More flexibility in the costs covered and the application process would in the view of the evaluators also offer DFF and the applicants additional leeway in making strategic decisions on timing, collaboration and jurisdiction for cases they are considering.
….the possibility to combine more than one type of grant – single instance, research, emergency – was not widely known with grantees
In its coordination and field building work, DFF has exceeded the outcomes that were expected and could reasonably have been expected, especially through the meetings it has organised. DFF’s practice of consultation on its strategies; consultation on the agenda of every meeting; its active push to broaden and deepen the group of participants in meetings; the open way in which the meetings are organised and facilitated; and the issues that are being discussed; all contributed to the evaluators’ assessment that DFF has had a positive impact on the field. This stems from an overall active strategy to broaden the field geographically, thematically and in terms of the people and groups who are given a voice in the field of digital human rights. The inclusive principles behind this approach are now being further developed through a decolonising strategy which has the potential to solidify and expand DFF’s impact in this regard.
Some elements could be further strengthened. As the demands on the field increase and DFF’s network grows, strategic meetings may need to be broken up into themes and the annual broad strategic meeting may need to become biannual, something DFF is already considering. The meeting format itself as well as the facilitation could – while generally appreciated – benefit from more diversification to better fit all participants, the specific themes discussed or the purpose of a meeting.
The inclusive principles behind this approach are now being further developed through a decolonising strategy
Regarding the use of pro bono support, this has not been taken up by the field even though DFF has promoted it consistently. This means that DFF could reconsider what it offers in order to facilitate pro or low bono support, as well as how it communicates about it towards those grantees who might benefit from it.
Finally, DFF could leverage its knowledge of the field and the issues more, to bring together different actors proactively, to support each other on issues outside of DFF’s mission, to work on themes or applications or represent the field in certain fora, or to find additional resources. DFF‘s position in between donors, digital rights community, legal experts and policy makers offers it a unique opportunity to add additional value by being more intentional in offering insights or suggestions to the field. While DFF should remain at somewhat of a distance so as not to take up the space of its grantees or partners, its role as facilitator could also be seen as a way to place partners more front and center, by asking them to develop themes or represent the field in certain fora.
Overall, the conclusion of this evaluation is unequivocally that DFF’s ToC is relevant, that the goals it set at the beginning of the pilot phase have been met, and that it has done so in an efficient way.
With that, the progress towards impact is impressive and impact is likely in the near future, while its impact on the field through its approach to field building is already noticeable. The impact of grants on specific cases is clear but the longer-term impact of these cases on policies, attitudes and case law are not yet known.
Going forward, DFF does need to evolve to remain as relevant as it is and to ensure that the field continues to be interested in applying for grants. This evolution should be geared towards remaining attractive as a grantmaker and becoming more intentional and dynamic by increasing the options it has in its grantmaking work (both in the application process and the grant itself), diversifying its meetings, and facilitating connections between actors more proactively.
DFF does need to evolve to remain as relevant as it is
As DFF is already moving ahead in all these areas, and has the appropriate systems and organisation in place, the evaluators are confident that this evolution is possible and likely. The evaluator’s strategic recommendations are targeted towards this as well:
- Broaden the focus on what is considered strategic in litigation to include organisational strength, timing, location, i.a.;
- Become more proactive in connecting digital rights actors, leveraging DFF’s unique position in the field;
- Roll out the decolonising approach and mainstream it in all aspects of the work and towards donors;
- Consider adapting the litigation support practice along three avenues:
– Limit iterations during the application process
– Increase the value of a grant for applicants through long-term grants covering multiple instances and funding adverse costs where necessary
– Integrate fieldbuilding work and grantmaking to further support applicants;
- Experiment with additional formats, styles and subjects for DFF’s field building meetings.
The external evaluators are grateful to all who took the time to provide input, think with us and connect us to other sources, in the course of the evaluation process. We especially wish to thank the DFF team for supporting us in facilitating the data collection in the current challenging circumstances and for being open and transparent in sharing information about their work.
Wouter de Iongh is Partner at ODS, a cooperative consultancy working exclusively with non-profit causes and organisations on Organisational Development, Monitoring, Evaluation & Learning, Strategy Development & Advice and Action Research.