How Measuring Systems Change Can Open the Door to Transformative Impact

It has been wisely observed elsewhere that “[n]ot everything that counts can be counted, and not everything that can be counted counts.” [1] As CEO of a global fund that aims to end modern slavery,^[2] I have experienced this truth first-hand. Tim Hanstad’s recent post on the CEP blog encouraging donors to embrace a systems change approach to measuring impact allowed me to reflect upon the opportunities and challenges that such a shift has presented to the Freedom Fund and our partners. Though definitions of “systems change” can vary widely, we can all acknowledge that it is both difficult to achieve and difficult to quantify. But I agree with Hanstad’s premise: when we base our approach to funding on what is easily measurable, we risk missing out on investing in deeply transformative work.

The Freedom Fund has always been highly focused on quantitative measures of impact. We started out in 2014 by measuring direct impact across our programs, mainly in the form of “lives liberated.” We included some other metrics that we saw as closely linked to reductions in vulnerability to modern slavery, like number of at-risk children enrolled in school, number of people accessing social and legal services and number of micro-enterprises started, as well as overall lives impacted by Freedom Fund-supported programs.

But we still had questions about the sustainability and depth of that impact. Questions like: While the number of people in situations of trafficking in a community may decrease over the lifespan of a five-year program, will the rates go back up after the program ends? Are the interventions we fund taking aim at alleviating the symptoms of exploitation, or actually getting to the root causes? If our mission is to permanently, not just temporarily, eradicate modern slavery, what should impact look like?

One important approach when it comes to measuring the efficacy of anti-slavery initiatives is to focus on prevalence, i.e., the percentage of population affected. If we can start by establishing a baseline prevalence rate in a particular area before we start funding there, then we should be able to compare that to prevalence three, five, or ten years later to understand change over time. Using this approach, the Freedom Fund began to build a robust body of evidence, most of it generated through evaluations by leading research institutions, that shows that our approach to funding has enabled dramatic reductions in modern slavery prevalence in targeted communities.

Like most funders who aim to tackle thorny issues, we understood that change doesn’t happen through just one type of intervention. Complex systems call for complex solutions. Thus, we began to seriously explore how to support efforts to dismantle the systems that underpin modern slavery. Our priority broadened from just helping individuals exit situations of slavery to ensuring those at risk never end up in situations of exploitation, in part by holding power-holders to account. This means making sure that anti-slavery laws are passed and properly implemented. It means helping vulnerable individuals get access to government benefits and subsidies they are entitled to, so they can avoid the economic desperation that often pushes people into riskier forms of work. It means shifting business incentives and challenging social norms.

Our grantee partners began engaging more with governments and businesses, helping to strengthen local social protections and improve workplace conditions. They were ambitious and strikingly successful, and we began to realize these efforts could have impact far beyond the life cycle of our support. When a law is changed, a legal precedent is set, or a business changes its sourcing practices, the effects are felt by populations far larger than those with whom our partners directly interact. We wanted to support this type of large-scale change, but our impact measurement framework was limited to direct impact figures. We had no way of counting the impact on those we couldn’t see. We therefore sought a new way to measure how our efforts and resources contribute to overall change, even in cases where traditional metrics may fall short.

So, we gave our research team a challenge: develop a framework for measuring indirect impact, defined as “those who do not come into immediate contact with our grantee partners and program activities, but nevertheless gain from the systems change that a program has contributed towards.” Our team analyzed various forms of systems change and came up with a three-level system of categorization: protective environment, resilient communities, and empowered movement. Each of these categories describes a different level of impact, ranging from the broadest population affected by changes, like national legislation and public attitudes (“protective environment”), down to the more micro-level shifts experienced by stronger collaboration between organizations (“empowered movement”).

Today, we categorize each systems change milestone and estimate the size of the population affected, being purposefully conservative to avoid overcounting. This model is far from perfect, but it has allowed us to evaluate our contributions and to compare the potential impact of various investments. We now estimate that through the Freedom Fund’s support, our partners have indirectly impacted 6.5 million lives. We have also set a new target of indirectly impacting 10 million individuals by 2025.

This new framework has changed our strategy and funding decisions. It has helped us to see and embrace the full impact of our partners’ work. Rather than choosing only to support work that fits a limited, often-nearsighted definition of immediately countable impact, we have adjusted our idea of impact to fit what is happening on the ground, enabling us to invest in longer-term projects with bolder ambitions. Our data now paints a much richer and more nuanced picture of the impact of both our partners and our funding.

Take, for example, our program in southeastern Nepal. Since 2014, the Freedom Fund has supported a group of frontline organizations in their campaign to free families trapped in a system of agricultural bonded labor called Harawa-Charawa. This form of modern slavery has targeted generations of marginalized Dalit communities, who are forced to take on high-interest debts and to work on their lenders’ land as tillers, cattle herders, or domestic servants for well below market wage. Unable to escape the cycle of debt, they face poverty, physical abuse, and threats of expulsion from their homes and the land they are share-cropping.

Over time, we’ve invested over $7m in the Nepal program, both in expanding direct support to the Harawa-Charawa community and strengthening our grantee partners’ internal systems. We have supported survivors to build the democratically elected Harawa-Charawa Network and invested in their capacity to advocate for their constituents and influence powerholders. We’ve funded these efforts knowing that most of the impact they produce is not immediately measurable, as compared to other programs.

In July, these groups celebrated a major breakthrough. Although bonded labor was officially abolished in Nepal in 2002, the 120,000 Harawa-Charawa were not legally designated as eligible for government rehabilitation programs. The Harawa-Charawa Network’s leaders were present as Nepal’s prime minister announced that all Harawa-Charawa must now be free from loans, recognized explicitly in law, and able to access government support. While much work is left to be done to make this commitment a reality, this was a major sign of progress and the culmination of many years of persistent advocacy.

I think back to eight years ago when we first made the case for investment in southeastern Nepal. If we hadn’t embraced the possibility of transformative change brought about through long-term advocacy, we would have limited our support to direct services that marginally, and often only temporarily, improve people’s lives, rather than also supporting efforts toward wholesale legislative changes that will cement the freedom of 120,000 people and their future generations. Expand our definition of impact allowed us to share the vision of our local staff and partners who believed that, despite great obstacles, such a milestone was possible.

We all know that societal-level change can be messy and uncertain. As donors, we have an opportunity to adjust our understanding of impact to better match reality — and invest accordingly. In doing so we may have to focus more on our (difficult to measure) contribution to longer-term change rather than claiming direct responsibility for short-term change — but the results can be transformative.

Nick Grono is CEO of The Freedom Fund and is currently writing a book on nonprofit leadership. Find him on LinkedIn.

[1] This quote is often attributed to Albert Einstein, but it appears the correct source is sociologist William Bruce Cameron.

^[2] “Modern slavery” is an umbrella term used to describe a number of forms of exploitation including forced labor, debt bondage, forced marriage, and human trafficking. It refers to situations in which an individual is exploited through threats, violence, coercion, or deception for another party’s commercial or personal gain.

This piece was originally published on the Center for Effective Philanthropy’s blog and has been reposted with permission.