Reimagining measurement Enhancing social impact through better monitoring, evaluation, and learning
Non-profits have to be accountable for the money they receive and foundations today are increasingly asking for hard data. But how do we reimagine measurement to measure social impact? Rhonda Evans and Tony Seisfeld throw light on the constituent voice, the “bright spot,” and asset-based framing.
Are we moving the needle on the abatement of hunger? Are we moving the needles on providing healthy, affordable food in so-called food deserts? I once heard a representative from a family foundation say the family had been providing help to this food kitchen in their community and the food kitchen proudly said, “Last year, we served 250,000 warm meals and this year, we’ve served 300,000 warm meals.” And he said, “Is that better or worse?”
TANYA OTT: We’re talking the world of non-profits today and how do we really know they’re making a difference?
I’m Tanya Ott and this is the Press Room—Deloitte’s podcast on the issues and ideas that matter to your business today. I’ve worked for non-profits my entire career. If you don’t count babysitting, I got my first job in 1986, and since then I have only worked for non-profits, with the exception of a year in college when I hawked fancy candles at the mall, and this podcasting gig. I wrote my first grant application nearly 20 years ago. And since then, I’ve secured well over a million dollars for non-profits—and I’m not even a professional grant writer.
There’s one thing I’ve noticed. Reporting requirements have gotten more rigorous. Funders want to know the money they’re giving is having an impact. And why shouldn’t they? Every year, foundations give tens of billions of dollars1 to the more than 1.5 million registered non-profits2—and that’s just in the US. Schools, churches, homeless shelters, sports programs, organizations that prevent cruelty to children and animals, civic leagues—they’re all expected to be accountable for the money they receive. And increasingly, they’re expected to prove it with hard data—the kind corporations have tracked for years.
RHONDA EVANS: We help organizations figure out how to best use data and information to make their organizations more effective.
TANYA OTT: That’s Rhonda Evans. She’s with the Monitor Institute by Deloitte, which focuses on serving foundations, non-profits, and occasionally government agencies.
RHONDA EVANS: A lot of the people who are sitting on boards are folks that come out of the tech industry, that come out of finance, and so they have much more quantitative backgrounds and expectations about access to data. Certainly, there’s a generational shift that’s happening, too. The millennials are now reaching the age of board membership. They’re starting to have leadership positions. There’s more comfort with data and technology. So there’s been a lot around “pay for success.” How can we know that we are getting effective use of public dollars?
TANYA OTT: What are some of the challenges that organizations face when they’re looking at that measurement question?
RHONDA EVANS: It’s hard because there’s not the same kind of straightforward ability to measure everything in dollars or return on investment.
TONY SEISFELD: And what makes social impact more difficult than traditional commercial enterprises are the things we’re doing.
TANYA OTT: That’s Tony Seisfeld. He’s a data analytics guy whose specialty is testing and measurement. He’s also Rhonda’s research partner at the Monitor Institute.
TONY SEISFELD: The way that most organizations have, in the past, proved that their efforts are working is through close field observation, sort of more anthropological. There’s been a press in the last 10 or 15 years to bring in more quantitative type of data into the conversation. If we’re doing livelihood development or we’re doing lifelong learning and job preparation, it may take 15, 20, or 30 years.
RHONDA EVANS: From a non-profit’s perspective, they’re often asked for a lot of data from foundations and they don’t provide the resources or the expertise that can help organizations gather and understand that data. There’s a kind of disconnect there.
TONY SEISFELD: In the world that we now live in, where in education and health care and many sectors, we are generating tons and tons and tons of data, the data is inexpensive, but the techniques for extracting patterns that are reliable and robust are only now being invented.
TANYA OTT: Over the past year, Rhonda and Tony have interviewed many people working in the field. They’ve also reviewed more than a 1,000 articles to pull out key trends that are happening in the social sector. They were looking for what they call “bright spots”—programs that are doing something different and having success with it. One of the themes they identified was the idea of constituent voice.
TONY SEISFELD: Traditionally, the privileged perspective has been that of the funder. We want to know how our money is working. And often the decision about whether it’s a good grant to make or a good investment to make is held by the foundation program officer.
The movement to get constituent voice is really a movement to say there shouldn’t be a single privileged perspective in this flow of decisions. It’s a system. And the people who are in the community [being served] are trying to make things better for themselves and they may have some strong ideas about what better looks like and how to get there.
RHONDA EVANS: Making [them] participants rather than passive recipients. And reframing who gets to define what the need is, what constitutes success, and what impact interventions are having.
TANYA OTT: Rhonda cites one of the so-called “bright spots”—a program called the Family Independence Initiative.
RHONDA EVANS: The Family Independence Initiative works with families to help them escape poverty. They have a very strong perspective that it is about helping the families help themselves or provide the tools and technologies that the families need to make their own choices. That is coming from a perspective that it’s about empowering their families, rather than telling the families what they need.
They have a technology platform. And they ask questions about finances, but they ask a whole range of questions. One of the things that is very important is that they ask about assets that other people don’t consider assets or don’t get tracked—informal assets like people who provide informal child care services for you or informal lending circles or other things like that. They also provide aggregated data for the families: You know, here’s what’s working for other families. Let’s help figure out what you need. And then they provide a technology platform also so that families can learn from each other. So, I’m trying to save money for a house and I’d love to talk to other folks who have done this and experienced it. It’s very much an empowerment perspective and enabling families to learn from each other, rather than the non-profit being the expert that has all of the answers.
TONY SEISFELD: There’s an organization that is talking about asset-based framing versus deficit-based framing in the way in which we think about how social action happens. Deficit framing tends to talk about the group you’re trying to serve by leading with the problem. There are youth from a low-income community who are at risk of going to jail. That framing frames the problem. The funder and the direct service providers are then the heroes of this story. You know, this direct service provider, its after-school program, and its funders are the heroes to help these disadvantaged, at-risk youth.
Asset-based framing would be to say, “There’s a population in this community that’s really seeking to address some of the challenges that they feel they face. And they’re looking for help with access to jobs, with skill building, and perhaps social emotional support as they enter the workforce for the first time.” That same population we have framed now in terms of an asset or in a more positive way, and civil society is in the service of what this population, these youths, are trying to do.
TANYA OTT: Tony and Rhonda say flipping that frame can lead to some unexpected outcomes.
RHONDA EVANS: They have a really great example. It’s about a mom who had three kids and her kids weren’t doing well in school. If you were looking at it from the outside, you would say, “We need an education intervention. We need mentoring or tutoring for these kids. Something education focused.” But she said, “What I really need is a car because what’s happening is, I have one child who has asthma and if my child has an asthma attack in the morning, I can’t take my other kids to school because we take the bus and I can’t take my asthmatic child on the bus.”
TONY SEISFELD: By receiving funding to get a reliable car, she was actually able to get faster, better medical treatment for her child. Her child was able to attend school. The absentee rate declined and she was able to get a more reliable, better paying job.
TANYA OTT: That’s a really, really compelling example of an individual situation, but if you’re running a project related to absenteeism, how do you scale that up?
TONY SEISFELD: Here’s why you actually need to have multiple perspectives in the system. So you are running a project in absenteeism. It’s a real issue. It’s an important issue. By engaging the community, by engaging the constituents who may or may not avail themselves of the program, you’ll actually do a better job of laying your resources against where the interventions are most likely to work.
If in a 100 cases, 25 of them are situations that are similar to the example I just gave you, your absentee interventions are not going to be as effective as they would be if your targeting were better. And your targeting is only going to be better if you actually talked to and involved in enrolling and enlisting the constituents for that particular intervention.
TANYA OTT: Here’s the point in the podcast where I stop to tell you that Tony and Rhonda identified lots of “bright spots”—more than 90 of them, which they detail in an article at deloitte.com/insights. But it’s not all rosy news. When they talked to people working in the trenches, they asked them, “Where would you like the social sector to be 10 years from now in terms of monitoring, evaluation, and learning?” And then, “Where do you think it will be if there’s no intervention?”
RHONDA EVANS: What we found is both that there was a lot of consensus about where people wanted to end up. And then simultaneously that they didn’t think that we were going to get there absent some intervention! I think there was more consensus than I expected going into the process, and then more pessimism, given all of the interesting changes that are happening around data availability, data aggregation, all of those things.
TONY SEISFELD: I once worked with a colleague who said, “When you’re planning something, you can never get from here to there. You can only get from there to here.”
Paint the picture of the destination and work backwards to understand what that path is. That path forward is always harder to imagine than the path backwards.
TANYA OTT: Is this a thing that can happen incrementally? Or is there a wholesale revolution that needs to take place in the social sector?
RHONDA EVANS: I’m going to say both. And, of course, there is one piece of this that really is about just spreading these practices. Making these “bright spot” examples more widely known and for people to say, “Yeah, I could actually do that in my organization.”
The other piece of it is more wholesale, though, because it really is about how the social sector operates and the relationship between foundations and non-profits. It’s about experimentation and trying new things and if we can get small pockets of people trying and experimenting, whether it’s new ways of doing grant reporting or ways of trying to learn collectively together, we can see what works and what doesn’t.
TANYA OTT: Of course, in the tech world, there’s that “fail fast and fall forward” idea. But having spent a very long time in non-profits, it feels like there’s so much at stake that failing is just not something that there’s a ton of appetite for.
RHONDA EVANS: Yeah.
TANYA OTT: So you’re agreeing with my observation, having worked from inside.
RHONDA EVANS: Yes.
TANYA OTT: It’s this weird conundrum because how do you evolve as a sector if you’re afraid of failure
RHONDA EVANS: Right. And that’s a huge issue. And failure in multiple ways, right, because we don’t want to fail our constituents. These are folks who we’re trying to help. And so not helping them or doing something that’s hurtful, of course people are very legitimately worried about that. Then there’s also a fear of failure that comes with funding, right? We’re afraid to tell a foundation that this didn’t work. But you can learn within constraints, right?
TONY SEISFELD: We cannot wave a magic wand to do this. The author Bhaskar Chakravorti wrote the book, The Slow Pace of Fast Change, and in it he talked a lot about how do you make [a] system change happen? His teaching is that the system wants to be in stasis. It likes to be where it is. Think of the field of measurement and evaluation inside the social sector as being a system. In order to change it and get it to a new stasis, you actually have to choose three or four places to push the system and try to move it to a new normal. One lesson is we have to run a bunch of different experiments at different parts in the system. And it’s everybody’s responsibility.
TANYA OTT: Tony Seisfeld and Rhonda Evans’ article is Reimagining measurement: Enhancing social impact through better monitoring, evaluation, and learning. You can find it at deloitte.com/insights. You’ll also find our podcast archives there—including a show on one of the most high-stakes positions in the C-suite. It’s not the CEO.
KHALID KARK: “Even if there is a single failure in the first few months, yes, you can, for a certain time frame, blame the previous leader. But I think your neck is at stake, right?”
TANYA OTT: We’ve got a new twitter handle and we’d love to hear from you! Tweet us or tag us @DeloitteInsight—that’s singular. No “S” at the end. I’m Tanya Ott for the Press Room. Catch you again in two weeks when we’ll be talking about Generation Z entering the workforce. You don’t want to miss it.
This podcast is provided by Deloitte and is intended to provide general information only. This podcast is not intended to constitute advice or services of any kind. For additional information about Deloitte, go to Deloitte.com/about.