Questioning assumptions about monitoring, evaluation, and learning in the social sector
We all have orthodoxies—deeply held beliefs about "how things are done" that often go unstated and unquestioned. These orthodoxies often become standard practices that help individuals and institutions function more efficiently. But they can also lead to a dogmatic resistance to change and blind spots that can prevent us from developing new and better ways of working. Challenging, and potentially “flipping” these orthodoxies can often lead to drastic improvements in practice.
- Explore the complete toolkit
- Outdated orthodoxies
- Putting decision making at the center
- Empowering constituents and promoting diversity, equity, and inclusion
- Productively learning at scale
Better understanding the orthodoxies within your organization and the field can be an important first step to identifying more beneficial practices. Below you can find a few examples of orthodoxies and organizations that are flipping them for each of the three characteristics of a better future.
For a complete set of orthodoxies, please download the Orthodoxies cards PDF. These cards can be used in an orthodoxy flipping exercise for challenging and flipping assumptions that may be holding you back. Specific instructions for the orthodoxy flipping exercise are listed on the cards.
Examples of orthodoxies to flip for more effectively putting decision making at the center
Orthodoxy: Data collection and analysis are expensive
- Acumen, a nonprofit impact investor, prioritizes the collection and use of data that its investees actually value. Its approach, which it calls Lean Data, uses surveys delivered largely by mobile to gather quality data directly from end consumers as cost and time efficiently as possible. This data enables Acumen and its social entrepreneurs to listen at scale to the people whose lives they aim to improve, allowing them to better understand the impact of their work as well as a range of fundamental customer feedback topics.
In the last two years, Acumen has worked with more than 40 companies inside and outside of its portfolio to implement Lean Data projects to have conversations with more than 25,000 base-of-the-pyramid customers. For example, Acumen used mobile interviews to reach the customers of d.light, a global enterprise that sells solar products. The data revealed a range of positive social benefits but also that 36 percent of customers that had experienced an issue with its product did not reach out to the company. The company has since made more proactive steps to reach out to customers.
Orthodoxy: Failures are something to avoid or hide
- Every year, Engineers without Borders Canada (EWB)1 publishes a failure report in conjunction with its annual report. The failure report includes cases written by leadership and staff members who describe national office failures, venture failures, and chapter failures, along with their lessons learned. By publically celebrating failures, EWB increases incentives to share information, take risks, and be creative.
In a better future:
- Information for on-the-ground decision making is prioritized
- Learning is embedded and continuous
- There is greater investment in monitoring, evaluation, and learning capacity
- The data and methods needed to inform decisions are available
Examples of orthodoxies to flip for better empowering constituents and promoting diversity, equity, and inclusion
Orthodoxy: Constituents are participants, not beneficiaries, of evaluation
- The Patient-Centered Outcomes Research Institute (PCORI), an independent nonprofit organization that aims to provide data to help patients and their caregivers make better informed health decisions, works to advance the shift in clinical health research from “investigator-driven” to “patient-centered” studies.
PCORI involves patients in all aspects of the research process—choosing research and outcomes topics, developing and conducting studies, and sharing the results. For example, patients, clinicians, and researchers share in governance of the network and data use decisions for PCORnet, PCORI’s initiative to harness patient data to facilitate more efficient and powerful research studies.
Orthodoxy: Including constituent voice in M&E is nice, but not necessary
- GlobalGiving, a global crowdfunding platform for nonprofits, incentivizes the collection of constituent feedback by tying it to perks on the platform that typically lead to increased funding. Through GlobalGiving Rewards, a program similar to a frequent flier program, organizations earn points for engagement (e.g., reporting on their progress) and effectiveness (e.g., listening to their constituents).
GlobalGiving uses points accumulated by nonprofits to measure growth and translates that growth into a status (e.g., Emerging, Superstar). The higher an organization’s status, the more GlobalGiving drives donations to that organization through its platform.
In a better future:
- Equity is consistently considered in and supported by monitoring, evaluation, and learning efforts
- Constituent feedback is an essential practice
- Constituents are empowered to make their own choices
- Data rights are secured
Examples of orthodoxies to flip for more productively learning at scale
Orthodoxy: Monitoring and evaluation are for experts
- Measure4Change, an initiative of Urban Institute and the World Bank Group, hosts a community of practice for nonprofit evaluation staff in the Washington, DC, area to enable them to learn from one another. The community meets quarterly and the sessions are intended to make performance management more accessible by helping nonprofit leaders learn from their peers. By participating in the community of practice, nonprofit leaders can better understand the range of performance management practices, assess their programs compared to their peers, and explore how they can advance their work.
In addition to the community of practice, Measure4Change also offers grant support and one-on-one technical assistance to small cohorts of nonprofits and a series of knowledge briefs bear on strategic management and policy decisions. Among other efforts, the project trains and places “data fellows” in partner organizations to provide in-house data analytics support, help develop evidence-informed policy, and improve the organization’s ability to leverage strategic analytics.
Orthodoxy: Collaboration is too hard
- The Humanitarian Data Exchange (HDX) is an open platform managed by the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) to share data about humanitarian crises. The goal of HDX is to make data easier to find and use for analysis. Over 200 organizations are sharing 4,400 datasets that are being accessed by users in almost every country in the world.
For example, HDX includes 84 datasets for the 2015 earthquake in Nepal, covering changes in global food prices, landslide locations, health infrastructure, and population movements. These datasets can be analyzed together to understand recovery efforts in Nepal. Building on the work of HDX, OCHA will establish a new Centre for Humanitarian Data in The Hague in mid-2017 to further increase data use and impact in the humanitarian sector.
In a better future:
- Data, learning, and knowledge are shared openly and widely
- Knowledge gaps and learning agendas are collaboratively undertaken
- Data is integrated at scale needed to assess social impact
- Evaluation synthesis, replication, and meta-evaluation are supported
1All information except the Failure Report comes from personal communications with the organizations. For information on the Failure Report, see Engineers Without Borders. ""Engineers Without Borders Failure Report." 2016."