By Abby Rolland
How can impact be measured, why do funders ask for it, and how can measurement be helpful to nonprofits? In previous blog posts, harp-weaver LLC staff discussed impact itself and in the context of evaluation and governance. Here, we seek to explore the question of measuring impact and relevant concerns about measurement.
A side note: Measuring impact is not easy in any sector. Longitudinal data, a commitment to learning, and understanding how impact may result from multiple “interventions” are key cornerstones of measuring impact for nonprofits.
Measuring Outputs, Outcomes, and Impact
When measuring an organization’s program’s effectiveness, one can sort the results into three categories: outputs, outcomes, and impact.
Outputs – What happened as an immediate result of those actions?
These are typically immediate, tangible results of your activities and are usually easy to count.
Outcomes – What changes (short-term, immediate-term, long-term) do you expect to see because of those actions? What does success look like?
Outcomes chain:
Short term – What do you expect to see from your activities?
Intermediate term – What do you anticipate seeing as a result of your activities?
Long-term – What do you hope to see from your activities?
Impact – The positive/negative, primary/secondary, direct/indirect, intended/unintended, long-term effects of an intervention.
In other words, the degree to which the outcomes observed by a nonprofit are attributable to its activities.
Outputs are fairly easy to measure. The numbers that are produced, e.g., this afterschool program served x number of students this year. An outcome that is harder to measure could be the number of students in your program who graduated high school and continued postsecondary education. The impact, which is the hardest to measure, could seek to understand how much your afterschool program contributed to a student’s graduation from high school and college and continuance to a successful career.
These measures are often found in a logic model, which we shared in the first blog post on impact.
Why funders measure and how it can benefit nonprofits
We detailed above in what ways an organization’s activities or program can be assessed or measured. But why do funders ask for programs or organizations to be measured?
There are a variety of reasons, including accountability (to Boards, community members), learning and improvement regarding program effectiveness, informing strategic decision-making, demonstrating impact to others, fostering transparency and trust, and ensuring sustainability.
On the flip side, measurement can also help nonprofits by identifying strengths and weaknesses of programs, allocating resources more efficiently, building credibility, demonstrating evidence of impact to current and potential donors, informing strategic planning, building learning within the organization, and creating a case for sustainability.
Measurement considerations
There are several considerations to think about when it comes to measurement. For one, a funder and nonprofit must decide on the type of evaluation. Formative evaluations, or process evaluations, aim to help strengthen what’s being evaluated (whether that’s a program or an organization). It asks the question, “what did we do? How well did we do it?” On the other hand, a summative evaluation measures/examines the effects or outcomes of a program/project. It questions, “what difference did it (the program or organization) make?”
Measuring a program also requires deciding what kind of data to collect – qualitative (narrative) or quantitative (numbers). Using numbers to measure outputs can be straightforward; however, measuring outcomes or impact may require a more nuanced and holistic approach by utilizing both qualitative and quantitative data to fully understand an organization’s/program’s impact.
A key question with measuring and crafting an evaluation is whether or not the resulting evaluation will be used to inform the work of both the nonprofit and the foundation. If not, then why request the evaluation (and a nonprofit’s time, energy, and resources)? At harp-weaver, we recommend that a funder and nonprofit co-design an evaluation and its measurements in order for it to be the most helpful to all parties involved.
Measurement concerns
While it may seem straightforward to ask, “did a program or project work?” or “is an organization effective at what is does?” measuring raises additional considerations.
Measurement completed incorrectly or in a silo can also inhibit or hurt nonprofits’ efforts, limit their ability to effectively carry out their programs (because they are busy with the evaluation), or not contribute to their efforts or further their understanding.
In some situations, foundations ask that nonprofits evaluate their programs and/or organization, but funders don’t provide the resources to complete that evaluation. In other words, nonprofits are expected to evaluate their work but don’t receive the resources such as staff capacity to be able to fully conduct the evaluation.
In addition, while the outcomes or impact seem straightforward enough in the afterschool program we mentioned above, it can be difficult (if not impossible) to directly attribute a student’s success to the program. Was the student involved in other programs? Were there other external factors that played a role in their success?
Without a full longitudinal study (and patience by the nonprofit that received the grant and the funder that gave it), it can be hard to measure long-term impact.
Final thoughts
As demonstrated, measurement is a complicated topic with a number of considerations. While these aren’t all of the factors to think about related to measurement and evaluation, they provide some context and questions as to the efficacy and equity of measurement work.
harp-weaver staff has experience working with funders and nonprofits to co-design measurement metrics and evaluations that benefit both parties and provide insights into process and outcomes. Reach out to teresa@harp-weaver.com if you’re interested in learning more!
Comentários