How do non-profits use program evaluation? What are the trends and what are the challenges for evaluation in the non-profit sector? A recent report by the Center for Effective Philanthropy, based on information drawn from 514 non-profit organizations found that:
- 56% of nonprofits collect information from programmatic assessments, or indicators of outcomes they seek to change.
- 43% collect Information about their organization’s reach: number of beneficiaries served or units of service provided.
- While 90% devote some portion of their budget toward performance assessment, 55% spend 2% or less of their budget on that effort.
- Nearly one-third of nonprofits use third-party evaluators to conduct formal assessments of their performance, and those that do typically have annual expenses greater than $1.4 million.
- 83% of nonprofits reported that they are using their performance information to improve their programs and services at least “to a great extent,” and 68%use their information to inform their strategic direction at least “to a great extent.”
- Only 36% reported that they tend to receive financial and/or non-monetary support from their foundation funders to help assess their performance, with 64% reporting they receive no such support.
As we’ve discussed in earlier blog posts, evaluation can be an important tool for helping non-profits’ to understand the positive effects they are achieving and the differences they are making. As the above study indicates, evaluation can also be a important source of information for program refinement and determining strategic direction.
For more discussion about the various uses and benefits of program evaluation for non-profits, please see our articles, “Evaluation Serves Community Foundations and Donors,” “Program Evaluation Supports Strategic Planning” and “Program Evaluation and Organization Development.”
Periodically, I share with you and your colleagues information and resources that are of special interest to the non-profit sector. In May, a colleague of mine, Matt Von Hendy, at Green Heron Information, will be offering a series of webinars that will help you and your organization to:
- Effectively search for, find, and respond to government Requests for Proposals (RFPs);
- Improve your on-line informational web search strategies; and
- Locate high quality, health-related information with an emphasis on evidence-based medical resources.
Links to registration for each of these webinars and more information about each, appear below:
Supercharge Your Search 2015 — Tuesday, May 5, 2-3 PM EDT
In the age of Google and Web 3.0, everyone is a researcher but the amount of information can be overwhelming. How do you minimize the amount of time spent looking at useless results just to get the right high quality information that you need? This webinar is designed to give you the strategies, techniques, and tools to help you find and organize twice the amount of high quality information in half the time. Click here to register.
Searching For and Finding RFPs (Government Contracting Opportunities): A Systematic Research Based Approach — Tuesday, May 12, 2-3 PM EDT
Searching for and finding RFPs (request for proposals) from government agencies can be a frustrating and time-consuming process. This webinar is designed to equip you with the strategies, techniques, and tools to help you more quickly and efficiently search for and locate federal, state, tribal and local government contracting opportunities. Click here to register.
Finding High Quality Health (Evidence Based) Resources on a Deadline and a Budget — Tuesday, May 19, 2-3 PM EDT
With increasing frequency, project sponsors are asking grantees to use high quality health resources in their proposals. Locating high quality health-related information can be a challenge – especially when you have an impending deadline and/or have a limited budget. This webinar covers strategies, resources and tools to quickly and effectively locate high quality low-cost health information with an emphasis on evidence-based medical (ebm) resources. Click here to register.
Each session costs $40 dollars, which covers all the standard webinar material AND also includes a 30 minute personalized post webinar session with Matt, where you can discuss your particular research needs.
About the presenter:
Matthew Von Hendy MA/MLS has more than 20 years of experience as a professional research librarian and has worked at EPA, NASA, and the National Academies of Science. In 2012, he established his own research and information consulting firm, Green Heron Information Services, and has been working closely with evaluation and other research professionals ever since. He frequently presents at local, regional, and national evaluation conferences.
Have questions or need more information? Contact Matt Von Hendy – firstname.lastname@example.org
You will recall that in an earlier post we discussed the importance of learning from program “failure.” (“Fail Forward: What We Can Learn from Program Failure”)
Below are some films and other resources about the value of risk and failure in helping us to learn and improve.
“I have not failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison
“By understanding how and why programs don’t achieve the results they intend, we can design and execute improved programs in the future. It is important to note that psychological research has shown that individuals learn more from failure than they do from success. Our goals should be to learn from our defeats and to surmount them—especially in programs that address critical social, educational, and human service needs. Learning from the challenges that confront these kinds of programs can have a powerful impact of the success of future programming.”
Before beginning an evaluation, it may be helpful to consider the following questions:
1. Why is the evaluation being conducted? What is/are the purpose(s) of the evaluation?
Common reasons for conducting an evaluation are to:
- monitor progress of program implementation and provide formative feedback to designers and program managers (i.e., a formative evaluation seeks to discover what is happening and why, for the purpose of program improvement and refinement.)
- measure final outcomes or effects produced by the program (i.e., a summative evaluation);
- provide evidence of a program’s achievements to current or future funders;
- convince skeptics or opponents of the value of the program;
- elucidate important lessons and contribute to public knowledge;
- tell a meaningful and important story;
- provide information on program efficiency;
- neutrally and impartially document the changes produced in clients or systems;
- fulfill contractual obligations;
- advocate for the expansion or reduction of a program with current and/or additional funders.
Evaluations may simultaneously serve many purposes. For the purpose of clarity and to ensure that evaluation findings meet the client’s and stakeholders’ needs, the client and evaluator may want to identify and rank the top two or three reasons for conducting the evaluation. Clarifying the purpose(s) of the evaluation early in the process will maximize the usefulness of the
2. What is the “it” that is being evaluated? (A program, initiative, organization, network, set of processes or relationships, services, activities?) There are many things that may be evaluated in any given program or intervention. It may be best to start with a few (2-4) key questions and concerns (See #4, below ). Also, for purposes of clarity, it may be useful to discuss what isn’t being evaluated.
3. What are the intended outcomes that the program or intervention intends to produce? What is the program meant to achieve? What changes or differences does the program hope to produce, and in whom? What will be different as the result of the program or intervention? Please note that changes can occur in individuals, organizations, communities, and other social environments. While evaluations often look for changes in persons, changes need not be restricted to alterations in individuals’ behavior, attitudes, or knowledge, but can extend to larger units of analysis, like changes in organizations, networks of organizations, and communities. For collective groups or institutions, changes may occur in: policies, positions, vision/mission, collective actions, communication, overall effectiveness, public perception, etc. For individuals: changes may occur in behaviors, attitudes, skills, ideas, competencies, etc.
How do programs know what they should be doing— which target populations require services, the types of services programs should provide, the amounts of services, which kinds of services will be most effective, etc.? Needs assessments are the best way to determine the needs of individuals, communities, and other populations. A needs assessment is a systematic process for identifying and determining such needs. Like program evaluations, needs assessments draw on a range of social science methods—from surveys and observations, to focus groups and individual interviews.
Needs assessments assume a clear definition of “a need”. As James Altschuld and Ryan Watkins point out in New Directions in Evaluation, No. 144, Winter, 2014 “A need, in the simplest sense, is a measurable gap between two conditions: what currently is, and what should be….This requires ascertaining what the circumstances are at a point in time, what is desired in the future, and a comparison of the two.” Needs assessments don’t just exclusively focus on what is and should be, but also on gathering and synthesizing data about how to narrow the gap between the existing state and the desired state. Needs assessments also prioritize needs so that users of the assessment can address specified needs in a reasonable order, and devote appropriate resources to meeting identified needs.
By gathering data from a range of stakeholders, needs assessments are able to determine the best means to achieve the desired results. To be effective, however, needs assessments must not simply focus on deficits in individuals and communities, but must also explore existing strengths, capacities, and assets. Too narrow a focus on “what’s missing” can blind researchers and program designers to the existing assets on which effective programming can be built. Effective needs assessments therefore, ask questions about: 1) on-going needs, 2) current strengths/assets/ capacities, 3) and desired states.
Needs assessments may differ in their design, but regardless of design, most needs assessment follow these phases:
- Explore and gather data about the current condition/state of affairs (including existing assets)
- Explore/identify desired or optimal condition/state of affairs
- Analyze data to understand the difference or “gap” between the current condition and desired condition.
- Prioritize identified needs and “gaps.”
- With needs (and assets) in mind, design program to address (diminish or eliminate) the gap between existing needs and desired state.
When conducted in a timely and thoughtful way, needs assessments can be of substantial utility in helping programs to effectively deliver services to those who most need them.
“Needs Assessment: Trends and a View Toward the Future,” New Directions in Evaluation, No. 144, Winter, 2014 James W. Altschuld and Ryan Watkins (eds.)
Definition of Needs Assessment
Comprehensive Needs Assessment
Methods for Conducting an Educational Needs Assessment