Below is a selection of web-based program evaluation resources that we hope will be helpful to you. These may be useful to experienced evaluators, new evaluators, and non-evaluators.
What is Program Evaluation and What are the Different Types of Evaluation?
Theory of Change
Video: “How to Develop a Theory of Change”
Video: “DIY—Theory of Change
“Needs Assessment: Trends and a View Toward the Future,” New Directions in Evaluation, No. 144, Winter, 2014 James W. Altschuld and Ryan Watkins (eds.)
Additional Resources hosted by the American Evaluation Association:
George Orwell once wrote that it is sometimes the duty of intelligent men (sic) to re-state the obvious. In support of Orwell’s dictum, I’d like to restate in this blogpost something that we all know: despite our hopes for the power of education to correct for social disadvantage and inequality—to transform children’s life chances—persistent childhood poverty negatively affects children’s educational outcomes. I raise this issue because it has been my experience in working with educators that they sometimes sanguinely believe that education can overcome the deleterious effects of poverty. Sadly, research shows that educational attainment is stunted by poverty. Among research findings:
- The gap in standardized test scores between affluent and low-income students has grown by about 40 percent since the 1960s, and is now double the testing gap between blacks and whites.
- 22% of US children live in poverty.
- 40% of children living in poverty aren’t prepared for primary schooling.
- Children who live below the poverty line are 1.3 times more likely to have developmental delays or learning disabilities than those who don’t live in poverty.
- Between birth and age six, wealthier children will have spent as many as 1,300 more hours than poor children on child enrichment activities such as music lessons, travel, and summer camp.
- By the end of the 4th grade, African-American, Hispanic and low-income students are already 2 years behind grade level. By the time they reach the 12th grade they are 4 years behind.
- Dropout rates of 16 to 24-years-old students who come from low income families are seven times more likely to drop out than those from families with higher incomes.
- A child from a poor family is two to four times as likely as a child from an affluent family to have classmates with low skills and behavior problems – attributes which have a negative effect on the learning of their fellow students.
- International data shows that compared with other economically advanced countries, only Romania has a higher child poverty rate than the United States.
The implication of these facts, of course, is not that we (American society) should abandon our long-held commitment to educating all children, but that we should disabuse ourselves of the idea that education alone can solve the social problems of disadvantage and inequality. We have looked to public education to solve many societal problems, but the fact is that as these deepen and multiply—from unemployment to growing racial inequality— the institution of education can’t rectify these on its own. Our commitment to the highest quality public education should be a moral and political one, not a delusional one.
What is a Theory of Change?
A theory of change is a method for planning, monitoring, and evaluating initiatives in the non-profit, philanthropic, and government sectors. A theory of change articulates and graphically illustrates the assumptions that inform a change initiative, the prospective set of changes the initiative hopes to make, and the logical and chronological order in which causes and anticipated outcomes will occur. Theories of change ask that program planners, supporters, staff, and in some cases, participants, outline the causal pathway between an initiative’s actions and its ultimate goals. Although a theory of change is like a logic model, a theory of change differs from a logic model in so far as it is a graphic representation of the steps necessary to address a given problem or issue. A logic model is typically more focused on a specific program’s deployment of resources to produce specified outcomes.
Typically, theories of change include an explanation of how and why anticipated changes will occur, rather than simply mapping the relationship among inputs, outputs, and outcomes. Consequently, theories of change are explanatory, while logic models are descriptive. Clark and Anderson argue that logic models “usually start with a program and illustrate its components” while “theories of change (work) best when starting with a goal, before deciding which programmatic approaches are needed.” (For more information on the differences between a logic model and a theory of change see, Clark and Anderson’s “Theories of Change and Logic Models: Telling Them Apart”).
Patricia Rogers writes, “A ‘theory of change’ explains how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a program, a policy, a strategy or an organization” (see UNICEF—Theory of Change).
Why Develop a Theory of Change?
Everyone who is involved in a change process has an implicit idea of how changes come about. Collaboratively developing a theory of change can help stakeholders to build a shared understanding of the causal steps (and the underlying implicit assumptions) that are prerequisite to achieving a desired goal. A theory of change can illustrate the causal logic in a way that says “if we do ‘x’, then ‘y’ is likely to occur.” Essentially, theories of change are a diagram of how things should work, the resources needed to achieve a goal or set of goals, and the causal relationships between various factors that lead to the achievement of a goal.
Reasons for developing a theory of change also include:
- assisting stakeholders to articulate how and why a change initiative will work
- graphically representing the implicit and often tacit causal mechanisms at work in a service delivery system or programmatic initiative
- enhancing an understanding of how information can be used to measure or indicate changes (outcomes and impacts)
- clarifying for stakeholders why things may be working and not working
- building a consensus about how an initiative should work, so that it reaches its goals
What is a theory of change?
Annie E. Casey Foundation: Theory of Change: A Practical Tool For Action, Results, and Learning
Video: “How to Develop a Theory of Change”
Using a Theory of Change in Program Design and Planning
Video: “DIY—Theory of Change
Theory of Change—Wikipedia
Unicef—Theory of Change
A Community Builder’s Approach to a Theory of Change
Differences between a Logic Model and a Theory of Change
Theory of Change Portal (Netherlands)
Harvard Family Research Institute—Theory of change
Theory of Change in the Philanthropic Sector
I like quotes and have collected them for a number of years. Quotes from literature, film, newspapers, and other sources, often capture in a few insightful words, the way the world really works or how it should work. Below are some quotes that maybe useful to you and your colleagues, who manage, staff, and fund programs.
Flying is learning how to throw yourself at the ground and miss.
— Douglas Adams
Courage is resistance to fear, mastery of fear– not absence of fear.
— Mark Twain
One doesn’t discover new lands without consenting to lose sight of the shore for a very long time.
— Andre Gide
You miss 100% of the shots you don’t take.
— Wayne Gretzky
Not everything that counts can be counted, and not everything that can be counted, counts.
— Albert Einstein
I myself am human and free only to the extent that I acknowledge the humanity and liberty of all my fellows… I am properly free when all the men and women about me are equally free. Far from being a limitation or a denial of my liberty, the liberty of another is its necessary condition and confirmation.
— Michael Bakunin
An optimist proclaims that we live in the best of all possible worlds; the pessimist fears this is true.
— James Bench Cabell
The struggle itself towards the heights is enough to fill a human heart. One must imagine that Sisyphus is happy.
— Albert Camus
Ignorance more frequently begets confidence than does knowledge.
— Charles Darwin
Sixty years ago I knew everything; now I know nothing; education is the progressive discovery of our ignorance.
— Will Durant
An error can never become true however many times you repeat it. The truth can never be wrong, even if no one hears it.
— Mahatma Gandhi
It is well to remember that the entire universe, with one trifling exception, is composed of others.
— John Andrew Holmes
One of my favorite quotes is from Butch Cassidy and the Sundance Kid. You can see the one minute scene here:
How do non-profits use program evaluation? What are the trends and what are the challenges for evaluation in the non-profit sector? A recent report by the Center for Effective Philanthropy, based on information drawn from 514 non-profit organizations found that:
- 56% of nonprofits collect information from programmatic assessments, or indicators of outcomes they seek to change.
- 43% collect Information about their organization’s reach: number of beneficiaries served or units of service provided.
- While 90% devote some portion of their budget toward performance assessment, 55% spend 2% or less of their budget on that effort.
- Nearly one-third of nonprofits use third-party evaluators to conduct formal assessments of their performance, and those that do typically have annual expenses greater than $1.4 million.
- 83% of nonprofits reported that they are using their performance information to improve their programs and services at least “to a great extent,” and 68%use their information to inform their strategic direction at least “to a great extent.”
- Only 36% reported that they tend to receive financial and/or non-monetary support from their foundation funders to help assess their performance, with 64% reporting they receive no such support.
As we’ve discussed in earlier blog posts, evaluation can be an important tool for helping non-profits’ to understand the positive effects they are achieving and the differences they are making. As the above study indicates, evaluation can also be a important source of information for program refinement and determining strategic direction.
For more discussion about the various uses and benefits of program evaluation for non-profits, please see our articles, “Evaluation Serves Community Foundations and Donors,” “Program Evaluation Supports Strategic Planning” and “Program Evaluation and Organization Development.”