Qualitative interviews can be an important source of program evaluation data. Both in-depth individual interviews and focus group interviews are important methods that provide insights and phenomenologically rich descriptive information that other, numerically oriented data collection methods (e.g., questionnaires and surveys), are often unable to capture. Typically, interviews are either structured, semi-structured, or unstructured. (Structured interviews are, essentially, verbally administered questionnaires, in which a list of predetermined questions is asked. Semi-structured interviews consist of several key questions that help to define the areas to be explored, but allow the interviewer to diverge from the questions in order to pursue or follow up to an idea or response. Unstructured interviews start with an opening question, but don’t employ a detailed interview protocol, favoring instead the interviewer’s spontaneous generation of subsequent follow-up questions. See P. Gill1, K. Stewart2, E. Treasure3 & B. Chadwick, cited below.)
Interviews, especially one-on-one, in-person, interviews, allow an evaluator to participate in direct conversations with program participants, program staff, community members, and other program stakeholders. These conversations enable the evaluator to learn about interviewees’ experiences, perspectives, attitudes, motivations, beliefs, personal history, and knowledge. Interviews can be the source of pertinent information that is often unavailable to other methodological approaches. Qualitative interviews enable evaluators to: elicit direct responses from interviewees, probe and ask follow-up questions, gather rich and detailed descriptive data, offer real-time opportunities to explain and clarify interview questions, observe the affective responses of interviewees, and ultimately, to conduct thorough-going explorations of topics and issues critical to the evaluation initiative.
Although less intimate, focus group interviews are another key source of evaluation data. Organized around a set of guiding questions, focus groups typically are composed of 6-10 people and a moderator who poses open-ended questions to focus group participants. Focus groups usually include people who are somewhat similar in characteristics or social roles. Participants are selected for their knowledge, reflectiveness, and willingness to engage topics or questions. Ideally – although not always possible – it is best to involve participants who don’t previously know one another. Focus groups can be especially useful in clarifying, qualifying, and/or challenging data collected through other methods. Consequently they can be useful as a tool for corroborating findings from other research methodologies.
Regardless of the specific form of qualitative interview—individual or focus group, in-person or telephone—qualitative interviews can be useful in providing data about participants’ experience in, and ultimately the effectiveness of, programs and initiatives.
Learning from Strangers: The Art and Method of Qualitative Interview Studies, Robert S. Weiss, Free Press.
Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences, Irving Seidman. Fourth Edition.
With the end of the year fast approaching, this is good time to review the array of online resources that we’ve shared in recent months with non-profit leaders, foundation staff, educators, and fellow program evaluators. The links below represent the most popular blog posts of 2015. If you haven’t yet had a chance to look at one or more of these, please feel free to do so. You’re also welcome to share these links with colleagues and to keep these links on hand for future reference.
There are also a number of additional program evaluation resources on the site.
We are looking forward to offering more thoughts and additional evaluation resources in 2016!
Below is a selection of web-based program evaluation resources that we hope will be helpful to you. These may be useful to experienced evaluators, new evaluators, and non-evaluators.
What is Program Evaluation and What are the Different Types of Evaluation?
Theory of Change
Video: “How to Develop a Theory of Change”
Video: “DIY—Theory of Change
“Needs Assessment: Trends and a View Toward the Future,” New Directions in Evaluation, No. 144, Winter, 2014 James W. Altschuld and Ryan Watkins (eds.)
Additional Resources hosted by the American Evaluation Association:
George Orwell once wrote that it is sometimes the duty of intelligent men (sic) to re-state the obvious. In support of Orwell’s dictum, I’d like to restate in this blogpost something that we all know: despite our hopes for the power of education to correct for social disadvantage and inequality—to transform children’s life chances—persistent childhood poverty negatively affects children’s educational outcomes. I raise this issue because it has been my experience in working with educators that they sometimes sanguinely believe that education can overcome the deleterious effects of poverty. Sadly, research shows that educational attainment is stunted by poverty. Among research findings:
- The gap in standardized test scores between affluent and low-income students has grown by about 40 percent since the 1960s, and is now double the testing gap between blacks and whites.
- 22% of US children live in poverty.
- 40% of children living in poverty aren’t prepared for primary schooling.
- Children who live below the poverty line are 1.3 times more likely to have developmental delays or learning disabilities than those who don’t live in poverty.
- Between birth and age six, wealthier children will have spent as many as 1,300 more hours than poor children on child enrichment activities such as music lessons, travel, and summer camp.
- By the end of the 4th grade, African-American, Hispanic and low-income students are already 2 years behind grade level. By the time they reach the 12th grade they are 4 years behind.
- Dropout rates of 16 to 24-years-old students who come from low income families are seven times more likely to drop out than those from families with higher incomes.
- A child from a poor family is two to four times as likely as a child from an affluent family to have classmates with low skills and behavior problems – attributes which have a negative effect on the learning of their fellow students.
- International data shows that compared with other economically advanced countries, only Romania has a higher child poverty rate than the United States.
The implication of these facts, of course, is not that we (American society) should abandon our long-held commitment to educating all children, but that we should disabuse ourselves of the idea that education alone can solve the social problems of disadvantage and inequality. We have looked to public education to solve many societal problems, but the fact is that as these deepen and multiply—from unemployment to growing racial inequality— the institution of education can’t rectify these on its own. Our commitment to the highest quality public education should be a moral and political one, not a delusional one.
What is a Theory of Change?
A theory of change is a method for planning, monitoring, and evaluating initiatives in the non-profit, philanthropic, and government sectors. A theory of change articulates and graphically illustrates the assumptions that inform a change initiative, the prospective set of changes the initiative hopes to make, and the logical and chronological order in which causes and anticipated outcomes will occur. Theories of change ask that program planners, supporters, staff, and in some cases, participants, outline the causal pathway between an initiative’s actions and its ultimate goals. Although a theory of change is like a logic model, a theory of change differs from a logic model in so far as it is a graphic representation of the steps necessary to address a given problem or issue. A logic model is typically more focused on a specific program’s deployment of resources to produce specified outcomes.
Typically, theories of change include an explanation of how and why anticipated changes will occur, rather than simply mapping the relationship among inputs, outputs, and outcomes. Consequently, theories of change are explanatory, while logic models are descriptive. Clark and Anderson argue that logic models “usually start with a program and illustrate its components” while “theories of change (work) best when starting with a goal, before deciding which programmatic approaches are needed.” (For more information on the differences between a logic model and a theory of change see, Clark and Anderson’s “Theories of Change and Logic Models: Telling Them Apart”).
Patricia Rogers writes, “A ‘theory of change’ explains how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a program, a policy, a strategy or an organization” (see UNICEF—Theory of Change).
Why Develop a Theory of Change?
Everyone who is involved in a change process has an implicit idea of how changes come about. Collaboratively developing a theory of change can help stakeholders to build a shared understanding of the causal steps (and the underlying implicit assumptions) that are prerequisite to achieving a desired goal. A theory of change can illustrate the causal logic in a way that says “if we do ‘x’, then ‘y’ is likely to occur.” Essentially, theories of change are a diagram of how things should work, the resources needed to achieve a goal or set of goals, and the causal relationships between various factors that lead to the achievement of a goal.
Reasons for developing a theory of change also include:
- assisting stakeholders to articulate how and why a change initiative will work
- graphically representing the implicit and often tacit causal mechanisms at work in a service delivery system or programmatic initiative
- enhancing an understanding of how information can be used to measure or indicate changes (outcomes and impacts)
- clarifying for stakeholders why things may be working and not working
- building a consensus about how an initiative should work, so that it reaches its goals
What is a theory of change?
Annie E. Casey Foundation: Theory of Change: A Practical Tool For Action, Results, and Learning
Video: “How to Develop a Theory of Change”
Using a Theory of Change in Program Design and Planning
Video: “DIY—Theory of Change
Theory of Change—Wikipedia
Unicef—Theory of Change
A Community Builder’s Approach to a Theory of Change
Differences between a Logic Model and a Theory of Change
Theory of Change Portal (Netherlands)
Harvard Family Research Institute—Theory of change
Theory of Change in the Philanthropic Sector