In a recent, themed issue devoted to the topic of validity in program evaluation, the journal New Directions in Evaluation (No. 142, Summer, 2014) revisited and commented on Ernest House’s influential 1980 book, Evaluating with Validity. House then argued that validity in evaluation must not just be limited to classic scientific conceptions of the valid (i.e. empirically describing things as they are), but that it must also include an expanded dimension of argumentative validity, in which an evaluation “must be true, coherent, and just.” Paying particular attention to social context, House argued that “there is more to validity than getting the facts right.” He wrote that “…the validity of an evaluation depends upon whether the evaluation is true, credible, and normatively correct.” House ultimately argued that evaluations must make compelling and persuasive arguments about what is true (about a program) and thereby bring “truth, beauty, and justice” to the evaluation enterprise.
In the same issue of New Directions in Evaluation, in her essay, “How ‘Beauty’ Can Bring Truth and Justice to Life,” E. Jane Davidson argues that the process of creating a clear, compelling, and coherent evaluative story ( i.e., a “beautiful” narrative account) is the key to unlocking “validity (truth) and fairness (justice).” To briefly summarize, Davidson argues that a coherent evaluation story weaves together quantitate evidence, qualitative evidence, and clear evaluative reasoning to produce an account of what happened and the value of what happened. She says that an effective evaluation—one that is truly accessible, assumption-unearthing, and values-explicit—enables evaluators to “arrive at robust conclusions about not just what has happened, but how good, valuable, and important it is.” (P.31)
House’s book and Davidson’s essay highlight how effective evaluations—ones that allow us to clearly see and understand what has happened in a program— rely on strong narrative accounts that tell a coherent and revealing story. Evaluations are not just tables and data—although these are necessary parts of any evaluation narrative—they are true, compelling, and fact-based stories about what happened, why things happened the way they did, and what the value (and meaning) is of the things that happened.
“When I reflect on what has improved the quality of my own work in recent years, it has been a relentless push toward succinctness and crystal clarity while grappling with some quite complex, and difficult material. For me this means striving to produce simple direct and clear answers to evaluation questions and being utterly transparent in the reasoning I have used to get to those conclusions.” (p.39)
Davidson further observes that evaluation reports often are often plagued by confusing, long-winded, and academic jargon, that make them not only difficult to read, but obfuscate the often muddled and ill-reasoned thinking behind the evaluation process itself. She argues that evaluation reporting must be clear, accessible, and simple—which does not mean that reports need to be simplistic, but that they must be coherent and comprehensible. I am reminded of a statement by the philosopher John Searle, ‘If you can’t say it clearly, you don’t understand it yourself’.
Reflecting on Davidson’s article, I realize that the best evaluation reports are the product of well thought-out and effectively conducted evaluation research, presented in a clear and cogent way. The findings from such research may be complex, but they need not be obscure or enigmatic. On the contrary, clear evaluation reports must be true stories, well told.
I recently participated in a workshop at Brandeis University for graduate students who were considering non-academic careers in the social sciences. During the workshop, one of the students asked about the difference between program evaluation and other kinds of social research. This is a valuable and important question to which I responded that program evaluation is a type of applied social research that is conducted with “a value, or set of values, in its denominator.” I further explained that I meant that evaluation research is always conducted with an eye to whether the outcomes, or results, of a program were achieved, especially when these outcomes are compared to a desired and valued standard or criterion. At the heart of program evaluation is the idea that outcomes, or changes, are valuable and desired. Evaluators conduct evaluation research to find out if these valuable changes (often expressed as program goals or objectives) are, in fact, achieved by the program.
Evaluation research shares many of the same methods and approaches as other social sciences, and indeed, natural sciences. Evaluators draw upon a range of evaluation designs (e.g. experimental design, quasi-experimental desing, non-experimental design) and a range of methodologies (e.g. case studies, observational studies, interviews, etc.) to learn what the effects of a given intervention have been. Did, for example, 8th grade students who received an enriched STEM curriculum do better on tests, than did their otherwise similar peers who didn’t receive the enriched curriculum? Do homeless women who receive career readiness workshops succeed at obtaining employment than do other similar homeless women who don’t participate in such workshops? (For more on these types of outcome evaluations, see our previous blog post, “What You Need to Know About Outcome Evaluations: The Basics,”) While not all program evaluations are outcome evaluations, all evaluations gather systematic data with which judgments about the program can be made.
Evaluation’s Differences From Other Kinds of Social Research
Evaluation research is distinct from other forms of applied social research in so far as it:
- seeks to determine the merit, value, and/or worth of a program’s activities and results.
- entails the systematic collection of empirical data that is used to measure the processes and/or outcomes of a program, with the goal of furthering the program’s development and improvement.
- provides actionable information for decision-makers and program stakeholders, so that, based on objective data, a program can be strengthened or curtailed.
- focuses on particular knowledge (usually about a program and its outcomes), rather than seeks widely generalizable and universal knowledge.
While evaluators share many of the same methods and approaches as other researchers, program evaluators must employ an explicit set of values against which to judge the findings of their empirical research. The means that evaluators must both be competent social scientists and exercise value-based judgments and interpretations about the meaning of data.
Research vs. Evaluation
Differences Between Research and Evaluation
Harvard Family Research Project’s “Ask an Expert” series.
See “Michael Scriven on the Differences Between Evaluation and Social Science Research,”
Office of Educational Assessment
Sandra Mathison’s “What is the Difference Between Evaluation and Research, and Why Do We Care?”
“Distinguishing Evaluation from Research”
In her book Evaluation (2nd Edition) Carol Weiss writes, “Outcomes define what the program intends to achieve.” (p.117) Outcomes are the results or changes that occur, either in individual participants, or targeted communities. Outcomes occur because a program marshals resources and mobilizes human effort to address a specified social problem. Outcomes, then, are what the program is all about; they are the reason the program exists.
In order to assess which outcomes are achieved, program evaluators design and conduct outcome evaluations. These evaluations are intended to indicate, or measure, the kinds and levels of change that occur for those affected by the program or treatment. “Outcome evaluations measure how clients and their circumstances change, and whether the treatment experience (or program) has been a factor in causing this change. In other words, outcome evaluations aim to assess treatment effectiveness. (World Health Organization)
Outcome evaluations, like other kinds of evaluations, may employ a logic model, or theory of change, which can help evaluators and their clients to identify the short-, medium-, and long-term changes that a program seeks to produce. (See our blog post “Using a Logic Model” ) Once intended changes are identified in the logic model, it is critical for the evaluator to further identify valid and effective measures of said changes, so that these changes are correctly documented. It is preferable to identify desired outcomes before the program begins operation, so that these outcomes can be tracked throughout the program’s life-span.
Most outcome evaluations employ instruments that contain measures of attitudes, behaviors, values, knowledge, and skills. Such instruments may be either standardized, often validated, or they may be uniquely designed, special- purpose instruments, (e.g., a survey designed specifically for this particular program.) Additionally, the measures contained in an instrument can be either “objective,” i.e., they don’t rely on individuals’ self-reports, or conversely, they can be “subjective,” i.e., based on informants’ self-estimates of effect. Ideally, outcome evaluations try to use objective measures, whenever possible. In many instances, however, it is desirable to use instruments that rely on participants’ self-reported changes and reports of program benefits.
It is important to note that outcomes (i.e. changes or results) can occur at different points in time in the life span of a program. Although outcome evaluations are often associated with “summative,” or end-of-program-cycle evaluations, because program outcomes can occur in the early or middle stages of a program’s operation, outcomes may be measured before the final stage of the program. It may even be useful for some evaluations to look at both short- and long-term outcomes, and therefore to be implemented at different points in time (i.e., early and late.)
Another issue relevant to outcome evaluation is dealing with unintended outcomes of a program. As you know, programs can have a range of intended goals. Some outcomes or results, however, may not be a part of the intended goals of the program. They nonetheless occur. It is critical for evaluations to try to capture the unintended consequences of programs’ operation as well as the intended outcomes.
Ultimately, outcome evaluations are the way that evaluators and their clients know if the program is making a difference, which differences it’s making, and if the differences it’s making, are the result of the program.
World Health Organization, Workbook 7
Measuring Program Outcomes: A Practical Approach (1996) United Way of America’s
Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources
Evaluation Methodology Basics, The Nuts and Bolts of Sound Evaluation,
E. Jane Davidson, Sage, 2005.
Evaluation (2nd Edition) Carol Weiss, Prentice Hall, 1998.
Pioneered by market researchers and mid-20th century sociologists, focus groups are a qualitative research method that involves small groups of people in guided discussions about their attitudes, beliefs, experiences, and opinions about a selected topic or issue. Often used by marketers who obtain feedback from consumers about a product or service, focus groups have also become an effective and widely recognized social science research tool that enables researchers to explore participants’ views, and to reveal rich data that often remain under-reported by other kinds of data collection strategies (e.g., surveys, questionnaires, etc. ).
Organized around a set of guiding questions, focus groups typically are composed of 6-10 people and a moderator who poses open-ended questions that allow participants to address questions. Focus groups usually include people who are somewhat similar in characteristics or social roles. Participants are selected for their knowledge, reflectiveness, and willingness to engage topics or questions. Ideally—although not always possible—it is best to involve participants who don’t previously know one another.
Focus group conversations enable participants to offer observations, define issues, pose and refine questions, and create informative debate/discussions. Focus group moderators must: be attentive, pose useful and creative questions, create a welcoming and non-judgmental atmosphere, be sensitive to non-verbal cues and the emotional tenor of participants. Typically, focus group sessions are recorded or videoed so that researchers can later transcribe and analyze participants’ comments. Often an assistant moderator will take notes during the focus group conversation.
Focus groups have advantages over other date collection methods. They often employ group dynamics that help to reveal information that would not emerge from an individual interview or survey: they produce relatively quick, low cost data (they produce an ‘economy of scale’ as compared to individual interviews); allow the moderator to pose appropriate and responsive follow-up questions; enable the moderator to observe non-verbal data; and often produce greater and richer data than a questionnaire or survey.
Focus groups also can have some disadvantages, especially if not conducted by an experienced and skilled moderator: Depending upon their composition, focus groups are not necessarily representative of the general population; respondents may feel social pressure to endorse other group members’ opinions or refrain from voicing their own; group discussions require effective “steering” so that key questions are answered, and participants don’t stray from the questions/topic.
Focus groups are often used in program evaluations. I have had extensive experience conducting focus groups with a wide-range of constituencies. During my 20 years of experience as a program evaluator, I’ve moderated focus groups composed of: homeless persons; disadvantaged youth; university professors and administrators; k-12 teachers; k-12 and university students, corporate managers; and hospital administrators. In each of these groups I’ve found that it’s been beneficial to: have a non-judgmental attitude, be genuinely curious; exercise a gentle guidance; and respect the opinions, beliefs, and experiences of each focus group member. A sense of humor can also be extremely helpful. (See our previous post: “Interpersonal Skills Enhance Program Evaluation,” Also “Listening to Those Who Matter Most, the Beneficiaries” )
About focus groups:
About focus groups:
How focus groups work:
Focus group interviewing:
‘Focus groups’ at Wikipedia
A needs assessment is a systematic research and planning process for determining the discrepancy between an actual condition or state of affairs, and a future desired condition or state of affairs. Needs assessments are undertaken not only to identify the gap between “what is” and “what should be” but also to identify the programmatic actions and resources that are required to address that gap. Typically, a needs assessment is a part of planning processes that is intended to yield improvements in individuals, education/training, organizations, and/ or communities. (https://en.wikipedia.org/wiki/Needs_assessment ) Ultimately, a needs assessment is “a systematic process whose aim is to acquire an accurate, thorough picture of a system’s strengths and weaknesses, in order to improve it and to meet existing and future challenges.”
( http://dictionary.reference.com/browse/needs+assessment) Needs assessments have a variety of purposes. They can be used to identify and address challenges in a community, to develop training strategies, or to improve the performance of organizations.
There are a variety of conceptual modules of needs assessments. (For a review of various models (See http://ryanrwatkins.com/na/namodels.html ) One of the most popular is the SWOT analysis, in which researchers and action teams conduct a study to determine the strengths, weaknesses, opportunities, and threats involved in a project or business venture. In Planning and Conducting Needs Assessments: A Practical Guide. Thousand Oaks, CA. Sage Publications. (1995) Witkin, and Altschuld, identify a three-stage model of needs assessment, which includes pre-assessment (exploration), assessment (data gathering), and post-assessment (utilization).
Although there are various approaches to needs assessment, most include the following essential components/steps:
- Identify issue/concern
- Conduct a gap analysis (where things are now vs. where they should be)
- Specify methods for collecting information/data
- Perform literature review
- Collect and analyze data
- Develop action plan
- Produce implementation report
- Disseminate report/recommendations to stakeholders.
Why Conduct a Needs Assessment
Needs assessments can be used to identify real-world challenges, to formulate plans to correct inequities, and to involve critical stakeholders in building consensus and mobilizing resources to address identified challenges. For non-profit organizations needs assessments: 1) use data to identify an unaddressed or under-addressed need; 2) help to more effectively utilize resources to address a given problem; 3) make programs measurable, defensible, and fundable; and 4) inform, mobilize, and re-energize stakeholders. Needs assessments can be used with an organizations internal and external stakeholders and constituents.
Brad Rose Consulting Inc. has extensive experience in designing and implementing needs assessments for non-profit organizations, educational institutions, and health and human service programs. We’d welcome a chance to speak with you and your colleagues about how we can help you to conduct a needs assessment
Pyramid Model of Needs Assessment
Needs Assessment: Strategies for Community Groups and Organizations
Needs Assessment 101
U.S. Department of Education
Needs Assessment: A User’s Guide