What is an “Outcome”
“Outcomes,” i.e., specific changes or results, are what programs seek to produce, and what funders seek to fund. Programs and initiatives, especially in the nonprofit sector, exist to produce valuable and desired changes. While the measurement of outcomes is essential to evaluation, a key question for both programs and for evaluators is, ‘What counts as a desired outcome?”
Typically, education and nonprofit programs strive to see improvements in things like’s students’ reading scores, homeless persons employability, young people’s job-readiness, availability of affordable housing for home-seekers, etc. While the desirability of such outcomes appear “natural” or taken-for-granted, outcomes are, in fact, “agreed upon” events, states, or entities.
Who determines the outcomes that are to be evaluated?
For those who administer and run programs, it is often the case that such programs are highly dependent upon what funders value and desire to see changed. Although desired changes are usually viewed as self-evident, outcomes are a socially and politically defined entity. They depend upon a negotiated understanding of what constitutes a valuable change, and how is such change should be measured or indicated. Additionally, as many program leaders and educators will attest, changes that are desirable, are often shaped by the ability of proponents and researchers to measure such changes. (I’m reminded here, of Einstein’s famous quote: “Not everything that counts can be measured, and not everything that can be measured counts.”) As Heather Douglas points out, “ We must value something to find it significant enough to measure, to pluck it from the complexity of human social life, and to see it as a set of phenomena worthy of study.” (See Heather Douglas, “Facts, Values, and Objectivity”).
Of course, funders alone don’t determine the outcomes that programs produce. Program stakeholders can have a significant influence on what constitutes a desired outcome (See our previous blogpost, “The Importance of Understanding Stakeholders”). In an educational program, for example, a wide range of stakeholders may influence the desired outcomes of programming. Parents, teachers, community members, state and federal policy makers, business interests, politicians—may all influence what counts as a desirable outcome of education. Therefore, what stakeholders value (i.e., view as significant), is often what is viewed as the desired outcome of a program.
It’s important for program leaders and staff, and for evaluators to discuss and identify which changes programming seeks to affect. While evaluators deploy a range of methods to indicate or measure such changes, what counts as a desirable change, as a desirable “outcome,” is a question as critical to the success an evaluation as it is to the success of a program.
The Measure of Reality: Quantification and Western Society 1250-1600, Alfred W. Crosby. Cambridge University Press, 1997.
For a radical critique on the power of the funders of nonprofits to exercise influence on nonprofits goals and outcomes, see “How Liberal Nonprofits Are Failing Those They’re Supposed To Protect,” by William Anderson
Every program evaluation is conducted in a context in which there are parties (persons, organizations, etc.) who have an interest, or a “stake,” in the operation and success of the program. In the corporate world, a “stakeholder” is any member of the “groups without whose support the organization would cease to exist.” (see Corporate Stakeholder). More recently, the idea of “a stakeholder” has been broadened to include “any group or individual who is affected by, or who can affect the achievement of, an organization’s objectives.” (The Stakeholder Theory Summary.)
Indeed, in the not-for-profit world, stakeholders may include an array of persons and organizations including funders, community members, program participants, family members, volunteers, staff, government agencies, and the broader public.
Stakeholders in non-profits usually fall into one of three categories of legal statuses:
- Constitutional stakeholders such as board members or trustees of the non-profit organization
- Contractual stakeholders, including paid staff, or any business, group or individual that has a formal relationship with the organization.
- Third-party stakeholders including all the people and groups that may be affected by what the organization does. That includes businesses, the local government, and the citizens who live in the community. (See What is a Stakeholder of a Non-profit.)
Nonprofit stakeholders may range from those who support an organization, to those who oppose an organization. Stakeholder can include advocates, supporters, critics, competitors, and opponents. In its analysis of stakeholders in policy change efforts, the World Bank uses the categories of “promoter,” “defender,” “latents”, and “apathetics.” (See What is a Stakeholder Analysis.)
Conducting a stakeholder analysis is very useful for both evaluation and strategic planning efforts. Identifying various stakeholders’ interests in an organization’s mission and programming can help non-profit leaders and staff to be sure that their efforts and initiatives are achieving desired goals. They can also be useful in ensuring that the needs of those served are being directly met. For both evaluation and strategic planning purposes, a stakeholder analysis is an important process for achieving a shared understanding of each stakeholder’s specific interest in, and relevance to, the work of the non-profit or educational organization.
Brad Rose Consulting has developed a unique approach to stakeholder analysis, one that can be extremely useful as organizations examine the purposes, goals, specific activities, and desired outcomes of their work. We often work with organizations to implement stakeholder analyses. These analyses are helpful both in identifying where an organization is, at any given point in time, and for identifying where it wants to go in the future. You can see our basic stakeholder analysis form here.
Surveys can be an efficient way to collect information from a substantial number of people (i.e., respondents) in order to answer evaluation research questions. Typically, surveys strive to collect information from a sample (portion) of a broader population. When the sample is selected via random selection of respondents from a specified sampling frame, findings can be confidently generalized to the entire population.
Surveys may be conducted by phone, in-person, on the web, or by mail. They may ask standardized questions so that each respondent replies to precisely the same inquiry. Like other forms of research, highly effective surveys depend upon the quality of the questions asked of respondents. The more specific and clear the questions, the more useful survey findings are likely to be. Good surveys present questions in a logical order, are simple and direct, ask about one idea at a time, and are brief.
Surveys can ask either closed-ended or open-ended questions. Closed-ended questions may include multiple choice, dichotomous, Lickert scales, rank order scales, and other types of questions for which there are only a few answer categories available to the respondent. Closed-ended questions provide easily quantifiable data, for example, the frequency and percentage of respondents who answer a question in a particular way.
Alternatively, open-ended survey questions provide narrative responses that constitute a form of qualitative data. They require respondent reflection on their experience or attitudes. Open-ended questions often begin with: “why,” “how,” “what,” “describe,” “tell me about…,” or “what do you think about…”(See Open Ended Questions.) Open-ended survey questions depend heavily upon the interest, enthusiasm, and literacy level of respondents, and require extensive analysis precisely because they are not comprised of a small number of response categories.
Administering Surveys and Analyzing Results
Surveys can be administered in a variety of ways; in-person, on the phone, via mail, via the web, etc. Regardless of specific venue, it’s important to consider from the point of view of the respondent the factors that will maximize respondents’ participation, including accessibility of the survey, convenience of format, logicality of organization, and clarity of both the survey’s purpose and its questions.
Once survey data are collected and compiled, analyses of the data may take a variety of forms. Analysis of survey data essentially entails looking at quantitative data to find relationships, patterns, and trends. “Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc…That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting. It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. (See Community Tool Box “Collecting and Analyzing Data”) While data analysis usually entails some kind of statistical/quantitative manipulation of numerical information, it may also entail the analysis of qualitative data, i.e., data that is usually composed of words and not immediately quantifiable (e.g., data from in-depth interviews, observations, written documents, video, etc.)
The analysis of both quantitative and qualitative survey data (the latter typically collected in surveys from open-ended questions) is performed primarily to answer key evaluation research questions like, “Did the program make a difference for participants?” Effectively reporting findings from survey research not only entails accurate representation of quantitative findings, but interpretation of what both quantitative and qualitative data mean. This requires telling a coherent and evocative story, based on the survey data.
Brad Rose Consulting has over two decades of experience designing and conducting surveys whose findings compose an essential component of program evaluation activities. The resources below provide additional sources of information on the basics of survey research.
A recent issue of New Directions in Evaluation, (No. 152, Winter, 2016) “Social Experiments in Practice: The What, Why, When, Where, and How of Experimental Design and Analyses,” is devoted to the use of randomized experiments in program evaluation. The eight articles in this thematic volume discuss different aspects of experimental design—the practical and theoretical benefits and challenges of applying randomized controlled trials (RCTs) to the evaluation of programs. Although it’s beyond the scope of this blogpost to discuss each of the articles in detail, I’d like to mention a few insights offered by the authors and review the advantages and challenges of experimental design.
Random assignment helps rule out alternative explanations for outcomes
Experimental design in the social sciences, are studies that randomly assign subjects (i.e., program participants) to treatment and control groups, then measure changes (i.e., average changes) in both groups to determine if a program, or “treatment,” has had a desired effect on those who receive the treatment. As the issue’s editor, Laura Peck, observes, “…when it comes to the question of cause and effect—the question of a program’s or policy’s impact, we assert that a randomized experiment should be the evaluation design of choice.” (p.11) Indeed, experimental design studies—whose origins are in the natural sciences, and whose benefits are perhaps most frequently demonstrated in FDA testing of pharmaceuticals—is thought to be the “gold standard” for scientifically establishing causation. Random assignment of individuals to two groups—one that receives treatment and one that does not receive treatment—is the best way to establish whether desired changes are the result of what happens in the treatment (i.e., program). As the editor observes, “This ‘coin toss’ (i.e., random assignment) to allocate access to treatment carries substantial power. It allows us to rule out alternative explanations for differences in outcomes between people who have access to a service and people who do not.” (p.11)
There are still concerns surrounding the use of experimental design
Although experimental design is viewed by many as the premier indicator of causation, it’s use in evaluations can have practical challenges. There are potentially legal and ethical concerns about non-treatment for control groups (especially in the fields of medicine and education). Additionally, some argue that experimental design, especially in complex social interventions, is unable to identify which specific component of a treatment is responsible for the observed differences in the treatment group (the “black box” phenomenon.). Michael Scriven observes that it is nearly impossible to create a truly “double blind” experiment in the social world (i.e. experiments where neither experimental subject nor the evaluator knows who is in the treatment who is in the control groups). Moreover, some argue that experimental design can be more labor and time-intensive than other study designs, and therefore, more costly.
Quasi- experimental design is useful for showing before and after changes
While experimental design is the most prestigious method for determining the causal effects of a program, initiative, or policy, it is far from a universally appropriate design for evaluations. Quasi-experimental design, for example, is often used to show pre- and post- changes in those who participate in a program or treatment, although quasi-experimental design is unable to unequivocally confirm whether such changes are attributable to the program. One form of a quasi-experimental design is the “non-equivalent (pre-test, post-test) control group design”. In this design, participants are assigned to two groups (although not randomly assigned.) Both groups take a pre-test and a post-test, but only one group, the experimental group, receives the treatment/program. (The key textbook resource on both experimental and non-experimental designs is Experimental and Quasi-Experimental Designs, by Shadish, Cook, and Campbell, Houghton Mifflin.)
There are, of course, a range of non-experimental designs that are used productively in evaluation. These range from case studies to observational studies, and rely on a variety of methods, largely qualitative, including phone and in-person interviews, focus groups, surveys, and document reviews. (See this page for a brief table comparing the characteristics of qualitative and quantitative methods of research. See also the National Science Foundation’s very helpful, “Overview of Qualitative Methods and Analytic Techniques”) Qualitative evaluation studies can be very effective, and are often used in a mixed methods approach to evaluation work.
In July, we posted a blog post titled, “Humans Need Not Apply: What Happens When There’s No More Work?”As we mentioned in that post, the rise of artificial intelligence, machine learning, and robotics, have increasingly ominous implications for the future of work and employment. In a recent New York Times article, “The Long-Term Jobs Killer Is Not China. It’s Automation,” Claire Caine Miller traces the effects of automation on those who have been employed in America’s once preeminent industries—steel, coal, newspapers, etc. She observes that it is neither immigration nor globalization that threatens American workers; it’s automation. Referring to the recent 2016 political campaigns Caine Miller notes, “No candidate talked much about automation on the campaign trail. Technology is not as convenient a villain as China or Mexico, there is no clear way to stop it, and many of the technology companies are in the United States and benefit the country in many ways.” She quotes one study that shows that roughly 13 percent of manufacturing job losses are due to trade, and the rest are due to enhanced productivity attributable to automation.
In another article, “Evidence That Robots Are Winning the Race for American Jobs,” Caine Miller writes, “The industry most affected by automation is manufacturing. For every robot per thousand workers, up to six workers lost their jobs and wages fell by as much as three-fourths of a percent, according to a new paper by the economists, Daron Acemoglu of M.I.T. and Pascual Restrepo of Boston University.” In “How to Make America’s Robots Great Again” Farhad Manjoo, (New York Times, January 25, 2017) states, “Thanks to automation, we now make 85 percent more goods than we did in 1987, but with only two-thirds the number of workers.”
Manufacturing however, is not the only area where AI and robots threaten to displace human employees. In “A Robot May Be Training to Do Your Job. Don’t Panic,” Alexandra Levit argues that automation in the form of “social robotics,’ affective computing, and emotional awareness software, are now making inroads into the helping/caring professions, like nursing. Levit writes, “Eventually, the moment will come when machines possess empathy, the ability to innovate and other traits we perceive as uniquely human. What then? How will we sustain our own career relevance?”
In “Actors, teachers, therapists – think your job is safe from artificial intelligence? Think again,” Dan Tynan writes, “A January, 2017 report from the McKinsey Global Institute estimated that roughly half of today’s work activities could be automated by 2055, (give or take 20 years.)…Thanks to advances in artificial intelligence, natural language processing, and inexpensive computing power, jobs that once weren’t considered good candidates for automation suddenly are.”
According to these and other writers, automation and Artificial Intelligence are poised to sweep away or profoundly transform a number of occupations, and thereby alter both industry and society. While some writers foresee productive partnerships between AI and human colleagues, others warn that automation is likely to reduce the needs for human labor, and relegate sectors of the population to hard scrabble redundancy. As Martin Ford points out, this industrial revolution is different than previous ones, because new technology is taking aim at both blue and white collar work. (See Rise of the Robots: Technology and the Threat of a Jobless Future, by Martin Ford.)
Lest we become disconsolate at the prospect that robots will take our jobs, Claire Caine Miller suggests that there are a number of things that the US can do to prepare and adapt to these employment- threatening developments. She suggests that: 1) the US provide more and different kinds of education to employees, including teaching technical skills, like coding and statistics, and skills that still give humans an edge over machines, like creativity and collaboration; 2) creating better jobs for human workers including government subsidized employment (creating public sector jobs) and building infrastructure; 3) creating more care-giving jobs, strengthening labor unions, and training some workers to work in advanced manufacturing; 4)expanding the earned-income tax credit, providing a universal basic income, in which the government gives everyone a guaranteed amount of money, and establishing “ portable benefits” that wouldn’t be tied to a job to get health insurance. Caine Miller also suggests raising the minimum wage and even taxing robots (the latter, a proposal supported by Bill Gates.)
Whether these proposals will prove to be politically feasible or economically viable is difficult to judge. Some of these seem wildly utopian and difficult to envision—especially given a new Administration that built substantial electoral support on promises to revive employment in ‘smokestack’ industries, like steel and coal. That said, until relatively recently, it was difficult to envision the meteoric “rise of the robots” and the consequent effects on employment and society that such a development would have. Not even robots can reliably predict the future.
“The Long-Term Jobs Killer Is Not China. It’s Automation.” Claire Caine Miller, New York Times, December 12, 2016
“Where machines could replace humans—and where they can’t (yet),” Michael Chui, James Manyika, and Mehdi Miremadi, July 2016, McKinsey Quarterly,
“Actors, teachers, therapists – think your job is safe from artificial intelligence? Think again.” Dan Tynan, The Guardian, February 9, 2017.
“How to Make America’s Robots Great Again” Farhad Manjoo New York Times, January 25, 2017
“Evidence That Robots Are Winning the Race for American Jobs,” Claire Cain Miller New York Times, March 28, 2017
“A Robot May Be Training to Do Your Job. Don’t Panic.” Alexandra Levit, New York Times September. 10, 2016
“EU supports Personhood status to robots.” Alex Hern, The Guardian, January 12, 2017
Rise of the Robots: Technology and the Threat of a Jobless Future, Martin Ford, Basic Books, 2015″