A recent issue of New Directions in Evaluation, (No. 152, Winter, 2016) “Social Experiments in Practice: The What, Why, When, Where, and How of Experimental Design and Analyses,” is devoted to the use of randomized experiments in program evaluation. The eight articles in this thematic volume discuss different aspects of experimental design—the practical and theoretical benefits and challenges of applying randomized controlled trials (RCTs) to the evaluation of programs. Although it’s beyond the scope of this blogpost to discuss each of the articles in detail, I’d like to mention a few insights offered by the authors and review the advantages and challenges of experimental design.
Random assignment helps rule out alternative explanations for outcomes
Experimental design in the social sciences, are studies that randomly assign subjects (i.e., program participants) to treatment and control groups, then measure changes (i.e., average changes) in both groups to determine if a program, or “treatment,” has had a desired effect on those who receive the treatment. As the issue’s editor, Laura Peck, observes, “…when it comes to the question of cause and effect—the question of a program’s or policy’s impact, we assert that a randomized experiment should be the evaluation design of choice.” (p.11) Indeed, experimental design studies—whose origins are in the natural sciences, and whose benefits are perhaps most frequently demonstrated in FDA testing of pharmaceuticals—is thought to be the “gold standard” for scientifically establishing causation. Random assignment of individuals to two groups—one that receives treatment and one that does not receive treatment—is the best way to establish whether desired changes are the result of what happens in the treatment (i.e., program). As the editor observes, “This ‘coin toss’ (i.e., random assignment) to allocate access to treatment carries substantial power. It allows us to rule out alternative explanations for differences in outcomes between people who have access to a service and people who do not.” (p.11)
There are still concerns surrounding the use of experimental design
Although experimental design is viewed by many as the premier indicator of causation, it’s use in evaluations can have practical challenges. There are potentially legal and ethical concerns about non-treatment for control groups (especially in the fields of medicine and education). Additionally, some argue that experimental design, especially in complex social interventions, is unable to identify which specific component of a treatment is responsible for the observed differences in the treatment group (the “black box” phenomenon.). Michael Scriven observes that it is nearly impossible to create a truly “double blind” experiment in the social world (i.e. experiments where neither experimental subject nor the evaluator knows who is in the treatment who is in the control groups). Moreover, some argue that experimental design can be more labor and time-intensive than other study designs, and therefore, more costly.
Quasi- experimental design is useful for showing before and after changes
While experimental design is the most prestigious method for determining the causal effects of a program, initiative, or policy, it is far from a universally appropriate design for evaluations. Quasi-experimental design, for example, is often used to show pre- and post- changes in those who participate in a program or treatment, although quasi-experimental design is unable to unequivocally confirm whether such changes are attributable to the program. One form of a quasi-experimental design is the “non-equivalent (pre-test, post-test) control group design”. In this design, participants are assigned to two groups (although not randomly assigned.) Both groups take a pre-test and a post-test, but only one group, the experimental group, receives the treatment/program. (The key textbook resource on both experimental and non-experimental designs is Experimental and Quasi-Experimental Designs, by Shadish, Cook, and Campbell, Houghton Mifflin.)
There are, of course, a range of non-experimental designs that are used productively in evaluation. These range from case studies to observational studies, and rely on a variety of methods, largely qualitative, including phone and in-person interviews, focus groups, surveys, and document reviews. (See this page for a brief table comparing the characteristics of qualitative and quantitative methods of research. See also the National Science Foundation’s very helpful, “Overview of Qualitative Methods and Analytic Techniques”) Qualitative evaluation studies can be very effective, and are often used in a mixed methods approach to evaluation work.
In July, we posted a blog post titled, “Humans Need Not Apply: What Happens When There’s No More Work?”As we mentioned in that post, the rise of artificial intelligence, machine learning, and robotics, have increasingly ominous implications for the future of work and employment. In a recent New York Times article, “The Long-Term Jobs Killer Is Not China. It’s Automation,” Claire Caine Miller traces the effects of automation on those who have been employed in America’s once preeminent industries—steel, coal, newspapers, etc. She observes that it is neither immigration nor globalization that threatens American workers; it’s automation. Referring to the recent 2016 political campaigns Caine Miller notes, “No candidate talked much about automation on the campaign trail. Technology is not as convenient a villain as China or Mexico, there is no clear way to stop it, and many of the technology companies are in the United States and benefit the country in many ways.” She quotes one study that shows that roughly 13 percent of manufacturing job losses are due to trade, and the rest are due to enhanced productivity attributable to automation.
In another article, “Evidence That Robots Are Winning the Race for American Jobs,” Caine Miller writes, “The industry most affected by automation is manufacturing. For every robot per thousand workers, up to six workers lost their jobs and wages fell by as much as three-fourths of a percent, according to a new paper by the economists, Daron Acemoglu of M.I.T. and Pascual Restrepo of Boston University.” In “How to Make America’s Robots Great Again” Farhad Manjoo, (New York Times, January 25, 2017) states, “Thanks to automation, we now make 85 percent more goods than we did in 1987, but with only two-thirds the number of workers.”
Manufacturing however, is not the only area where AI and robots threaten to displace human employees. In “A Robot May Be Training to Do Your Job. Don’t Panic,” Alexandra Levit argues that automation in the form of “social robotics,’ affective computing, and emotional awareness software, are now making inroads into the helping/caring professions, like nursing. Levit writes, “Eventually, the moment will come when machines possess empathy, the ability to innovate and other traits we perceive as uniquely human. What then? How will we sustain our own career relevance?”
In “Actors, teachers, therapists – think your job is safe from artificial intelligence? Think again,” Dan Tynan writes, “A January, 2017 report from the McKinsey Global Institute estimated that roughly half of today’s work activities could be automated by 2055, (give or take 20 years.)…Thanks to advances in artificial intelligence, natural language processing, and inexpensive computing power, jobs that once weren’t considered good candidates for automation suddenly are.”
According to these and other writers, automation and Artificial Intelligence are poised to sweep away or profoundly transform a number of occupations, and thereby alter both industry and society. While some writers foresee productive partnerships between AI and human colleagues, others warn that automation is likely to reduce the needs for human labor, and relegate sectors of the population to hard scrabble redundancy. As Martin Ford points out, this industrial revolution is different than previous ones, because new technology is taking aim at both blue and white collar work. (See Rise of the Robots: Technology and the Threat of a Jobless Future, by Martin Ford.)
Lest we become disconsolate at the prospect that robots will take our jobs, Claire Caine Miller suggests that there are a number of things that the US can do to prepare and adapt to these employment- threatening developments. She suggests that: 1) the US provide more and different kinds of education to employees, including teaching technical skills, like coding and statistics, and skills that still give humans an edge over machines, like creativity and collaboration; 2) creating better jobs for human workers including government subsidized employment (creating public sector jobs) and building infrastructure; 3) creating more care-giving jobs, strengthening labor unions, and training some workers to work in advanced manufacturing; 4)expanding the earned-income tax credit, providing a universal basic income, in which the government gives everyone a guaranteed amount of money, and establishing “ portable benefits” that wouldn’t be tied to a job to get health insurance. Caine Miller also suggests raising the minimum wage and even taxing robots (the latter, a proposal supported by Bill Gates.)
Whether these proposals will prove to be politically feasible or economically viable is difficult to judge. Some of these seem wildly utopian and difficult to envision—especially given a new Administration that built substantial electoral support on promises to revive employment in ‘smokestack’ industries, like steel and coal. That said, until relatively recently, it was difficult to envision the meteoric “rise of the robots” and the consequent effects on employment and society that such a development would have. Not even robots can reliably predict the future.
“The Long-Term Jobs Killer Is Not China. It’s Automation.” Claire Caine Miller, New York Times, December 12, 2016
“Where machines could replace humans—and where they can’t (yet),” Michael Chui, James Manyika, and Mehdi Miremadi, July 2016, McKinsey Quarterly,
“Actors, teachers, therapists – think your job is safe from artificial intelligence? Think again.” Dan Tynan, The Guardian, February 9, 2017.
“How to Make America’s Robots Great Again” Farhad Manjoo New York Times, January 25, 2017
“Evidence That Robots Are Winning the Race for American Jobs,” Claire Cain Miller New York Times, March 28, 2017
“A Robot May Be Training to Do Your Job. Don’t Panic.” Alexandra Levit, New York Times September. 10, 2016
“EU supports Personhood status to robots.” Alex Hern, The Guardian, January 12, 2017
Rise of the Robots: Technology and the Threat of a Jobless Future, Martin Ford, Basic Books, 2015″
In our last blog post “The Implications of Public School Privatization,” I referred to an article by Diane Ravitch that recently appeared in The New York Review of Books. That article claimed that the school privatization movement is largely composed of social conservatives, corporations, and business-friendly foundations. In a recent article by Janet Reitman, in Rolling Stone, “Betsy DeVos’ Holy War,” the author argues that the movement to privatize public schools is also sponsored, at least in part, by those who, like Betsy DeVos, would prefer to de-secularize schools and create institutions that reflect market friendly Christian values.
Betsy DeVos, embodies a nexus of wealth and hyper conservative Christianity. Her goals include support of “school choice” (i.e., voucher systems that direct tax money for public schools toward private and parochial schools) and, according to Reitman, the promotion of the religious colonization of public education, and more broadly, American society. (See also, “Betsy DeVos Wants to Use America’s Schools to Build ‘God’s Kingdom’” Kristina Rizga, Mar/Apr 2017 Issue, Mother Jones.
DeVos, as is well documented, is not deeply acquainted with public education—neither she nor her children attended public schools; she has never served on a school board, nor been an educator. DeVos who hails from a wealthy, Calvinist, Western Michigan dynasty that includes among other resources, her husband’s multi-billion dollar Amway fortune, and her father’s auto parts fortune (among other profitable ventures, her father, Edgar Prince, invented the lighted, automobile sun visor) now finds herself at the helm of the federal Department of Education. She appears to be even less a friend of public education than she is familiar with it. DeVos has devoted a substantial part of her political and philanthropic career to advocating for the privatization of public schools, and her home sate of Michigan has the highest number of for-profit charter schools in the nation. To learn more about DeVos’ plans for public schools in America, you can read this intriguing article “Betsy DeVos’ Holy War,” The resources below offer additional insights into Secretary DeVos and her plans for public schools.
“Six astonishing things Betsy DeVos said — and refused to say — at her confirmation hearing” Valerie Strauss, Washington Post, January 18, 2016
“Education for Sale?” Linda Darling Hammond The Nation, March 27, 2017
“Betsy DeVos: Fighter for kids or destroyer of public schools?” Lori Higgins, Kathleen Gray, November 23, 2016, Detroit Free Press.
“The Betsy DeVos Hearing Was an Insult to Democracy” Charles Pierce, Esquire
“Betsy DeVos Wants to Use America’s Schools to Build ‘God’s Kingdom’” Kristina Rizga, Mar/Apr 2017 Issue, Mother Jones
“Why are Republicans so cruel to the poor? Paul Ryan’s profound hypocrisy stands for a deeper problem” Chauncey DeVega, March 23, 2017
In a recent review of two books, Education and the Commercial Mindset and School Choice: The End of Public Education, which appears in the December 8, 2016 New York Review of Books, Diane Ravitch, the former Assistant Secretary of Education during the George HW Bush presidency, discusses the implications of corporate designs on public education. Ravitch begins her review by reminding us that, “Privatization means that a public service is taken over by for-profit business whose highest goal is profit.” In the name of market-driven efficiency, she argues, the “education industry” is likely to become increasingly similar to privatized hospitals and prisons. In these industries, as in many others, corporate owners, in their loyalty to investors’ desire for profits, tend to eliminate unions, reduce employee benefits, continually cut costs of operation, and orient to serving those who are least expensive to serve.
Ravitch sketches some of the challenges posed by charter schools, noting that “…they can admit the students they want, exclude those they do not want, and push out the ones who do not meet their academic or behavioral standards.” She says, that charters not only “drain away resources from public schools” but they also “leave the neediest, most expensive students to the public schools to educate.” Moreover, as Josh Moon recently noted in his article, “’School choice’ is an awful choice,” “If the “failing school” is indeed so terrible that we’re willing to reroute tax money from it to a private institution that’s not even accredited, then what makes it OK for some students to attend that failing school?”
While some argue that charter schools can “save children from failing public schools” research on student outcomes for charter school has shown mixed results. For example, The Education Law Center, in “Charter School Achievement: Hype vs Evidence” reports:
Research on charter schools paints a mixed picture. A number of recent national studies have reached the same conclusion: charter schools do not, on average, show greater levels of student achievement, typically measured by standardized test scores, than public schools, and may even perform worse.
The Center for Research on Education Outcomes (CREDO) at Stanford University found in a 2009 report that 17% of charter schools outperformed their public school equivalents, while 37% of charter schools performed worse than regular local schools, and the rest were about the same. A 2010 study by Mathematica Policy Research found that, on average, charter middle schools that held lotteries were neither more nor less successful than regular middle schools in improving student achievement, behavior, or school progress. Among the charter schools considered in the study, more had statistically significant negative effects on student achievement than statistically significant positive effects. These findings are echoed in a number of other studies.
In Michigan, Secretary of Education, Betsy Devos’s home state, 80 percent of charter schools operate as for-profit organizations. Ravitch says, “They perform no better than public schools, and according to the Detroit Free Press, they make up a publicly subsidized $1 billion per year industry with no accountability.”
Ravitch tells us that the privatization movement is largely composed of social conservatives, corporations, and business-friendly foundations. “These days, those who call themselves “education reformers” are likely to be hedge fund managers, entrepreneurs, and billionaires, not educators. The “reform” movement loudly proclaims the failure of American public education and seeks to turn public dollars over to entrepreneurs, corporate chains, mom-and-pop operations, religious organizations, and almost anyone else who wants to open a school.”
The Trump administration is likely to further advance a public-school privatization and school voucher agenda. The extent and results of such “reforms” are hard to predict. That said, as Ravitch argues “… there is no evidence for the superiority of privatization in education. Privatization divides communities and diminishes commitment to that which we call the common good. When there is a public school system, citizens are obligated to pay taxes to support the education of all children in the community, even if they have no children in the schools themselves. We invest in public education because it is an investment in the future of society.” How continued privatization of public k-12 education will affect an increasingly economically privatized and socially and politically divided society is not yet known.
One of the most important questions to consider before starting an evaluation is, “How will evaluation findings be used?” There are a number of ways the findings from an evaluation can be used. These include:
- to improve the program (formative evaluation)
- to make judgements about the ultimate value/benefits of the program to participants and stakeholders (summative evaluation)
- to sustain and/or expand the program
- to document and publicize the program’s achievements (outreach and marketing)
- to curtail the program
By being clear about the intended uses of evaluation findings, organizations can choose the most effective and stakeholder-relevant evaluation design. If, for example, a program funder wants to know whether their investment yielded desired outcomes for program participants, it may not be useful to focus the evaluation on gathering data that primarily indicate ways to improve the program’s delivery. On the other hand, if an organization is curious about how its programming is being implemented, and if such implementation is in fidelity with a research-based program design, the evaluation will want to focus more on implementation processes, rather than ultimate program outcomes.
Needless to say, stipulating the use of evaluation findings is closely related to the question of who are the major audiences for the evaluation. See our previous post, Identifying Evaluation Stakeholders. By clarifying potential audiences for findings, organizations and evaluators can identify the most useful evaluation strategy and design. In the next blog post we will discuss a related question, i.e., “What information will stakeholders find useful and valuable?”