Program evaluations entail research. Research is a systematic “way of finding things out.” Evaluation research depends on the collection and analysis of data (i.e., evidence, facts) that indicate the outcomes (i.e., effects, results, etc.) of the operation of programs. Typically, evaluations want to discover evidence of whether valued outcomes have been achieved. (Other kinds of evaluations, like formative evaluations, seek to discover, through the collection and analysis of data, ways that a program may be strengthened.)
Data can be either qualitative (descriptive, entail words and observations) or quantitative (numerical). What counts as data depends upon the design and character of the evaluation research. Quantitative evaluations rely primarily on the collection of countable information like measurements and statistical data. Qualitative evaluations depend upon language-based data and other descriptive data. Usually, program evaluations combine the collection of quantitative and qualitative data.
There are a range of data sources for any evaluation. These can include: observations of programs’ operation; interviews with program participants, program staff, and program stakeholders; administrative records, files, and tracking information; questionnaires and surveys; focus groups; and visual information, such as video data and photographs.
The selection of the kinds of data to collect and the ways of collecting such data will be contingent on the evaluation design, the availability and accessibility of data, economic considerations about the cost of data collection, and both the limitations and potentials of each data source. The kinds of evaluation questions and the design of the evaluation research will, together, help to determine the optimal kinds of data that will need to be collected. (See our articles
“Questions Before Methods” and “Using Qualitative Interviews in Program Evaluations”)
Evaluation, Carole Weiss, Prentice Hall; 2nd edition (1997)
“Collaboration” and “teamwork” are the catchphrases of the contemporary workplace. Since the 1980s in the U.S., work teams have been hailed as the solution to assembly line workers’ alienation and disaffection, and white-collar workers’ isolation and disconnection. Work teams have been associated with increased productivity, innovation, employee satisfaction, and reduced turnover. Additionally, teams at work are said to have beneficial effects on employee learning, problem-solving, communication, company loyalty, and organizational cohesiveness. Teams are now found throughout the for-profit, non-profit, and governmental sectors, and much of the work of the field of organization development (OD) is devoted to fostering and sustaining teams at work.
In his recent article “Stop Wasting Money on Team Building,” Harvard Business Review, September 11, 2018, Carlos Valdes-Dapena, argues that teams are less effective than many believe them to be. Based on research conducted at Mars, Inc. “a 35 billion dollar global corporation with a commitment to collaboration,” Valdes-Dapena argues that while employees like the idea of teams and team work, employees don’t, in fact, much collaborate in teams. After conducting 125 interviews and administering questionnaires with team members, he writes “If there was one dominant theme from the interviews, it is summarized in this remarkable sentiment: “I really like and value my teammates. And I know we should collaborate more. We just don’t.”
Valdes-Dapena reports that employees “…felt the most clarity about their individual objectives, and felt a strong sense of ownership for the work they were accountable for.” He also shows that “Mars was full of people who loved to get busy on tasks and responsibilities that had their names next to them. It was work they could do exceedingly well, producing results without collaborating. On top of that, they were being affirmed for those results by their bosses and the performance rating system.” Essentially, Valdes-Dapena, argues, teams may sound good in theory, but it is probably better to tap individual self-interest, if you really want to get the job done.
In “3 Types of Dysfunctional Teams and How To Fix Them,” Patty McManus says that there are different types of dysfunctional work teams. She characterizes these different team types as: “The War Zone,” “The Love Fest,” and “The Unteam.” In “War Zone” teams, competition and factionalism among members obscure or derail the potential benefits of teamwork. In the “Love Fest” team, there is a focus on muting disagreements, highlighting areas of agreement, and avoidance of tough issues in the interest of maintaining good feelings. “The Unteam” is characterized by meetings that are used for top-down communication and status updates, and fail to build shared perspective about the organization. In the “Unteam” members may get along as individuals, but they have little connection to one another or a larger purpose they all share.
McManus claims that the problems of teams may be overcome by what she terms “ecosystems teams,” i.e., teams that surface and manage differences, build healthy inter-dependence among members, and engage the organization—beyond the mere confines of the team.
Matthew Swyers also sees problems in teams at work. In “7 Reasons Good Teams Become Dysfunctional,” (Inc. September 27, 2012,) Swyers writes that there are seven types of problems that teams may experience:
- absence of a strong and competent leader
- team members more interested in individual glory than achieving team objectives
- failure to define team goals and desired outcomes
- disproportionately place too much of the team’s work on a few of its members’ shoulders
- lack focus and endless debate, without moving toward an ultimate goal
- lack of accountability and postponed timetables
- failure of decisiveness.
Each of these writers highlight the vulnerabilities of teams at work. Although the work of these writers doesn’t foreclose the positive possibilities of team organization at work, they raise important questions about both the enthusiasm for, and the effectiveness of, teams. Additionally, each author suggests that with enlightened modifications, organizations can overcome the liabilities of teams and begin to reap the benefits of team-based employee collaboration. That said, none of these writers, and few among the other U.S. based writers who have engaged this topic, treat the underlying assumptions of workplace reform—that work can be made more habitable and humane without the independent organizations that have traditionally represented workers’/employees’ interests in the workplace. For discussion of models of workplace reform that genuinely represent workers’ interest in more humane, collaborative, and ultimately, productive working environments, we will need to look elsewhere.
“Importance of Teamwork at Work,” Tim Zimmer
“Importance of Teamwork in Organizations,” Aaron Marquis
“What Makes Teams Work?” Regina Fazio Maruca, Fast Company
“Stop Wasting Money on Team Building,” Carlos Valdes-Dapena, Harvard Business Review, September 11, 2018
“3 Types Of Dysfunctional Teams And How To Fix Them,” Patty McManus, Fast Company
“When Is Teamwork Really Necessary?” Michael D. Watkins, Harvard Business Review, August 16, 2018
“7 Reasons Good Teams Become Dysfunctional,” Matthew Swyers , Inc. Sept 27 2012
“Why Teams Don’t Work,” Diane Coutu, Harvard Business Review, May 2009
Brad recently presented a program evaluation workshop to grantees of the Foundation of MetroWest.
The two-hour workshop served to introduce grantees and other attendees to the basics of evaluation, including the:
- basic kinds and purposes of program evaluation
- types of questions that evaluations ask and answer
- ways that values and criteria inform an evaluation
- types of evaluation designs
- use of logic models
You can access the PowerPoint for the presentation below.
Rather than “customers,” nonprofits, educational institutions, and philanthropies typically have “stakeholders.” Stakeholders are individuals and organizations that have an interest in, and may be affected by, the activities, actions, and policies of non-profits, schools, and philanthropies. Stakeholders don’t just purchase products and services (i.e. commodities), they have an interest, or “stake” in the outcomes of an organization’s or program’s operation.
There are a number of persons or entities who may be a stakeholder in a nonprofit organization. Nonprofit stakeholders may include funders/sponsors, program participants, staff, communities, and government agencies. It’s important to note that stakeholders can be either internal or external to the organization, and that stakeholders are able to exert influence— either positive or negative — over the outcomes of the organization or program.
While many nonprofits are sensitive to, and aware of, the interests of their multiple stakeholders, quite often both nonprofit leaders and nonprofit staff hold implicit, unexamined ideas about who their various stakeholders are. Often, stakeholders are not delineated, and consequently, there isn’t a shared understanding of who is and isn’t a stakeholder. Conducting a stakeholder analysis can be a useful process because it raises awareness of staff and managers about who is interested in, and who potentially influences the success of an organization’s desired outcomes. A stakeholder analysis is a simple way to help nonprofits to clarify those who have a “stake” in the success of the organization and its discrete programs. It can sharpen strategic planning, clarify goals, and build consensus about an organization’s purpose.
Pablo Picasso once said, “It takes a long time to become young.” The same may be said about education and the process of becoming educated. While we often associate formal education with youth and early adulthood, the fact is that education is an increasingly recognized lifelong endeavor that occurs far beyond the confines of early adulthood and traditional educational institutions.
In a recent article “Lifetime Learner” by John Hagel III, John Seely Brown, Roy Mathew, Maggie Wooll & Wendy Tsu, The Atlantic the authors discuss the emergence of a rich and ever-expanding “ecosystem” of organizations and institutions that have arisen to serve the unmet educational needs and expectations of learners who are not enrolled in formal, traditional educational institutions (e.g. community colleges, colleges, and universities). “This ecosystem of semi-structured, unorthodox learning providers is emerging at “the edges” of the current postsecondary world, with innovations that challenge the structure and even the existence of traditional education institutions.”
Hagel III, et al. argue that economic forces together with emerging technologies are enabling learners to do an “end run” around traditional educational providers and to gain access to knowledge and information in new venues. The growing availability of, and access to, MOOCs (Massive Online Open Courses), Youtube, Open Educational Resources, and other online learning platforms enable more and more learners to advance their learning and career goals outside the purview of traditional post-secondary institutions.
While the availability of alternative, lifelong educational resources is helping some non-traditional students to advance their educational goals, it is also having an effect on traditional post-secondary institutions. Hagel III, Seely Brown, Wooll and Tsu, argue that, “The educational institutions that succeed and remain relevant in the future …will likely be those that foster a learning environment that reflects the networked ecosystem and (that will become) meaningful and relevant to the lifelong learner. This means providing learning opportunities that match the learner’s current development and stage of life.” The authors site as examples, community colleges that are now experimenting with “stackable” credentials that provide short-term skills and employment value, while enabling students to return over time and assemble a coherent curriculum that helps them progress toward career and personal goals” and “some universities (that) have started to look at the examples coming from both the edges of education and areas such as gaming and media to imagine and conduct experiments in what a future learning environment could look like.”
The authors say that in the future colleges and universities will benefit from considering such things as:
- Providing the facilities and locations for a variety of learning experiences, many of which will depend external sources for content
- Aggregating knowledge resources and connecting these resources with appropriate learners rather than acting as sole “vendors” of knowledge
- Acting as a lifelong “agents” for learners by helping learners to navigate a world of exponential change and an abundance of information
While these goals are ambitious, they highlight the remarkably changing terrain in continuing education. Educational “consumers” are increasingly likely to seek inexpensive and more accessible pathways to knowledge. As the authors point out, individuals’ lifelong learning needs are likely to continue to increase, so correspondingly, the pressures on traditional post-secondary education are likely to grow. Whether learners’ needs are more effectively addressed by re-orienting traditional post-secondary institutions or by the patchwork “ecosystem” of semi-structured, unorthodox learning-providers who inhabit what the authors of “Lifetime Learner” term “the edges” of the postsecondary world, is difficult to predict.
Lifelong learning, Wikipedia
“Lifetime Learner” by John Hagel III, John Seely Brown, Roy Mathew, Maggie Wooll & Wendy Tsu, The Atlantic
“The Third Education Revolution: Schools are moving toward a model of continuous, lifelong learning in order to meet the needs of today’s economy” by Jeffrey Selingo, The Atlantic, Mar 22, 2018
We’ve previously written about the rise of artificial intelligence and the current and anticipated effects of AI upon employment. (See links to previous blog posts, below) Two recent articles treat the effects of AI on the assessment of students and the hiring of employees.
In her recent article for NPR, “More States Opting To ‘Robo-Grade’ Student Essays By Computer” Tovia Smith discusses how so-called “robo-graders” (i.e., computer algorithms) are increasingly being used to grade students’ essays on state standardized tests. Smith reports that Utah and Ohio currently use computers to read and grade students’ essays and that soon, Massachusetts will follow suit. Peter Foltz, a research professor at the University of Colorado, Boulder observes, “We have artificial intelligence techniques which can judge anywhere from 50 to 100 features…We’ve done a number of studies to show that the (essay) scoring can be highly accurate.” Smith also notes that Utah, which once had humans review students’ essays after they had been graded by a machine, now relies on the machines almost exclusively. Cyndee Carter, assessment development coordinator for the Utah State Board of Education reports “…the state began very cautiously, at first making sure every machine-graded essay was also read by a real person. But…the computer scoring has proven “spot-on” and Utah now lets machines be the sole judge of the vast majority of essays.”
Needless to say, despite support for “robo-graders”, there are critics of automated essay assessments. Smith details how one critic, Les Perelman at MIT, has created an essay-generating program, the BABEL generator, that creates nonsense essays designed to trick the algorithmic “robo-graders” for the Graduate Record Exam (GRE). When Perelman submits a nonsense essay to the GRE computer, the algorithm gives the essay a near perfect score. Perelman observes, “”It makes absolutely no sense,” shaking his head. “There is no meaning. It’s not real writing. It’s so scary that it works….Machines are very brilliant for certain things and very stupid on other things. This is a case where the machines are very, very stupid.”
Critics of “robo-graders” are also worried that students might learn how to game the system, that is, give the algorithms exactly what they are looking for, and thereby receive undeservedly high scores. Cyndee Carter, the assessment development coordinator for the Utah State Board of Education, describes instances of students gaming the state test: “…Students have figured out that they could do well writing one really good paragraph and just copying that four times to make a five-paragraph essay that scores well. Others have pulled one over on the computer by padding their essays with long quotes from the text they’re supposed to analyze, or from the question they’re supposed to answer.”
Despite these shortcomings, computer designers are learning and further perfecting computer algorithms. It’s anticipated that more states will soon use refined algorithms to read and grade student essays.
Grading student essays is not the end of computer assessment. Once you’ve left school and start looking for a job, you may find that your resume is read not by an employer eager to hire a new employee, but by an algorithm whose job it is to screen for appropriate job applicants. In the brief article, “How Algorithms May Decide Your Career: Getting a job means getting past the computer,” The Economist reports that most large firms now use computer programs, or algorithms, for screening candidates seeking junior jobs. Applicant Tracking Systems (ATS) can reject up to 75% of candidates, so it becomes increasingly imperative for applicants to send resumes filled with key words that will peak screening computers’ interests.
Once your resume passes the initial screening, some companies use computer driven visual interviews to further screen and select candidates. “Many companies, including Vodafone and Intel, use a video-interview service called HireVue. Candidates are quizzed while an artificial-intelligence (AI) program analyses their facial expressions (maintaining eye contact with the camera is advisable) and language patterns (sounding confident is the trick). People who wave their arms about or slouch in their seat are likely to fail. Only if they pass that test will the applicants meet some humans.”
Although one might think that computer-driven screening systems might avoid some of the biases of traditional recruitment processes, it seems that AI isn’t bias free, and that algorithms may favor applicants who have the time and monetary resources to continually retool their resumes so that these present the code words that employers are looking for. (This is similar to gaming the system, described above.) “There may also be an ‘arms race’ as candidates learn how to adjust their CVs to pass the initial AI test, and algorithms adapt to screen out more candidates.”
“More States Opting To ‘Robo-Grade’ Student Essays By Computer,” Tovia Smith, NPR, June 30, 2018
“How Algorithms May Decide Your Career: Getting a job means getting past the computer” The Economist, June 21, 2018
“Welcoming our New Robotic Overlords,” Sheelah Kolhatkar, The New Yorker, October 23 2017
“AI, Robotics, and the Future of Jobs,” Pew Research Center
“Artificial intelligence and employment,” Global Business Outlook
Asking questions is a critical aspect of learning. We’ve previously written about the importance of questions in our blog post “Evaluation Research Interviews: Just Like Good Conversations.” In a recent article, “The Surprising Power of Questions,” which appears in the Harvard Business Review, May-June, 2018, authors Alison Wood Brooks and Leslie K. John offer suggestions for asking better questions.
As Brooks and John report, we often don’t ask enough questions during our conversations. Too often we talk rather than listen. Brooks and John, however, note that recent research shows that by asking good questions and genuinely listening to the answers, we are more likely to achieve both genuine information exchange and effective self-presentation. “Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding.”
Although asking more questions in our conversations is important, the authors show that asking follow-up questions is critical. Follow-up questions “…signal to your conversation partner that you are listening, care, and want to know more. People interacting with a partner who asks lots of follow-up questions tend to feel respected and heard.”
Another critical component of a question-asking is to be sure that we ask open-ended questions, not simply categorial (yes/no) questions. “Open-ended questions …can be particularly useful in uncovering information or learning something new. Indeed, they are wellsprings of innovation—which is often the result of finding the hidden, unexpected answer that no one has thought of before.”
Asking effective questions depends, of course, on the purpose and context of conversations. That said, it is vital to ask questions in an appropriate sequence. Counterintuitively, asking tougher questions first, and leaving easier questions until later “…can make your conversational partner more willing to open up.” On the other hand, asking tough questions too early in the conversation, can seem intrusive and sometimes offensive. If the ultimate goal of the conversation is to build a strong relationship with your interlocutor, especially with someone who you don’t know, or don’t know well, it may be better opening with less sensitive questions and escalate slowly. Tone and attitude are also important: “People are more forthcoming when you ask questions in a casual way, rather than in a buttoned-up, official tone.”
While question-asking is a necessary component of learning, the authors remind us that “The wellspring of all questions is wonder and curiosity and a capacity for delight. We pose and respond to queries in the belief that the magic of a conversation will produce a whole that is greater than the sum of its parts. Sustained personal engagement and motivation—in our lives as well as our work—require that we are always mindful of the transformative joy of asking and answering questions.”
“The Surprising Power of Questions,” Alison Wood Brooks and Leslie K. John. Harvard Business Review, May–June 2018 (pp.60–67)
In a recent article in the May 2, 2018 Harvard Business Review, “Learning Is a Learned Behavior. Here’s How to Get Better at It,” Ulrich Boser rejects the idea that our capacities for learning are innate and immutable. He argues, instead, that a growing body of research shows that learners are not born, but made. Boser says that we can all get better at learning how to learn, and that improving our knowledge-acquisition skills is a matter of practicing some basic strategies.
Learning how to learn is a matter of:
- setting clear and achievable targets about what we want to learn
- developing our metacognition skills (“metacognition” is a fancy way to say thinking about thinking) so that as we learn, we ask ourselves questions like, Could I explain this to a friend? Do I need to get more background knowledge? etc.
- reflecting on what we are learning by taking time to “step away” from our deliberate learning activities so that during periods of calm and even mind-wondering, new insights emerge
Boser says that research shows we’re more committed, if we develop a learning plan with clear objectives, and that periodic reflection on the skills and concepts we’re trying to master, i.e., utilizing metacognition, makes each of us a better learner.
In a recent article “Against Metrics: How Measuring Performance by Numbers Backfires,” Jerry Z Muller argues that companies, educational institutions, government agencies, and philanthropies are now in the grip of what he calls “metric fixation,” “…the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardized data (metrics).”
In this brief and important article, Muller critiques the growing phenomenon of paying employees for performance. He points out that such schemes often lead to a narrowing measure of what is desirable for the organization, leads members of an organization to “game the system”, often undermines organizations ability to think more broadly about their purposes, and most importantly, impedes innovation.
Looking at the unintended outcomes of metric fixation, he writes:
“When reward is tied to measured performance, metric fixation invites just this sort of gaming. But metric fixation also leads to a variety of more subtle unintended negative consequences. These include goal displacement, which comes in many varieties: when performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a pay rise or raising the stock price at the time that stock options are vested), people focus on satisfying those measures – often at the expense of other, more important organizational goals that are not measured. The best-known example is ‘teaching to the test’, a widespread phenomenon that has distorted primary and secondary education in the United States since the adoption of the No Child Left Behind Act of 2001.”
Pay for performance schemes, however, are not alone in eliciting a narrowing of goals, and a tendency to game the system. Metric fixation (or what I term the “tyranny of measurement”) can be a risk for a range of non-profit organizations and educational institutions who often feel that demands for accountability can be addressed by merely counting the number of participants who receive services, or the number of students who score well on reading tests. While it is important to have clear goals, and to be able to indicate if these goals are met, organizations, in their rush to address demands from funders and other stakeholders for accountability, must be careful not to reduce their goals—indeed their organizations’ vision— to only a few countable variables. “What can and does get measured is not always worth measuring, may not be what we really want to know, and may draw effort away from the things we care about” (Muller). As Albert Einstein observed, “Not everything that counts can be counted, and not everything that can be counted, counts.”
“Against Metrics: How Measuring Performance by Numbers Backfires,” Aeon, April 24, 2018
The Tyranny of Metrics, Jerry Z. Muller, Princeton University Press, 2018
Brad recently presented an introductory seminar on program evaluation at Nonprofit Net, a Lexington, Massachusetts-based organization. Nonprofit Net is a forum for Massachusetts nonprofit leaders and nonprofit consultants which offers seminars on topics of importance to the nonprofit community. Brad’s presentation provided an introduction to program evaluation, outlined the benefits of evaluating outcomes in order to demonstrate programs’ achievements and challenges, introduced the use of logic models, and reviewed the key questions that nonprofit leaders should consider as they approach a program evaluation.
The seminar was based on the materials in the following white papers, which you can download for free:
Recent revelations about Facebook and Cambridge Analytica’s use of personal data have raised serious concerns about internet privacy. It would appear that we inhabit a world in which privacy is increasingly under assault—not just from leering governments, but also from panoptic corporations.
Although the right to privacy in the US is not explicitly protected by the Constitution, constitutional amendments and case law have provided some protections to what has become a foundational assumption of American citizens. The “right to privacy” (what Supreme Court Justice Louis Brandis once called “the right to be left alone,”) is a widely held value, both in the U.S. and throughout the world. But why is privacy important?
In “Ten Reasons Why Privacy Matters” Danial Solove, Professor of Law at George Washington University Law School, lists ten important reasons including: limiting the power of government and corporations over individuals; the need to establish important social boundaries; creating trust; and as a precondition for freedom of speech and thought. Solove also notes, “Privacy enables people to manage their reputations. How we are judged by others affects our opportunities, friendships, and overall well-being.”
Julie E. Cohen, of Georgetown argues that privacy is not just a protection, but an irreducible environment in which individuals are free to develop who they are and who they will be. “Privacy is shorthand for breathing room to engage in the process of … self-development. What Cohen means is that since life and contexts are always changing, privacy cannot be reductively conceived as one specific type of thing. It is better understood as an important buffer that gives us space to develop an identity that is somewhat separate from the surveillance, judgment, and values of our society and culture.” (See “Why Does Privacy Matter? One Scholar’s Answer” Jathan Sadowski, The Atlantic, Feb 26) In the Harvard Law Review, (“What Privacy Is For” Julie E. Cohen, Harvard Law Review, Vol. 126, 2013) Cohen writes, “Privacy shelters dynamic, emergent subjectivity from the efforts of commercial and government actors to render individuals and communities fixed, transparent, and predictable. It protects the situated practices of boundary management through which self-definition and the capacity for self-reflection develop.”
Cohen’s argument that privacy is a pre-condition for the development of an autonomous and thriving self is a critical and often overlooked point. If individuals are to develop, individuate, and thrive, they need room to do so, without interference or unwanted surveillance. Such conditions are also necessary for the maintenance of individual freedom vs. slavery. As Orlando Patterson argued in his book, Freedom Vol.1 Freedom in the Making of Western Culture (Basic Books, 1991) freedom historically developed in the West as a long struggle against chattel slavery. Slavery, of course, entails the subjugation of the individual/person, and depends upon the thwarting of autonomy. While slavery may not fully eradicate the full and healthy development of the “self,” it may deform and distort that development. Autonomous selves are both the product of and the condition of social freedom.
Privacy, which is crucial to the development of a person’s autonomy and subjectivity, when reduced by surveillance or restrictive interference—either by governments or corporations who gather and sell our private information—may interfere not just with social and political freedom, but with the development and sustenance of the self. “Transparency” (especially when applied to personal information) may seem like an important feature to those who gather “Big Data,” but it may also represent an intrusion and an attempt to whittle away the environment of privacy that the self depends upon for its full and healthy development. As Cohen observes, “Efforts to repackage pervasive surveillance as innovation — under the moniker “Big Data” — are better understood as efforts to enshrine the methods and values of the modulated society at the heart of our system of knowledge production. In short, privacy incursions harm individuals, but not only individuals. Privacy incursions in the name of progress, innovation, and ordered liberty jeopardize the continuing vitality of the political and intellectual culture that we say we value.” (See “What Privacy Is For” Julie E. Cohen, Harvard Law Review, Vol. 126, 2013)
Privacy is not just important to the protection of individuals from governments and commercial interests, it is also essential for the development of full, autonomous, and healthy selves.
“Ten Reasons Why Privacy Matters” Daniel Solove
“Why Does Privacy Matter? One Scholar’s Answer” Jathan Sadowski, The Atlantic, Feb 26, 2013
“What Privacy Is For” Julie E. Cohen, Harvard Law Review, Vol. 126, 2013
Orlando Patterson argued in his book, Freedom Vol.1 Freedom in the Making of Western Culture (Basic Books, 1991)
“Facebook and Cambridge Analytica, What You Need to Know as Fallout Widens” Kevin Granville, New York Times, Mar 19, 2018
“I Downloaded the Information that Facebook Has on Me. Yikes” Brian Chen, New York Times, Apr 11, 2018
“Right to Privacy: Constitutional Rights & Privacy Laws” Tim Sharp, Livescience, June 12, 2013
Surveillance Capitalism, Shoshone Zuboff
Listen to “Facebook and the Reign of Surveillance Capitalism” Radio Open Source
Read a review of Surveillance Capitalism
“How to Save Your Privacy from the Internet’s Clutches” Natasha Lomas, Romain Dillet, TC, Apr 14, 2018
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are, Seth Stephens-Davidowitz, Dey St., 2017
Rationalization and Bureaucracy
In the early years of the 20th century, the German sociologist Max Weber argued that modern society increasingly relies upon the “rationalization” of social organizations and institutions. He maintained that Western society is increasingly reliant upon reason, efficiency, predictability, and means/ends calculation. He further believed that modern society is highly dependent upon both public and private bureaucracies (e.g., the nation state and the modern corporation) as a way to achieve important societal goals (education, social welfare, medical care, business administration, governance, etc.) Bureaucracies are, “Highly organized networks of hierarchy and command structures (which are) necessary to run any ordered society – especially ones large in scope.” (See “Max Weber’s Theory of Rationalization: What it Can Tell Us of Modernity,” ) As one form of social organization, bureaucracy is distinguished by its: (1) clear hierarchy of authority, (2) rigid division of labor, (3) written and inflexible rules, regulations, and procedures, and (4) impersonal relationships. (For this and additional definitions, see the BusinessDictionary)
Weber and subsequent social theorists saw the process of rationalization and bureaucratization as replacing traditional modes of life, traditional values, and religious orientations with a society characterized by growing calculability, pursuit of individuals’ self-interest, efficiency, and ordered control. (Weber termed the loss of tradition that accompanied the increasing rationalization of Western society as the “disenchantment” of society.) As modernity transforms traditional social forms and social values, rationality and bureaucracy come to dominate the various spheres of contemporary society. Moreover, as society becomes ever more rationalized, it increasingly depends upon bureaucratic regimes of governance and management by impersonal rules and the exercise of technical knowledge by experts. Today, various social arenas—ranging from government to corporate organizations, from healthcare to public education—have become suffused with the ethos of bureaucracy and rationality. “…rationalization means a historical drive towards a world in which ‘one can, in principle, master all things by calculation.’” (See, Max Weber, in Stanford Encyclopedia of Philosophy)
Advantages and Disadvantages of Bureaucracy
Although “bureaucracy’ is often thought of as a pejorative term, bureaucracy has some advantages over other forms of social organization. Bureaucracy creates and utilizes rules and laws (vs. fiat decisions by a powerful notable, such as a king), mobilizes the knowledge of educated experts, promotes meritocracy, delineates and sets boundaries for the exercise of social power, establishes a formal chain of command and specifies organizational authority, and provides a technically efficient form of organization for dealing with routine matters that concern large numbers of persons. These advantages however, are accompanied by what are thought by many to be substantial disadvantages, including, compelling officials to conform to fixed rules and detailed procedures, sponsoring bureaucrats’ focus on narrow objectives, and supporting bureaucrats to become defensive, rigid, and unresponsive to the urgent individual needs and concerns of private citizens. “…individual officials working under bureaucratic incentive systems frequently find it to be in their own best interests to adhere rigidly to internal rules and formalities in a ritualistic fashion, behaving as if “proper procedure” were more important than the larger goals for serving their clients or the general public that they are supposedly designed to accomplish (i.e., the “red tape” phenomenon).” (See Bureaucracy)
Education and Bureaucracy
If we look at public education in contemporary society, we see many features associated with bureaucracy. State education agencies, districts, and schools: 1) are run by trained experts (e.g., credentialed teachers and administrators), 2) feature rigid hierarchies of authority, 3) have a strict division of labor, 4) depend upon and are run by formal and impersonal rules of administration and control, and 5) credential students by relying on impersonal and standardized methods for assessing student achievement. Additionally, students are routinely segregated into age-specified categories (classes) and are subjected not to individually tailored curricula, but to routine and standardized curricula that attempt to teach students en masse.
Does Standardized Testing Support Educational Bureaucracy?
Standardized testing of student achievement is one of the bureaucratic characteristics of modern public education. Although assessment is thought to be a necessary means for measuring student learning, it is also a means by which educational organizations categorize students, assign them social statues, and allocate them to various social trajectories (“life chances” to use Weber’s terminology). Standardized testing regimes also assist the educational bureaucracy by creating different categories of clientele (i.e., students) who can then be served en masse by large-scale routinized educational programs and mass-produced textbooks. Some would even argue that students are made to fit school as much as schools are made to fit the student. (For a summary of the problems associated with standardized testing, see “What’s Wrong with Standardized Tests.”)
While students are subject to the rule of bureaucracy, so too are faculty and administrators. Like the students they teach and oversee, faculty and administrators are subject to formal structures of authority, adhere to a strict division of labor, follow formal rules and regulations, and must be credentialed and certified. Like their students, teachers are also subject to assessment and review. (See our previous blog post “Too Much Assessment in Higher Education,” for an example of the effects of assessment on higher education.) Schools are also reviewed and rated by State Departments of Education.
While some feel that the stultifying aspects of bureaucracy may be ameliorated, the original theorist of rationalization and bureaucracy, Max Weber, was pessimistic about the reform of bureaucracy. As he surveyed the early 20th century and considered the likely developmental direction of Western society, he said that citizens of society were likely to find themselves increasingly entrapped in what he termed the “iron cage of bondage,” which continued to be cemented by the growth of rationalization and bureaucracy. Whether this dark prognosis is generally true for Western society is still very much debatable. That said, it is difficult to imagine large-scale public education without many of the features of bureaucracy that Weber first described —including standardized student testing. (For examples of reform efforts as they apply to standardized tests, see The National Center for Fair & Open Testing)
“What’s Wrong with Standardized Tests,” The National Center for Fair & Open Testing
In an important article, “The Misguided Drive to Measure ‘Learning Outcomes’”, New York Times, Feb 23, 2018, Molly Worthen argues that the growth of, and seeming obsession with, the assessment of learning outcomes in higher education has profoundly shaped curricula and instruction, undercut the unique capacities of colleges and universities to foster genuine critical thinking, and proven to be both bureaucracy bloating and extremely expensive. Worthen shows that, driven largely by the interests of accrediting agencies and for-profit tech and consulting companies, higher education’s rush to demonstrate student learning and skill-acquisition disproportionately affects non-elite schools, and often compels these under-resourced institutions to devote scarce dollars to obtaining evidence of instructional impact, “…more and more university administrators want campus-wide, quantifiable data that reveal what skills students are learning. Their desire has fed a bureaucratic behemoth known as learning outcomes assessment.”
In order to show that students graduate with job-ready skills and attitudes, Worthen argues that higher education institutions’ focus on assessment obscures the “real crisis” in higher education: “the system’s deepening divide into a narrow tier of elite institutions primarily serving the rich and a vast landscape of glorified trade schools for everyone else.” She notes the cruel irony in the mania for assessment is that there is little evidence beyond occasional anecdotes, that regimes of assessment actually improve student learning. Moreover, more selective (i.e., elite) institutions, themselves, don’t utilize assessment of learning outcomes at the same rate as less prestigious institutions. “Research indicates that the more selective a university, the less likely it is to embrace assessment.”
Perhaps the greatest irony, Worthen writes, is that assessment regimes subvert the unique purposes and capacities of higher education. “The value of universities to a capitalist society depends on their ability to resist capitalism, to carve out space for intellectual endeavors that don’t have obvious metrics or market value.” She further observes, “Producing thoughtful, talented graduates is not a matter of focusing on market-ready skills. It’s about giving students an opportunity that most of them will never have again in their lives: the chance for serious exploration of complicated intellectual problems, the gift of time in an institution where curiosity and discovery are the source of meaning.”
“The Misguided Drive to Measure ‘Learning Outcomes’”, Molly Worthen, New York Times, 23 February 2018
The development and deployment of robotics and artificial intelligence continues to affect the world of work. As we’ve discussed in previous blogposts
“Humans Need Not Apply: What Happens When There’s No More Work?”, “Will President Trump’s Wall Keep Out the Robots?” and “Dark Factories” , AI and robotics are transforming both blue-collar jobs and professional occupations. New technologies promise to change not just how we work and are employed, but also to alter the traditional meanings of work and employment that have been central to peoples’ self-conceptions and identities.
In “How Automation Will Change Work, Purpose, and Meaning,” by Robert C. Wolcott, Harvard Business Review, January 11, 2018, Wolcott says that new technologies not only raise the question, “How are the spoils of technology to be distributed?” but equally baffling, “When technology can do nearly anything, what should I do, and why?” He cites Hannah Arendt’s writings in The Human Condition about the importance of moving from a self-conception that identifies work as purpose, to one that encompasses the idea of the Vita Activa, the active life, in which humans, when freed from much of the drudgery of labor, will need to aspire to integrate non-labor activity in the world with contemplation about the world. Wolcott asks, “When our machines release us from ever more tasks, to what will we turn our attentions? This will be the defining question of our coming century.”
In “The Meaning of Life in a World Without Work” by Yuval Noah Harari, The Guardian, May 8, 2017, Harari writes that as new technologies increasingly displace humans from work, the real problem will be to keep occupied the masses of people (i.e., members of “the useless class” as Harari defines them) who are no longer involved in work. Harari says that one possible scenario might be the deployment of virtual reality computer games. “Economically redundant people might spend increasing amounts of time within 3D virtual reality worlds, which would provide them with far more excitement and emotional engagement than the “real world” outside.” He likens such virtual reality to the world’s religions, which Harari says, are filled with practices and beliefs that give meaning to adherents’ lives, but are not themselves necessary or ‘real’ in any objective way. Harari asserts that it doesn’t much matter whether one finds stimulation from the ‘real’ world or from computer-simulated reality, because ultimately, both rely on what’s happening inside our brains. Further, he observes, “Hence virtual realities are likely to be key to providing meaning to the useless class (created by) the post-work world. Maybe these virtual realities will be generated inside computers. Maybe they will be generated outside computers, in the shape of new religions and ideologies. Maybe it will be a combination of the two. The possibilities are endless, and nobody knows for sure what kind of deep play will engage us in 2050.”
Although Harari’s sketch of possible futures seems shockingly Huxleyan, it does attempt to imagine a future in which large swaths of the population will be unnecessary to the functioning of the productive economy. Anticipating criticism of the brave new world that he’s sketched, Harari, referring to the world’s religions, writes, “But what about truth? What about reality? Do we really want to live in a world in which billions of people are immersed in fantasies, pursuing make-believe goals and obeying imaginary laws? Well, like it or not, that’s the world we have been living in for thousands of years already.”
The challenges, and some might say the catastrophies, associated with the new technologies are not merely technological. They are political, and will be shaped by the kinds of political institutions and social policies that nations use to deal with them. In the December 27 2017, New York Times article, “The Robots are Coming and Sweden is Fine,” Peter S. Goodman notes that Swedish workers appear less threatened by the introduction of robotics and AI because Sweden’s history of social democracy and the relatively strong influence of unions temper the effects of new technologies on Swedish workers. Goodman argues that, unlike much of the rest of the world, the fear that robots will steal jobs “… has little currency in Sweden or its Scandinavian neighbors, where unions are powerful, government support is abundant, and trust between employers and employees runs deep. Here, robots are just another way to make companies more efficient. As employers prosper, workers have consistently gained a proportionate slice of the spoils — a stark contrast to the United States and Britain, where wages have stagnated even while corporate profits have soared.”
How AI and robotics will affect the U.S. is still uncertain, although as we’ve discussed in “Humans Need Not Apply…” some researchers believe that within two decades, half of U.S. jobs could be handled by machines (For example, check out the video “Why Amazon Go Is Being Called the Next Big Job Killer” below). The character of work, and the consequent effects on the population will be determined, in part, by the strength of institutions that have mediated the relationship between employers and employees. In the U.S. sadly, those institutions and social agreements have largely been weakened or eliminated in the last 35-40 years. The introduction of robotics and AI in America is likely to follow a far different path than in Sweden.
Evaluation is a research enterprise whose primary goal is to identify whether desired changes have been achieved. Evaluation is a type of applied social research that is conducted with a value, or set of values, in its “denominator.” Evaluation research is always conducted with an eye to whether the desired outcomes, or results, of a program, initiative, or policy were achieved, especially as these outcomes are compared to a standard or criterion. At the heart of program evaluation is the idea that outcomes, or changes, are valuable and desired. Some outcomes are more valuable than others. Evaluators conduct evaluation research to find out if these valued changes are, in fact, achieved by the program or initiative.
Evaluation research shares many of the same methods and approaches as other social sciences, and indeed, natural sciences. Evaluators draw upon a range of evaluation designs (e.g. experimental design, quasi-experimental design, non-experimental design) and a range of methodologies (e.g. case studies, observational studies, interviews, etc.) to learn what the effects of a given intervention have been. Did, for example, 8th grade students who received an enriched STEM curriculum do better on tests, than did their otherwise similar peers who didn’t receive the enriched curriculum? Do homeless women who receive career readiness workshops succeed at obtaining employment at greater rates than do other similar homeless women who don’t participate in such workshops? (For more on these types of outcome evaluations, see our previous blog post, “What You Need to Know About Outcome Evaluations: The Basics,”) While not all evaluations are outcome evaluations, all evaluations gather systematic data with which judgments about the program or initiative can be made.
Another way to differentiate social research from evaluation research is to understand that social research seeks to find out “what is the case?” “What is out there?” “How does the world really work?” etc. For example, in political science, researchers may want to find out how citizens of California vote in national elections, or, what are their attitudes towards certain candidates or policies. Sociology may investigate the causes of racial segregation or the relationship(s) between race and class. These instances of social research are primarily interested in discovering what is the case, regardless of the value we might attribute to the findings of the research. Researchers in political science are neutral about the percentages of California voters who vote Republican, Democrat, Independent, Green, etc. They are most interested in knowing how people vote, not if they vote for one particular party.
Although evaluation research is interested in a truthful accurate description of what is the case, it is ALSO interested in discovering whether findings indicate that what is there (i.e., is present) is valuable, important, desired, etc. When evaluators look for outcomes they don’t just want to know if anything at all happened, or changed, they want to discover if something specific and valued happened. Evaluators don’t just set their sites on describing the world, but on determining whether certain valued and worthwhile things happened. While evaluators use many of the same methods and approaches as other researchers, evaluators must employ an explicit set of values against which to judge the findings of their empirical research. The means that evaluators must both be competent social scientists AND exercise value-based judgments and interpretations about the meaning of data.
Program evaluations seldom occur in stable, scientifically controlled environments. Often programs are implemented in complex and rapidly evolving settings that make traditional evaluation research approaches—which depend upon the stability of the “treatment” and the designation of predetermined outcomes—difficult to utilize.
Michael Quinn Patton, one of the originators of Developmental Evaluation, says that “Developmental evaluation processes include asking evaluative questions and gathering information to provide feedback and support developmental decision-making and course corrections along the emergent path. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation, and intentional change. The evaluator’s primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and to facilitate data-based assessments and decision-making in the unfolding and developmental processes of innovation.”
In their paper, “A Practitioners Guide to Developmental Evaluation,” Dozios and her colleagues note, “Developmental Evaluation differs from traditional forms of evaluation in several key ways:”
- The primary focus is on adaptive learning rather than accountability to an external authority.
- The purpose is to provide real-time feedback and generate learnings to inform development.
- The evaluator is embedded in the initiative as a member of the team.
- The DE role extends well beyond data collection and analysis; the evaluator actively intervenes to shape the course of development, helping to inform decision-making and facilitate learning.
- The evaluation is designed to capture system dynamics and surface innovative strategies and ideas.
- The approach is flexible, with new measures and monitoring mechanisms evolving as understanding of the situation deepens and the initiative’s goals emerge
Development evaluation is especially useful for social innovators who often find themselves inventing the program as it is implemented, and who often don’t have a stable and unchanging set of anticipated outcomes. Following Patton, Dozois, Langlois, and Blanchet-Cohen observe that Developmental Evaluation is especially well suited to situations that are:
- Highly emergent and volatile (e.g., the environment is always changing)
- Difficult to plan or predict because the variables are interdependent and non-linear
- Socially complex— requiring collaboration among stakeholders from different organizations, systems, and/or sectors
- Innovative, requiring real-time learning and development
Developmental Evaluation, however, is increasingly appropriate for use in the non-profit world, especially where the stability of programs’ key components including the program’s core treatment and eventual, often evolving, outcomes, are not as certain or firm as program designers might wish.
Brad Rose Consulting is experienced in working with program’s whose environments are volatile and whose iterative program designs are necessarily flexible. We are adept at collecting data that can inform the on-going evolution of a program, and have 20+ years of providing meaningful data to program designers and implementers that help them to adjust to rapidly changing and highly variable environments.
A Practitioner’s Guide to Developmental Evaluation, Elizabeth Dozois, Marc Langlois, and Natasha Blanchet-Cohen
Organizations vs. Programs
Organizations are social collectivities that have: members/employees, norms (rules for, and standards of, behavior), ranks of authority, communications systems, and relatively stable boundaries. Organizations exist to achieve purposes (objectives, goals, and missions) and usually exist in a surrounding environment (often composed of other organizations, individuals, and institutions.) Organizations are often able to achieve larger-scale and more long-lasting effects than individuals are able to achieve. Organizations can take a variety of forms including corporations, non-profits, philanthropies, and military, religious, and educational organizations.
Programs are discreet, organized activities and actions (or sets of activities and actions) that utilize resources to produce desired, typically targeted, outcomes (i.e., changes and results). Programs typically exist within organizations. (It may be useful to think of programs as nested within one or, in some cases, more than one organization.) In seeking to achieve their goals, organizations often design and implement programs that use resources to achieve specific ends for program participants and recipients. Non-profit organizations, for example, implement programs that mobilize resources in the form of activities, services, and products that are intended to improve the lives of program participants/recipients. In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, services, and materials, to positively affect the lives of those they serve.
What is Program Evaluation?
Program evaluation is an applied research process that examines the effects and effectiveness of programs and initiatives. Michael Quinn Patton notes that “Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs in order to make judgements about the program, to improve program effectiveness, and/or to inform decisions about future programming. Program evaluation can be used to look at: the process of program implementation, the intended and unintended results produced by programs, and the long-term impacts of interventions. Program evaluation employs a variety of social science methodologies–from large-scale surveys and in-depth individual interviews, to focus groups and review of program records.” Although program evaluation is research-based, unlike purely academic research, it is designed to produce actionable and immediately useful information for program designers, managers, funders, stakeholders, and policymakers.
Organization Development, Strategic Planning, and Program Evaluation
Organization Development is a set of processes and practices designed to enhance the ability of organizations to meet their goals and achieve their overall mission. It entails “…a process of continuous diagnosis, action planning, implementation and evaluation, with the goal of transferring (or generating) knowledge and skills so that organizations can improve their capacity for solving-problems and managing future change.” (See: Organizational Development Theory, below) Organization Development deals with a range of features, including organizational climate, organizational culture (i.e., assumptions, values, norms/expectations, patterns of behavior) and organizational strategy. It seeks to strengthen and enhance the long-term “health” and performance of an organization, often by focusing on aligning organizations with their rapidly changing and complex environments through organizational learning, knowledge management, and the specification of organizational norms and values.
Strategic Planning is a tool that supports organization development. Strategic planning is a systematic process of envisioning a desired future for an entire organization (not just a specific program), and translating this vision into broadly defined set of goals, objectives, and a sequence of action steps to achieve these. Strategic planning is an organization’s process of defining its strategy, or direction, and making decisions about allocating its resources to pursue this strategy.
Strategic plans typically identify where and organization is at and where it wants to be in the future. It includes statements about how to “close the gap,” between its current state and its desired, future state. Additionally, strategic planning requires making decisions about allocating resources to pursue an organizations strategy. Strategic planning generally involves not just setting goals and determining actions to achieve the goals, but also mobilizing resources.
Program evaluation is uniquely able to contribute to organization development–the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency. Although evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations collect, synthesize, and report information that can be used to improve the broader operation and health of the organization that hosts the program. Additionally, program evaluation can aid the strategic planning process, by using data about an organization’s programs to indicate whether the organization is successfully realizing its goals and mission through its current programming.
Brad Rose Consulting works at the intersection of evaluation and organization development. While our projects begin with a focus on discrete programs and initiatives, the answers to the questions that drive our evaluation research provide vital insights into the effectiveness of the organizations that host, design, and fund those programs. Findings from our evaluations often have important implications for the development and sustainability of the entire host organization.
Organizations: Structures, Processes, and Outcomes, Richard H. Hall and Pamela S Tolbert, Pearson Prentice Hall, 9th edition.
Utilization Focused Evaluation, Michael Quinn Patton, Sage, 3rd edition, 1997
What a Strategic Plan Is and Isn’t
Ten Keys to Successful Strategic Planning for Nonprofit and Foundation Leaders
Types of Strategic Planning
Understanding Strategic Planning
Five Steps to a Strategic Plan
Five Steps to a Strategic Plan
(See our previous blog posts: “Humans Need Not Apply: What Happens When There’s No More Work?” and “Will President Trump’s Wall Keep Out the Robots?”) Today we’d like to refer readers to an important article, “Dark Factory”, that recently appeared in The New Yorker. “Dark Factory” explores the growing impact of robotics and AI on the manufacturing and service sectors of the U.S. Economy.
In “Dark Factory”, Sheelah Kolhatkar discusses her visit to Steelcase, the manufacturer of office furniture. Steelcase, like much of American manufacturing, has had its economic ups and downs over the years. Kolhatkar describes how, in recent years, the company has increasingly employed robotics as a means to improve manufacturing efficiency, and as a result, now relies on fewer workers than it has in the past. Representative of an ever-growing number of manufacturing companies in the U.S., Steelcase employs fewer and fewer high school graduates, and now seeks college educated employees with technological skills, so that these higher skilled workers can supervise an expanding army of manufacturing robots.
As Kolhatkar shows, while efficiency gains are good for Steelcase and other manufacturing companies that employ robots and AI— and even, in some cases, make work more tolerable and less grueling for the remaining employees on the shop floor— the net effect of these technologies is to displace large swaths of the work force and to shift wealth to the owners of companies. Kolhatkar cites research that shows that the use of industrial robots, like those at Steelcase, is directly related to decline in both the number of manufacturing jobs and declining pay for remaining workers.
Kolhatkar also discusses “dark factories”— factories and warehouses whose use of robots and AI are so extensive that they need not turn on the lights because there are so few human workers. While such factories and warehouses are not yet wide-spread, major U.S. corporations are looking to use robotics and AI to run nearly employee-less operations. Although some companies may not be eager to begin utilizing robotic warehouses, competitive pressure is sure to compel U.S. companies to implement fully robotized facilities, or lose competitive battles with other nations that do adopt these technologies.
AI and Robotics Not Limited to Manufacturing
The result of the growing use of robotics and AI in the U.S. is, of course, the declining demand for workers in what where once fairly labor-intensive human-dominated work environments. (Manufacturing now employees only about 10% of the US workforce, and these jobs are under constant threat by new technologies.) Although displaced manufacturing workers often seek jobs in the service sector, this sector is now hardly immune to automation. MacDonald’s, for example, is introducing “digital ordering kiosks” where customers electronically enter their orders and pay for their meals. MacDonald’s is expected to automate in 5500 restaurants by the end of 2018. Uber and Google continue to invest in the development of autonomous driving technologies, and the U.S. trucking industry is eager to adapt autonomous vehicles so that it can reduce substantial labor costs associated with trucking. Amazon has purchased Kiva, a robotics company, and is developing robots that can zoom around Amazon warehouses and fulfill orders. (A Deutsch Bank report estimates that Amazon could save 22 million dollars a year in each of its warehouses, simply by introducing warehouse robots to replace human workers.)
As Kolhatkar’s “Dark Factory” shows, while the future looks increasingly promising for the shareholders of companies who introduce labor-displacing robotics and AI, it doesn’t appear quite so sunny for those humans who must work for a living—especially in the manufacturing and service sectors of the U.S. economy. Like the “dark factories” that promise to displace them, for many workers, the future too, will be dark.
“Welcoming our New Robotic Overlords”, Sheelah Kolhatkar, The New Yorker, October 23 2017
“AI, Robotics, and the Future of Jobs”, Pew Research Center
“Artificial intelligence and employment”, Global Business Outlook
“Advances in Artificial Intelligence Could Lead to Mass Unemployment, Experts Warn”, James Vincent, The Independent, Wednesday 29 January 2014
What is an “Outcome”
“Outcomes,” i.e., specific changes or results, are what programs seek to produce, and what funders seek to fund. Programs and initiatives, especially in the nonprofit sector, exist to produce valuable and desired changes. While the measurement of outcomes is essential to evaluation, a key question for both programs and for evaluators is, ‘What counts as a desired outcome?”
Typically, education and nonprofit programs strive to see improvements in things like’s students’ reading scores, homeless persons employability, young people’s job-readiness, availability of affordable housing for home-seekers, etc. While the desirability of such outcomes appear “natural” or taken-for-granted, outcomes are, in fact, “agreed upon” events, states, or entities.
Who determines the outcomes that are to be evaluated?
For those who administer and run programs, it is often the case that such programs are highly dependent upon what funders value and desire to see changed. Although desired changes are usually viewed as self-evident, outcomes are a socially and politically defined entity. They depend upon a negotiated understanding of what constitutes a valuable change, and how is such change should be measured or indicated. Additionally, as many program leaders and educators will attest, changes that are desirable, are often shaped by the ability of proponents and researchers to measure such changes. (I’m reminded here, of Einstein’s famous quote: “Not everything that counts can be measured, and not everything that can be measured counts.”) As Heather Douglas points out, “ We must value something to find it significant enough to measure, to pluck it from the complexity of human social life, and to see it as a set of phenomena worthy of study.” (See Heather Douglas, “Facts, Values, and Objectivity”).
Of course, funders alone don’t determine the outcomes that programs produce. Program stakeholders can have a significant influence on what constitutes a desired outcome (See our previous blogpost, “The Importance of Understanding Stakeholders”). In an educational program, for example, a wide range of stakeholders may influence the desired outcomes of programming. Parents, teachers, community members, state and federal policy makers, business interests, politicians—may all influence what counts as a desirable outcome of education. Therefore, what stakeholders value (i.e., view as significant), is often what is viewed as the desired outcome of a program.
It’s important for program leaders and staff, and for evaluators to discuss and identify which changes programming seeks to affect. While evaluators deploy a range of methods to indicate or measure such changes, what counts as a desirable change, as a desirable “outcome,” is a question as critical to the success an evaluation as it is to the success of a program.
The Measure of Reality: Quantification and Western Society 1250-1600, Alfred W. Crosby. Cambridge University Press, 1997.
For a radical critique on the power of the funders of nonprofits to exercise influence on nonprofits goals and outcomes, see “How Liberal Nonprofits Are Failing Those They’re Supposed To Protect,” by William Anderson
Every program evaluation is conducted in a context in which there are parties (persons, organizations, etc.) who have an interest, or a “stake,” in the operation and success of the program. In the corporate world, a “stakeholder” is any member of the “groups without whose support the organization would cease to exist.” (see Corporate Stakeholder). More recently, the idea of “a stakeholder” has been broadened to include “any group or individual who is affected by, or who can affect the achievement of, an organization’s objectives.” (The Stakeholder Theory Summary.)
Indeed, in the not-for-profit world, stakeholders may include an array of persons and organizations including funders, community members, program participants, family members, volunteers, staff, government agencies, and the broader public.
Stakeholders in non-profits usually fall into one of three categories of legal statuses:
- Constitutional stakeholders such as board members or trustees of the non-profit organization
- Contractual stakeholders, including paid staff, or any business, group or individual that has a formal relationship with the organization.
- Third-party stakeholders including all the people and groups that may be affected by what the organization does. That includes businesses, the local government, and the citizens who live in the community. (See What is a Stakeholder of a Non-profit.)
Nonprofit stakeholders may range from those who support an organization, to those who oppose an organization. Stakeholder can include advocates, supporters, critics, competitors, and opponents. In its analysis of stakeholders in policy change efforts, the World Bank uses the categories of “promoter,” “defender,” “latents”, and “apathetics.” (See What is a Stakeholder Analysis.)
Conducting a stakeholder analysis is very useful for both evaluation and strategic planning efforts. Identifying various stakeholders’ interests in an organization’s mission and programming can help non-profit leaders and staff to be sure that their efforts and initiatives are achieving desired goals. They can also be useful in ensuring that the needs of those served are being directly met. For both evaluation and strategic planning purposes, a stakeholder analysis is an important process for achieving a shared understanding of each stakeholder’s specific interest in, and relevance to, the work of the non-profit or educational organization.
Brad Rose Consulting has developed a unique approach to stakeholder analysis, one that can be extremely useful as organizations examine the purposes, goals, specific activities, and desired outcomes of their work. We often work with organizations to implement stakeholder analyses. These analyses are helpful both in identifying where an organization is, at any given point in time, and for identifying where it wants to go in the future. You can see our basic stakeholder analysis form here.
Surveys can be an efficient way to collect information from a substantial number of people (i.e., respondents) in order to answer evaluation research questions. Typically, surveys strive to collect information from a sample (portion) of a broader population. When the sample is selected via random selection of respondents from a specified sampling frame, findings can be confidently generalized to the entire population.
Surveys may be conducted by phone, in-person, on the web, or by mail. They may ask standardized questions so that each respondent replies to precisely the same inquiry. Like other forms of research, highly effective surveys depend upon the quality of the questions asked of respondents. The more specific and clear the questions, the more useful survey findings are likely to be. Good surveys present questions in a logical order, are simple and direct, ask about one idea at a time, and are brief.
Surveys can ask either closed-ended or open-ended questions. Closed-ended questions may include multiple choice, dichotomous, Lickert scales, rank order scales, and other types of questions for which there are only a few answer categories available to the respondent. Closed-ended questions provide easily quantifiable data, for example, the frequency and percentage of respondents who answer a question in a particular way.
Alternatively, open-ended survey questions provide narrative responses that constitute a form of qualitative data. They require respondent reflection on their experience or attitudes. Open-ended questions often begin with: “why,” “how,” “what,” “describe,” “tell me about…,” or “what do you think about…”(See Open Ended Questions.) Open-ended survey questions depend heavily upon the interest, enthusiasm, and literacy level of respondents, and require extensive analysis precisely because they are not comprised of a small number of response categories.
Administering Surveys and Analyzing Results
Surveys can be administered in a variety of ways; in-person, on the phone, via mail, via the web, etc. Regardless of specific venue, it’s important to consider from the point of view of the respondent the factors that will maximize respondents’ participation, including accessibility of the survey, convenience of format, logicality of organization, and clarity of both the survey’s purpose and its questions.
Once survey data are collected and compiled, analyses of the data may take a variety of forms. Analysis of survey data essentially entails looking at quantitative data to find relationships, patterns, and trends. “Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc…That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting. It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. (See Community Tool Box “Collecting and Analyzing Data”) While data analysis usually entails some kind of statistical/quantitative manipulation of numerical information, it may also entail the analysis of qualitative data, i.e., data that is usually composed of words and not immediately quantifiable (e.g., data from in-depth interviews, observations, written documents, video, etc.)
The analysis of both quantitative and qualitative survey data (the latter typically collected in surveys from open-ended questions) is performed primarily to answer key evaluation research questions like, “Did the program make a difference for participants?” Effectively reporting findings from survey research not only entails accurate representation of quantitative findings, but interpretation of what both quantitative and qualitative data mean. This requires telling a coherent and evocative story, based on the survey data.
Brad Rose Consulting has over two decades of experience designing and conducting surveys whose findings compose an essential component of program evaluation activities. The resources below provide additional sources of information on the basics of survey research.
A recent issue of New Directions in Evaluation, (No. 152, Winter, 2016) “Social Experiments in Practice: The What, Why, When, Where, and How of Experimental Design and Analyses,” is devoted to the use of randomized experiments in program evaluation. The eight articles in this thematic volume discuss different aspects of experimental design—the practical and theoretical benefits and challenges of applying randomized controlled trials (RCTs) to the evaluation of programs. Although it’s beyond the scope of this blogpost to discuss each of the articles in detail, I’d like to mention a few insights offered by the authors and review the advantages and challenges of experimental design.
Random assignment helps rule out alternative explanations for outcomes
Experimental design in the social sciences, are studies that randomly assign subjects (i.e., program participants) to treatment and control groups, then measure changes (i.e., average changes) in both groups to determine if a program, or “treatment,” has had a desired effect on those who receive the treatment. As the issue’s editor, Laura Peck, observes, “…when it comes to the question of cause and effect—the question of a program’s or policy’s impact, we assert that a randomized experiment should be the evaluation design of choice.” (p.11) Indeed, experimental design studies—whose origins are in the natural sciences, and whose benefits are perhaps most frequently demonstrated in FDA testing of pharmaceuticals—is thought to be the “gold standard” for scientifically establishing causation. Random assignment of individuals to two groups—one that receives treatment and one that does not receive treatment—is the best way to establish whether desired changes are the result of what happens in the treatment (i.e., program). As the editor observes, “This ‘coin toss’ (i.e., random assignment) to allocate access to treatment carries substantial power. It allows us to rule out alternative explanations for differences in outcomes between people who have access to a service and people who do not.” (p.11)
There are still concerns surrounding the use of experimental design
Although experimental design is viewed by many as the premier indicator of causation, it’s use in evaluations can have practical challenges. There are potentially legal and ethical concerns about non-treatment for control groups (especially in the fields of medicine and education). Additionally, some argue that experimental design, especially in complex social interventions, is unable to identify which specific component of a treatment is responsible for the observed differences in the treatment group (the “black box” phenomenon.). Michael Scriven observes that it is nearly impossible to create a truly “double blind” experiment in the social world (i.e. experiments where neither experimental subject nor the evaluator knows who is in the treatment who is in the control groups). Moreover, some argue that experimental design can be more labor and time-intensive than other study designs, and therefore, more costly.
Quasi- experimental design is useful for showing before and after changes
While experimental design is the most prestigious method for determining the causal effects of a program, initiative, or policy, it is far from a universally appropriate design for evaluations. Quasi-experimental design, for example, is often used to show pre- and post- changes in those who participate in a program or treatment, although quasi-experimental design is unable to unequivocally confirm whether such changes are attributable to the program. One form of a quasi-experimental design is the “non-equivalent (pre-test, post-test) control group design”. In this design, participants are assigned to two groups (although not randomly assigned.) Both groups take a pre-test and a post-test, but only one group, the experimental group, receives the treatment/program. (The key textbook resource on both experimental and non-experimental designs is Experimental and Quasi-Experimental Designs, by Shadish, Cook, and Campbell, Houghton Mifflin.)
There are, of course, a range of non-experimental designs that are used productively in evaluation. These range from case studies to observational studies, and rely on a variety of methods, largely qualitative, including phone and in-person interviews, focus groups, surveys, and document reviews. (See this page for a brief table comparing the characteristics of qualitative and quantitative methods of research. See also the National Science Foundation’s very helpful, “Overview of Qualitative Methods and Analytic Techniques”) Qualitative evaluation studies can be very effective, and are often used in a mixed methods approach to evaluation work.
In July, we posted a blog post titled, “Humans Need Not Apply: What Happens When There’s No More Work?”As we mentioned in that post, the rise of artificial intelligence, machine learning, and robotics, have increasingly ominous implications for the future of work and employment. In a recent New York Times article, “The Long-Term Jobs Killer Is Not China. It’s Automation,” Claire Caine Miller traces the effects of automation on those who have been employed in America’s once preeminent industries—steel, coal, newspapers, etc. She observes that it is neither immigration nor globalization that threatens American workers; it’s automation. Referring to the recent 2016 political campaigns Caine Miller notes, “No candidate talked much about automation on the campaign trail. Technology is not as convenient a villain as China or Mexico, there is no clear way to stop it, and many of the technology companies are in the United States and benefit the country in many ways.” She quotes one study that shows that roughly 13 percent of manufacturing job losses are due to trade, and the rest are due to enhanced productivity attributable to automation.
In another article, “Evidence That Robots Are Winning the Race for American Jobs,” Caine Miller writes, “The industry most affected by automation is manufacturing. For every robot per thousand workers, up to six workers lost their jobs and wages fell by as much as three-fourths of a percent, according to a new paper by the economists, Daron Acemoglu of M.I.T. and Pascual Restrepo of Boston University.” In “How to Make America’s Robots Great Again” Farhad Manjoo, (New York Times, January 25, 2017) states, “Thanks to automation, we now make 85 percent more goods than we did in 1987, but with only two-thirds the number of workers.”
Manufacturing however, is not the only area where AI and robots threaten to displace human employees. In “A Robot May Be Training to Do Your Job. Don’t Panic,” Alexandra Levit argues that automation in the form of “social robotics,’ affective computing, and emotional awareness software, are now making inroads into the helping/caring professions, like nursing. Levit writes, “Eventually, the moment will come when machines possess empathy, the ability to innovate and other traits we perceive as uniquely human. What then? How will we sustain our own career relevance?”
In “Actors, teachers, therapists – think your job is safe from artificial intelligence? Think again,” Dan Tynan writes, “A January, 2017 report from the McKinsey Global Institute estimated that roughly half of today’s work activities could be automated by 2055, (give or take 20 years.)…Thanks to advances in artificial intelligence, natural language processing, and inexpensive computing power, jobs that once weren’t considered good candidates for automation suddenly are.”
According to these and other writers, automation and Artificial Intelligence are poised to sweep away or profoundly transform a number of occupations, and thereby alter both industry and society. While some writers foresee productive partnerships between AI and human colleagues, others warn that automation is likely to reduce the needs for human labor, and relegate sectors of the population to hard scrabble redundancy. As Martin Ford points out, this industrial revolution is different than previous ones, because new technology is taking aim at both blue and white collar work. (See Rise of the Robots: Technology and the Threat of a Jobless Future, by Martin Ford.)
Lest we become disconsolate at the prospect that robots will take our jobs, Claire Caine Miller suggests that there are a number of things that the US can do to prepare and adapt to these employment- threatening developments. She suggests that: 1) the US provide more and different kinds of education to employees, including teaching technical skills, like coding and statistics, and skills that still give humans an edge over machines, like creativity and collaboration; 2) creating better jobs for human workers including government subsidized employment (creating public sector jobs) and building infrastructure; 3) creating more care-giving jobs, strengthening labor unions, and training some workers to work in advanced manufacturing; 4)expanding the earned-income tax credit, providing a universal basic income, in which the government gives everyone a guaranteed amount of money, and establishing “ portable benefits” that wouldn’t be tied to a job to get health insurance. Caine Miller also suggests raising the minimum wage and even taxing robots (the latter, a proposal supported by Bill Gates.)
Whether these proposals will prove to be politically feasible or economically viable is difficult to judge. Some of these seem wildly utopian and difficult to envision—especially given a new Administration that built substantial electoral support on promises to revive employment in ‘smokestack’ industries, like steel and coal. That said, until relatively recently, it was difficult to envision the meteoric “rise of the robots” and the consequent effects on employment and society that such a development would have. Not even robots can reliably predict the future.
“The Long-Term Jobs Killer Is Not China. It’s Automation.” Claire Caine Miller, New York Times, December 12, 2016
“Where machines could replace humans—and where they can’t (yet),” Michael Chui, James Manyika, and Mehdi Miremadi, July 2016, McKinsey Quarterly,
“Actors, teachers, therapists – think your job is safe from artificial intelligence? Think again.” Dan Tynan, The Guardian, February 9, 2017.
“How to Make America’s Robots Great Again” Farhad Manjoo New York Times, January 25, 2017
“Evidence That Robots Are Winning the Race for American Jobs,” Claire Cain Miller New York Times, March 28, 2017
“A Robot May Be Training to Do Your Job. Don’t Panic.” Alexandra Levit, New York Times September. 10, 2016
“EU supports Personhood status to robots.” Alex Hern, The Guardian, January 12, 2017
Rise of the Robots: Technology and the Threat of a Jobless Future, Martin Ford, Basic Books, 2015″
In our last blog post “The Implications of Public School Privatization,” I referred to an article by Diane Ravitch that recently appeared in The New York Review of Books. That article claimed that the school privatization movement is largely composed of social conservatives, corporations, and business-friendly foundations. In a recent article by Janet Reitman, in Rolling Stone, “Betsy DeVos’ Holy War,” the author argues that the movement to privatize public schools is also sponsored, at least in part, by those who, like Betsy DeVos, would prefer to de-secularize schools and create institutions that reflect market friendly Christian values.
Betsy DeVos, embodies a nexus of wealth and hyper conservative Christianity. Her goals include support of “school choice” (i.e., voucher systems that direct tax money for public schools toward private and parochial schools) and, according to Reitman, the promotion of the religious colonization of public education, and more broadly, American society. (See also, “Betsy DeVos Wants to Use America’s Schools to Build ‘God’s Kingdom’” Kristina Rizga, Mar/Apr 2017 Issue, Mother Jones.
DeVos, as is well documented, is not deeply acquainted with public education—neither she nor her children attended public schools; she has never served on a school board, nor been an educator. DeVos who hails from a wealthy, Calvinist, Western Michigan dynasty that includes among other resources, her husband’s multi-billion dollar Amway fortune, and her father’s auto parts fortune (among other profitable ventures, her father, Edgar Prince, invented the lighted, automobile sun visor) now finds herself at the helm of the federal Department of Education. She appears to be even less a friend of public education than she is familiar with it. DeVos has devoted a substantial part of her political and philanthropic career to advocating for the privatization of public schools, and her home sate of Michigan has the highest number of for-profit charter schools in the nation. To learn more about DeVos’ plans for public schools in America, you can read this intriguing article “Betsy DeVos’ Holy War,” The resources below offer additional insights into Secretary DeVos and her plans for public schools. And to learn more about our work with schools visit our Higher education & K-12 page.
“Six astonishing things Betsy DeVos said — and refused to say — at her confirmation hearing” Valerie Strauss, Washington Post, January 18, 2016
“Education for Sale?” Linda Darling Hammond The Nation, March 27, 2017
“Betsy DeVos: Fighter for kids or destroyer of public schools?” Lori Higgins, Kathleen Gray, November 23, 2016, Detroit Free Press.
“The Betsy DeVos Hearing Was an Insult to Democracy” Charles Pierce, Esquire
“Betsy DeVos Wants to Use America’s Schools to Build ‘God’s Kingdom’” Kristina Rizga, Mar/Apr 2017 Issue, Mother Jones
“Why are Republicans so cruel to the poor? Paul Ryan’s profound hypocrisy stands for a deeper problem” Chauncey DeVega, March 23, 2017
In a recent review of two books, Education and the Commercial Mindset and School Choice: The End of Public Education, which appears in the December 8, 2016 New York Review of Books, Diane Ravitch, the former Assistant Secretary of Education during the George HW Bush presidency, discusses the implications of corporate designs on public education. Ravitch begins her review by reminding us that, “Privatization means that a public service is taken over by for-profit business whose highest goal is profit.” In the name of market-driven efficiency, she argues, the “education industry” is likely to become increasingly similar to privatized hospitals and prisons. In these industries, as in many others, corporate owners, in their loyalty to investors’ desire for profits, tend to eliminate unions, reduce employee benefits, continually cut costs of operation, and orient to serving those who are least expensive to serve.
Ravitch sketches some of the challenges posed by charter schools, noting that “…they can admit the students they want, exclude those they do not want, and push out the ones who do not meet their academic or behavioral standards.” She says, that charters not only “drain away resources from public schools” but they also “leave the neediest, most expensive students to the public schools to educate.” Moreover, as Josh Moon recently noted in his article, “’School choice’ is an awful choice,” “If the “failing school” is indeed so terrible that we’re willing to reroute tax money from it to a private institution that’s not even accredited, then what makes it OK for some students to attend that failing school?”
While some argue that charter schools can “save children from failing public schools” research on student outcomes for charter school has shown mixed results. For example, The Education Law Center, in “Charter School Achievement: Hype vs Evidence” reports:
Research on charter schools paints a mixed picture. A number of recent national studies have reached the same conclusion: charter schools do not, on average, show greater levels of student achievement, typically measured by standardized test scores, than public schools, and may even perform worse.
The Center for Research on Education Outcomes (CREDO) at Stanford University found in a 2009 report that 17% of charter schools outperformed their public school equivalents, while 37% of charter schools performed worse than regular local schools, and the rest were about the same. A 2010 study by Mathematica Policy Research found that, on average, charter middle schools that held lotteries were neither more nor less successful than regular middle schools in improving student achievement, behavior, or school progress. Among the charter schools considered in the study, more had statistically significant negative effects on student achievement than statistically significant positive effects. These findings are echoed in a number of other studies.
In Michigan, Secretary of Education, Betsy Devos’s home state, 80 percent of charter schools operate as for-profit organizations. Ravitch says, “They perform no better than public schools, and according to the Detroit Free Press, they make up a publicly subsidized $1 billion per year industry with no accountability.”
Ravitch tells us that the privatization movement is largely composed of social conservatives, corporations, and business-friendly foundations. “These days, those who call themselves “education reformers” are likely to be hedge fund managers, entrepreneurs, and billionaires, not educators. The “reform” movement loudly proclaims the failure of American public education and seeks to turn public dollars over to entrepreneurs, corporate chains, mom-and-pop operations, religious organizations, and almost anyone else who wants to open a school.”
The Trump administration is likely to further advance a public-school privatization and school voucher agenda. The extent and results of such “reforms” are hard to predict. That said, as Ravitch argues “… there is no evidence for the superiority of privatization in education. Privatization divides communities and diminishes commitment to that which we call the common good. When there is a public school system, citizens are obligated to pay taxes to support the education of all children in the community, even if they have no children in the schools themselves. We invest in public education because it is an investment in the future of society.” How continued privatization of public k-12 education will affect an increasingly economically privatized and socially and politically divided society is not yet known. To find out more about the work we do with schools click here.
One of the most important questions to consider before starting an evaluation is, “How will evaluation findings be used?” There are a number of ways the findings from an evaluation can be used. These include:
- to improve the program (formative evaluation)
- to make judgements about the ultimate value/benefits of the program to participants and stakeholders (summative evaluation)
- to sustain and/or expand the program
- to document and publicize the program’s achievements (outreach and marketing)
- to curtail the program
By being clear about the intended uses of evaluation findings, organizations can choose the most effective and stakeholder-relevant evaluation design. If, for example, a program funder wants to know whether their investment yielded desired outcomes for program participants, it may not be useful to focus the evaluation on gathering data that primarily indicate ways to improve the program’s delivery. On the other hand, if an organization is curious about how its programming is being implemented, and if such implementation is in fidelity with a research-based program design, the evaluation will want to focus more on implementation processes, rather than ultimate program outcomes.
Needless to say, stipulating the use of evaluation findings is closely related to the question of who are the major audiences for the evaluation. See our previous post, Identifying Evaluation Stakeholders. By clarifying potential audiences for findings, organizations and evaluators can identify the most useful evaluation strategy and design. In the next blog post we will discuss a related question, i.e., “What information will stakeholders find useful and valuable?” And to learn more about our data collection and evaluation click here.
- Who are the major stakeholders or interested parties for evaluation results?
- Who wants/needs to know?
- Who will benefit from knowing about the effectiveness of your program’s efforts?
- Make a list of key stakeholders and what they might want to know.
- Current funders
- Potential funders
- Community stakeholders
- Advocacy organizations
- Government agencies
- Colleagues/Internal organizational constituencies
- Peer organizations
- Program managers and program staff
- The “field”/general public
- Political allies
This is the first in a series of blog posts, that outlines the key elements of an evaluation.
There are many things in the world that can be evaluated. You will want to decide and define the specific object (i.e. the “evaluand”) of the evaluation. One useful approach is for program managers, and program implementers to speak with the evaluator, and together, consider these defining questions about what needs to be evaluated:
Whether you are managing a program, funding an initiative, or about to undertake a policy review, you will want to first consider:
What is the “it” that’s being evaluated?
- set of processes or relationships
Once you’ve identified what is being evaluated, it will be useful to consider these additional questions:
- What’s being done? What are the intentional actions that are being undertaken?
- Who does what to whom, and when do they do it?
- Why—for what set of reasons—are these things being done? Describe the need for the initiative or program?
- Who (which people, and which positions) are doing/carrying out the of the program, initiative, policy?
- What resources (i.e. inputs) are involved? (not just money and time, but knowledge, cooperation of others, networks of collaborators, etc.)
- Who (or what) are implementers working to change? What specifically is supposed to change, or be different as a result of the program or initiative doing what it does?
- Describe who benefits (e.g. program participants, beneficiaries, consumers, etc.) and what the benefits are.
By clearly identifying a determinate entity (i.e., “evaluand”) and the effects (i.e., outcomes) of the operation or implementation of that entity, you will help to ensure a successful evaluation. The clearer you can be in describing what the program, initiative, or policy is, the more effective the evaluation will be. Answering the above questions and sketching out a logic model of the program’s operation (which we’ll discuss in a subsequent blog post), will ensure an effective, accurate, and ultimately highly useful evaluation.
Also, for purposes of clarity, it may be useful for program managers and implementers to consider and discuss with the evaluator what isn’t being evaluated. This conversation can help to define the necessary boundaries around the program and will help to prevent evaluators expending unnecessary and potentially costly efforts to design evaluations of things that exist outside the boundary of the initiative, program or policy. To learn more about our evaluation practices visit our Impact & Assessment page.
What is ‘Organization Development’?
Organization Development is an intentional set of processes and practices designed to enhance the ability of organizations to meet their goals. It entails “…a process of continuous diagnosis, action planning, implementation and evaluation, with the goal of transferring (or generating) knowledge and skills so that organizations can improve their capacity for solving-problems and managing future change.”(See: Organizational Development Theory) Organization Development deals with a range of features, including organizational climate, organizational culture (i.e., assumptions, values, norms/expectations, patterns of behavior) and organizational strategy. It seeks to strengthen and enhance the long-term “health” and performance of an organization, often by focusing on aligning organizations with their rapidly changing and complex environments through organizational learning, knowledge management and transformation of organizational norms and values.
What is an organization?
At the most abstract level, an organization is a collectivity of people, a social entity, which seeks to achieve specific aims and goals, and typically is characterized by a structure of designated roles, established rules, a system or structure of authority, a process for decision-making, and a division of labor among organizational members. Organizations are composed of a discreet “membership,” that is, a limited population of incumbents or personnel. Organizations exist in, affect, and are affected by, the larger social and economic environment. Formal, large-scale organizations may take the form of businesses, schools, the military, churches, prisons, foundations, and non-profits.
The Relationship Between Organizations and Evaluation
In seeking to achieve goals, organizations often design and implement discreet initiatives, policies, and programs. They mobilize resources to achieve specific ends. Non-profit organizations, for example, implement programs that mobilize resources in the form of activities, services, and products that are intended to improve the lives of program participants/recipients. “A program is a collection of resources in an organization and is geared to accomplish a certain goal or set of goals. Programs are one major aspect of the non-profit’s structure. The typical non-profit organizational structure is built around programs, that is, the non-profit provides certain major services, each of which is usually formalized into a program.”(See: Overview of Non-Profit Program Planning). In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, services, and materials, to positively affect the lives of those they serve.
Although evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations often collect, synthesize, and report information that can be used to improve the broader operation and health of the organization that hosts the program. Program evaluation thus can contribute to organization development, the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency and to enable the organization to achieve its strategic goals.
Brad Rose Consulting Aids Organizational Development
Brad Rose Consulting works at the intersection of evaluation and organization development. While our projects often begin with a focus on discrete initiatives and programs, the questions that drive out evaluation research provide insights into the effectiveness of the organizations that host, design, and fund those programs. Findings from our evaluations often have important implications for the development and sustainability of the entire host organization. This is especially true in the case of small-to-medium sized nonprofit organizations and educational organizations, whose core programs often comprise the bulk of the organization’s structure and raison d’être. Information from our evaluations can be used to clarify the organization’s goals and objectives, to identify key organizational challenges and help to develop ways to address these, and to strengthen the overall effectiveness of the organization’s efforts. Additionally, Brad Rose Consulting’s evaluations offer an ideal opportunity for an organization to reflect on its practices and purposes, to rethink ways to achieve the organization’s mission, and to identify new data-based strategies for enhancing the organization’s long-term viability and well-being. To learn more about our program development practice visit our Program development & Funding page.
See our previous post: Helpful Resources: Program Evaluation Supports Strategic Planning
Overview of Nonprofit Program Planning by Carter McNamara
Group facilitation is a critical element in many evaluations. Although the definition of “facilitation” varies, one view of it is, “A process in which a person whose selection is acceptable to all members of the group, who is substantively neutral, and who has no substantive decision-making authority, diagnoses and intervenes to help a group improve how it identifies and solves problems and makes decisions…” (R. Schwartz “The Skilled Facilitator Approach,” in Schwartz and Davidson, eds., The Skilled Facilitator Handbook, Jossey-Baas, 2005). Correlatively, a facilitator is, “an individual who enables groups and organizations to work more effectively; to collaborate and achieve synergy. He or she is a ‘content neutral’ party who by not taking sides or expressing or advocating a point of view during the meeting, can advocate for fair, open, and inclusive procedures to accomplish the group’s work” (Kaner, S. with Lind, L., Toldi, C., Fisk, S. and Berger, D. Facilitator’s Guide to Participatory Decision-Making, (2007) Jossey-Bass).
In the Spring 2016 issue of New Directions in Evaluation, which is dedicated to the subject of evaluation and facilitation, the editors Rita Sinorita Fierro, Alissa Schwartz, and Dawn Hanson Smart, note the multiple uses that facilitation can play in evaluations. “Many evaluations are undertaken with groups, and require evaluators to play a facilitative role—to engage stakeholders in meaningful conversations, to structure these discussions in ways that surface multiple perspectives, and to conduct focus groups on key data and make progress toward next steps or decision-making. Facilitation can help groups map theories of change, undertake data collection through focus groups or other dialogues, participate in analysis of findings, and help craft appropriate recommendations based on these findings.”
In “Facilitating Evaluation to Lead to Meaningful Change,” (which appears in the same issue, pp. 19-29) Tessie Tzavaras Catsambas writes that, “…evaluations typically require the evaluator to work with groups of people for organizing and managing an evaluation, as well as in collecting and analyzing data… evaluators must frequently facilitate group interactions to conclude an evaluation successfully.” Additionally, Tzavaras Catsambas observes, “Evaluators may often find themselves in situations in which they have to ‘facilitate’ their way through political sensitivities, misunderstandings, competing interests, confusion, apathy or refusal to cooperate.”
Although evaluation and facilitation share many of the same principles and practices, including: respect, asking good questions, engagement of/with others, and promoting effective communication and participation, the most effective facilitations and evaluations require even deeper capacities and skills. Among these: the ability to sense the mood of others, to listen deeply, to capture group-generated information, to reflect, and to move both individuals and groups toward action. Effective facilitation and evaluation also require the ability of the evaluator to build and maintain constructive relationships, to engage stakeholders, to incorporate the perspective and views of others, and to work with integrity and objectivity. (See also “Interpersonal Skills Enhance Program Evaluation” and “Establishing Essential Competencies for Evaluators” by L. Stevahn, J. King, G. Ghere, and J. Minnema). While evaluations are of course, distinct from group facilitation, facilitation can be a critical tool for evaluators. To find out about our evaluation methods visit our Data collection & Outcome measurement page.
What Happens When There’s No More Work? (7/6/2016)
In the June 25- July 1, 2016 issue of New Scientist, Michael Bond and Joshua Howgego report that a recent study by Oxford University concludes that within two decades, one-half of all jobs in the US could be done by machines. Artificial intelligence (AI) and advanced automation are having a profound effect on work and employment, especially in the advanced industrial economies. (See “When Machine’s Take Over: What Will Humans Do When Computers Run the World?” New Scientist, June 25- July 1, 2016, Vol. 230, Issue 3079, p29 &ff.)
Martin Ford’s 2015 book, Rise of the Robots: Technology and the Threat of a Jobless Future, explores in greater depth the impact that AI and robotics on employment. Ford traces the powerful (and disturbing) effects of robotization and artificial intelligence on a range of sectors in the economy, and argues that in addition to job elimination, the current AI-driven revolution in the world of work promises to displace both blue collar, manual laborers and white collar, college-educated professionals—the latter including but not limited to, lawyers,computer programmers, managers, office and retail workers. The current and anticipated “rise of the robots” thus threatens to create an increasingly jobless future for all; a future, Ford argues, that cannot be addressed with more education and upskilling of the workforce, because those jobs for which displaced blue collar workers once retrained increasingly will be carried out by robots and smart machines.
Ford’s book, as does Bond and Howgego’s article, underscore both the ominous changes in the economy and the profound losses that such changes portend. Bond and Howgego’s article explores the significant role work has played, especially in the advanced economies—not only as source of income and livelihood, but also as an important source of employees’ sense of purpose, identity, and meaning. For instance, they cite a recent Gallop poll that shows that 50% of manual workers, and 70% of college educated employees report that they get a sense of identify from their jobs. They also discuss the health benefits associated with the performance of meaningful work, and how the risks of diseases such as dementia and Alzheimer’s may be reduced for those who work more years, and postpone retirement.
As work continues to change because of employers’ preference for AI and automation, and fewer people are able to find employment, how will society deal with what looks like an imminent if not current, tidal wave of unemployment and forced ‘leisure’? Ford shows how recent history has been characterized by diminishing job creation, lengthening jobless recoveries, and soaring long-term unemployment—all of which are certain to lead to significant social and economic results—if not adequately addressed.
Ford, Bond, and Howgego all suggest that society will necessarily need to rethink the distribution of wealth and society’s assets. Ford, for example, argues for a guaranteed basic income of 10,000 dollars annually for all citizens (augmentable of course, by paid employment), and says that if the guaranteed income was not set too high, it would be likely to avoid the pitfalls of creating disincentives to work. He estimates such a plan would cost about 2 trillion dollars annually—about one half of which would be recouped through cost savings on discontinued welfare programs (e.g., food stamps, housing assistance programs, Earned Income Tax Credits, etc.) and the other half which might be raised by new taxes, like a carbon tax. Bond, and Howgego also explore basic incomes, but also discuss alternative income-supporting plans, such as a negative tax program, in which poor people receive a guaranteed annual income, middle earners aren’t taxed, and the wealthy are taxed.
Whether society is culturally and politically ready for the introduction of a guaranteed minimum income remains to be seen. Ominously, current and forthcoming changes in work and the resulting displacement of workers is likely to necessitate a sweeping examination of the economic and moral implications of the disappearance of paid employment. AI and robotic technology, as these writers convincingly show, will continue to eliminate jobs and make human employment increasingly rare.
“When Machine’s Take Over: What Will Humans Do When Computers Run the World,” Michael Bond and Joshua,” New Scientist, Vol 230, Issue 3079, p29 &ff.)
Rise of the Robots: Technology and the Threat of a Jobless Future, Martin Ford, Basic Books, 2015
Inventing the Future: Post-Capitalism and a World Without Work, Nick Srnicek and Alex Williams, Verso, 2016
“Evidence-based” – What is it?
“Evidenced-based” has become a common adjectival term for identifying and endorsing the effectiveness of various programs and practices in fields ranging from medicine to education, from psychology to nursing, from criminal justice to psychology. The motivation for marshalling objective evidence in order to guide practices and policies in these diverse fields has been the result of the growing recognition that professional practices—whether they be doctoring or teaching, social work, or nursing—need to be based on something more sound than custom/tradition, practitioners’ habit, professional culture, received wisdom, and hearsay.
What does “evidenced-based” mean?
While definitions of “evidence-based” vary, the most common characteristics of evidence-based research include: objective, empirical research that is valid and replicable, whose findings are based on a strong theoretical foundation, and include high quality data and data collection procedures. The most common definition of Evidence-Based Practice (EBP) is drawn from Dr. David Sackett’s, original (1996) definition of “evidence-based” practices in medicine, i.e., “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research” (Sackett D, 1996). This definition was subsequently amended to, “a systematic approach to clinical problem solving which allows the integration of the best available research evidence with clinical expertise and patient values” (Sackett DL, Strauss SE, Richardson WS, et al. Evidence-based medicine: how to practice and teach EBM. London: Churchill-Livingstone,2000). (See, “Definition of Evidenced-based Medicine”).
An evidenced-based program, whether it be in youth development or education, is comprised of a set of coordinated services/activities that demonstrate effectiveness, as such effectiveness has been established by sound research, preferably, scientifically based research. (See, “Introduction to Evidence-Based Practice“).
In education, evidence-based practices are those practices that are based on sound research that shows that desired outcomes follow from the employment of such practices. “Evidence-based education is a paradigm by which education stakeholders use empirical evidence to make informed decisions about education interventions (policies, practices, and programs). ‘Evidence-based’ decision making is emphasized over ‘opinion-based’ decision making.” Additionally, “the concept behind evidence-based approaches is that education interventions should be evaluated to prove whether they work, and the results should be fed back to influence practice. Research is connected to day-to-day practice, and individualistic and personal approaches give way to testing and scientific rigor.” (See, “What is Evidence-Based Education?“).
Of course there are different kinds of evidence that can be used to show that practices, programs, and policies are effective. In a subsequent blog post I will discuss the range of evidence-based studies—from individual case studies and quasi experimental designs, to randomized controlled trials (RCT). The quality of the evidence as well as the quality of the study in which such evidence appears is a critical factor in deciding whether the practice or program is not just “evidence-based,” but in fact, effective. To learn more about our data collection and measurement click here.
Nonprofit organizations and program evaluators are increasingly being called upon to evaluate multi-stakeholder initiatives. These initiatives often depend upon collaboration among various organizations and agencies. As a consequence, the need to evaluate collaborations/coalitions has become an important requirement for a wide-range of contemporary program evaluations.
In a recent article, “Evaluating Collaboration for Effectiveness: Conceptualization and Measurement,” (American Journal of Evaluation, 2015, Vol. 36, p. 67-85) Lydia I. Marek, Donna-Jean Brock and Jyoti Savla discuss the customary difficulties in assessing collaborations, including, “lack of validated tools, undefined or immeasurable community outcomes, the dynamic nature of coalitions, and the length of time typical for interventions to effect community outcomes…the diversity and complexity of collaborations and the increasingly intricate political and organizational structures that pose challenges for evaluation design” (p.68).
Building on previous research by Mattesich and Monesy (Collaboration: What Makes it Work? Fieldstone Alliance, 1992) which outlines a six category inventory of characteristics typical of collaborations ( i.e., environment, membership characteristics, process/structure, communication, purpose, and resources), Marek, et al., argue for seven key factors that typify successful inter-organizational collaborations and coalitions:
1. Context—the shared history among coalition partners, the context in which they function, and the coalition’s role within the community
2. Membership—individual coalition members’ skills, attitudes, and beliefs that together contribute to, or detract from, successful outcomes
3. Process and organizational factors like flexibility and adaptability of members, and members’ clear understanding of their roles and responsibilities
4. Communication—formal and informal communication among members, and communication with the community
5. Function—the determination and articulation of coalition goals
6. Resources—the coordination of financial and human resources required for the coalition to achieve its goals
7. Leadership—strong leadership skills, including organizing and relationship – building skills
Marek et al. offer a tool for assessing the effectiveness of collaboration, a 69 item Collaboration Assessment Tool (CAT) survey instrument, which asks a series of questions for each of the factors identified above. For example, the “function” domain of the survey asks organizational respondents the following questions:
- This coalition has clearly defined the problem that it wished to address
- The goals and objectives of the coalition are based upon key community needs
- This coalition has clearly defined short-term goals and objectives
- This coalition has clearly defined long-term goals and objectives
- Members agree upon the goals and objectives
- The goals and objectives set for this coalition can be realistically attained
- Members view themselves as interdependent in achieving the goals and objectives of this coalition
- The goals and objectives of this coalition differ, at least in part, from the goals and objectives of each of the coalition members
“Evaluating Collaboration for Effectiveness: Conceptualizations and Measurement,” offers a valuable tool for assessing the often complex inter-organizational relationships that constitute multi-stakeholder collaborations. The CAT offers a system for gathering data for evaluating the quality of collaborations, and implicitly suggests an outline of the key characteristics of successful collaborations. The CAT tool will be useful for organizations who are embarking on collaborations, and for program evaluators who are charged with evaluating the success of such inter-organizational collaborations. To learn about our evaluation methods visit our Data collection & Outcome measurement page.
“Evaluating Collaboration for Effectiveness: Conceptualization and Measurement,” Lydia I. Marek, Donna-Jean Brock and Jyoti Savla, (American Journal of Evaluation, 2015, Vol. 36, p. 67-85)
Dawn Bentley of the Michigan Association of Special Educators recently drew my attention to an important article appearing in the Huffington Post, “Proven Programs vs. Local Evidence,” by Robert Slavin, of Johns Hopkins University. “Proven Programs vs. Local Evidence” compares and contrasts two kinds of evaluations of educational programs.
On the one hand, Slavin says, there are evaluations that are conducted of large-scale, typically federally funded programs. These programs represent program structures, that once found to be effective, can be replicated in a variety of settings. Evaluation findings from such programs are usually generalizable, that is they are applicable to a broader range of contexts than the individual case under study. Slavin terms such evaluations, “Proven Programs.” “Proven Program” evaluations are becoming increasingly important because the federal government is interested in funding efforts that are researched-based and show strong evidence of effectiveness. Examples of such “Proven Programs” include School Improvement Grants (SIGs) and Title II Seed grants.
On the other hand, Slavin notes, there are locally specific evaluations that are “not intended to produce answers to universals problems,” and whose findings typically are not generalizable. These evaluations are conducted on programs of a more limited, usually local, scope, and tend to be of interest principally to local program stakeholders, rather than state or national policy makers. Slavin calls these evaluations, “Local Evidence” because they yield evidence that typically isn’t generalizable to larger contexts.
Slavin notes that these two kinds of program evaluations are not necessarily mutually exclusive, for example, when a district or state implements and evaluates a replicable program that responds to its own needs. That said, Slavin says that “proven programs” are likely to contribute to national evidence and experience of what works, while “Local Evidence” evaluations are more likely to be of interest to local educators and local stakeholders. He notes that “Local Evidence” evaluations are more likely to result in stakeholders utilizing and acting on evaluation findings.
While Brad Rose Consulting, Inc. has experience in working with the U.S. Dept. of Education in conducting evaluations of national scope initiatives, we are also have extensive experience and are strongly committed to assisting state-level and district-level education agencies to design and conduct evaluation research to produce evaluation findings that will constructively inform both local policy and programming innovations. To find out more about our work in education visit our Higher education & K-12 page.
Qualitative interviews can be an important source of program evaluation data. Both in-depth individual interviews and focus group interviews are important methods that provide insights and phenomenologically rich descriptive information that other, numerically oriented data collection methods (e.g., questionnaires and surveys), are often unable to capture. Typically, interviews are either structured, semi-structured, or unstructured. (Structured interviews are, essentially, verbally administered questionnaires, in which a list of predetermined questions is asked. Semi-structured interviews consist of several key questions that help to define the areas to be explored, but allow the interviewer to diverge from the questions in order to pursue or follow up to an idea or response. Unstructured interviews start with an opening question, but don’t employ a detailed interview protocol, favoring instead the interviewer’s spontaneous generation of subsequent follow-up questions. See P. Gill1, K. Stewart2, E. Treasure3 & B. Chadwick, cited below.)
Interviews, especially one-on-one, in-person, interviews, allow an evaluator to participate in direct conversations with program participants, program staff, community members, and other program stakeholders. These conversations enable the evaluator to learn about interviewees’ experiences, perspectives, attitudes, motivations, beliefs, personal history, and knowledge. Interviews can be the source of pertinent information that is often unavailable to other methodological approaches. Qualitative interviews enable evaluators to: elicit direct responses from interviewees, probe and ask follow-up questions, gather rich and detailed descriptive data, offer real-time opportunities to explain and clarify interview questions, observe the affective responses of interviewees, and ultimately, to conduct thorough-going explorations of topics and issues critical to the evaluation initiative.
Although less intimate, focus group interviews are another key source of evaluation data. Organized around a set of guiding questions, focus groups typically are composed of 6-10 people and a moderator who poses open-ended questions to focus group participants. Focus groups usually include people who are somewhat similar in characteristics or social roles. Participants are selected for their knowledge, reflectiveness, and willingness to engage topics or questions. Ideally – although not always possible – it is best to involve participants who don’t previously know one another. Focus groups can be especially useful in clarifying, qualifying, and/or challenging data collected through other methods. Consequently they can be useful as a tool for corroborating findings from other research methodologies.
Regardless of the specific form of qualitative interview—individual or focus group, in-person or telephone—qualitative interviews can be useful in providing data about participants’ experience in, and ultimately the effectiveness of, programs and initiatives. To find out more about our use of qualitative data visit our Data collection & Outcome measurement page.
Learning from Strangers: The Art and Method of Qualitative Interview Studies, Robert S. Weiss, Free Press.
Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences, Irving Seidman. Fourth Edition.
With the end of the year fast approaching, this is good time to review the array of online resources that we’ve shared in recent months with non-profit leaders, foundation staff, educators, and fellow program evaluators. The links below represent the most popular blog posts of 2015. If you haven’t yet had a chance to look at one or more of these, please feel free to do so. You’re also welcome to share these links with colleagues and to keep these links on hand for future reference.
There are also a number of additional program evaluation resources on the site.
We are looking forward to offering more thoughts and additional evaluation resources in 2016!
Below is a selection of web-based program evaluation resources that we hope will be helpful to you. These may be useful to experienced evaluators, new evaluators, and non-evaluators.
What is Program Evaluation and What are the Different Types of Evaluation?
Theory of Change
Video: “How to Develop a Theory of Change”
Video: “DIY—Theory of Change
“Needs Assessment: Trends and a View Toward the Future,” New Directions in Evaluation, No. 144, Winter, 2014 James W. Altschuld and Ryan Watkins (eds.)
Additional Resources hosted by the American Evaluation Association:
George Orwell once wrote that it is sometimes the duty of intelligent men (sic) to re-state the obvious. In support of Orwell’s dictum, I’d like to restate in this blogpost something that we all know: despite our hopes for the power of education to correct for social disadvantage and inequality—to transform children’s life chances—persistent childhood poverty negatively affects children’s educational outcomes. I raise this issue because it has been my experience in working with educators that they sometimes sanguinely believe that education can overcome the deleterious effects of poverty. Sadly, research shows that educational attainment is stunted by poverty. Among research findings:
- The gap in standardized test scores between affluent and low-income students has grown by about 40 percent since the 1960s, and is now double the testing gap between blacks and whites.
- 22% of US children live in poverty.
- 40% of children living in poverty aren’t prepared for primary schooling.
- Children who live below the poverty line are 1.3 times more likely to have developmental delays or learning disabilities than those who don’t live in poverty.
- Between birth and age six, wealthier children will have spent as many as 1,300 more hours than poor children on child enrichment activities such as music lessons, travel, and summer camp.
- By the end of the 4th grade, African-American, Hispanic and low-income students are already 2 years behind grade level. By the time they reach the 12th grade they are 4 years behind.
- Dropout rates of 16 to 24-years-old students who come from low income families are seven times more likely to drop out than those from families with higher incomes.
- A child from a poor family is two to four times as likely as a child from an affluent family to have classmates with low skills and behavior problems – attributes which have a negative effect on the learning of their fellow students.
- International data shows that compared with other economically advanced countries, only Romania has a higher child poverty rate than the United States.
The implication of these facts, of course, is not that we (American society) should abandon our long-held commitment to educating all children, but that we should disabuse ourselves of the idea that education alone can solve the social problems of disadvantage and inequality. We have looked to public education to solve many societal problems, but the fact is that as these deepen and multiply—from unemployment to growing racial inequality— the institution of education can’t rectify these on its own. Our commitment to the highest quality public education should be a moral and political one, not a delusional one. If you want to learn more about our commitment to improving education visit this page.
What is a Theory of Change?
A theory of change is a method for planning, monitoring, and evaluating initiatives in the non-profit, philanthropic, and government sectors. A theory of change articulates and graphically illustrates the assumptions that inform a change initiative, the prospective set of changes the initiative hopes to make, and the logical and chronological order in which causes and anticipated outcomes will occur. Theories of change ask that program planners, supporters, staff, and in some cases, participants, outline the causal pathway between an initiative’s actions and its ultimate goals. Although a theory of change is like a logic model, a theory of change differs from a logic model in so far as it is a graphic representation of the steps necessary to address a given problem or issue. A logic model is typically more focused on a specific program’s deployment of resources to produce specified outcomes.
Typically, theories of change include an explanation of how and why anticipated changes will occur, rather than simply mapping the relationship among inputs, outputs, and outcomes. Consequently, theories of change are explanatory, while logic models are descriptive. Clark and Anderson argue that logic models “usually start with a program and illustrate its components” while “theories of change (work) best when starting with a goal, before deciding which programmatic approaches are needed.” (For more information on the differences between a logic model and a theory of change see, Clark and Anderson’s “Theories of Change and Logic Models: Telling Them Apart”).
Patricia Rogers writes, “A ‘theory of change’ explains how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a program, a policy, a strategy or an organization” (see UNICEF—Theory of Change).
Why Develop a Theory of Change?
Everyone who is involved in a change process has an implicit idea of how changes come about. Collaboratively developing a theory of change can help stakeholders to build a shared understanding of the causal steps (and the underlying implicit assumptions) that are prerequisite to achieving a desired goal. A theory of change can illustrate the causal logic in a way that says “if we do ‘x’, then ‘y’ is likely to occur.” Essentially, theories of change are a diagram of how things should work, the resources needed to achieve a goal or set of goals, and the causal relationships between various factors that lead to the achievement of a goal.
Reasons for developing a theory of change also include:
- assisting stakeholders to articulate how and why a change initiative will work
- graphically representing the implicit and often tacit causal mechanisms at work in a service delivery system or programmatic initiative
- enhancing an understanding of how information can be used to measure or indicate changes (outcomes and impacts)
- clarifying for stakeholders why things may be working and not working
- building a consensus about how an initiative should work, so that it reaches its goals
What is a theory of change?
Annie E. Casey Foundation: Theory of Change: A Practical Tool For Action, Results, and Learning
Video: “How to Develop a Theory of Change”
Using a Theory of Change in Program Design and Planning
Video: “DIY—Theory of Change
Theory of Change—Wikipedia
Unicef—Theory of Change
A Community Builder’s Approach to a Theory of Change
Differences between a Logic Model and a Theory of Change
Theory of Change Portal (Netherlands)
Harvard Family Research Institute—Theory of change
Theory of Change in the Philanthropic Sector
I like quotes and have collected them for a number of years. Quotes from literature, film, newspapers, and other sources, often capture in a few insightful words, the way the world really works or how it should work. Below are some quotes that maybe useful to you and your colleagues, who manage, staff, and fund programs.
Flying is learning how to throw yourself at the ground and miss.
— Douglas Adams
Courage is resistance to fear, mastery of fear– not absence of fear.
— Mark Twain
One doesn’t discover new lands without consenting to lose sight of the shore for a very long time.
— Andre Gide
You miss 100% of the shots you don’t take.
— Wayne Gretzky
Not everything that counts can be counted, and not everything that can be counted, counts.
— Albert Einstein
I myself am human and free only to the extent that I acknowledge the humanity and liberty of all my fellows… I am properly free when all the men and women about me are equally free. Far from being a limitation or a denial of my liberty, the liberty of another is its necessary condition and confirmation.
— Michael Bakunin
An optimist proclaims that we live in the best of all possible worlds; the pessimist fears this is true.
— James Bench Cabell
The struggle itself towards the heights is enough to fill a human heart. One must imagine that Sisyphus is happy.
— Albert Camus
Ignorance more frequently begets confidence than does knowledge.
— Charles Darwin
Sixty years ago I knew everything; now I know nothing; education is the progressive discovery of our ignorance.
— Will Durant
An error can never become true however many times you repeat it. The truth can never be wrong, even if no one hears it.
— Mahatma Gandhi
It is well to remember that the entire universe, with one trifling exception, is composed of others.
— John Andrew Holmes
One of my favorite quotes is from Butch Cassidy and the Sundance Kid. You can see the one minute scene here:
How do non-profits use program evaluation? What are the trends and what are the challenges for evaluation in the non-profit sector? A recent report by the Center for Effective Philanthropy, based on information drawn from 514 non-profit organizations found that:
- 56% of nonprofits collect information from programmatic assessments, or indicators of outcomes they seek to change.
- 43% collect Information about their organization’s reach: number of beneficiaries served or units of service provided.
- While 90% devote some portion of their budget toward performance assessment, 55% spend 2% or less of their budget on that effort.
- Nearly one-third of nonprofits use third-party evaluators to conduct formal assessments of their performance, and those that do typically have annual expenses greater than $1.4 million.
- 83% of nonprofits reported that they are using their performance information to improve their programs and services at least “to a great extent,” and 68%use their information to inform their strategic direction at least “to a great extent.”
- Only 36% reported that they tend to receive financial and/or non-monetary support from their foundation funders to help assess their performance, with 64% reporting they receive no such support.
As we’ve discussed in earlier blog posts, evaluation can be an important tool for helping non-profits’ to understand the positive effects they are achieving and the differences they are making. As the above study indicates, evaluation can also be a important source of information for program refinement and determining strategic direction. To learn about our evaluation methods visit our Data collection & Outcome measurement page.
Or for more discussion about the various uses and benefits of program evaluation for non-profits, please see our articles, “Evaluation Serves Community Foundations and Donors,” “Program Evaluation Supports Strategic Planning” and “Program Evaluation and Organization Development.”
Periodically, I share with you and your colleagues information and resources that are of special interest to the non-profit sector. In May, a colleague of mine, Matt Von Hendy, at Green Heron Information, will be offering a series of webinars that will help you and your organization to:
- Effectively search for, find, and respond to government Requests for Proposals (RFPs);
- Improve your on-line informational web search strategies; and
- Locate high quality, health-related information with an emphasis on evidence-based medical resources.
Links to registration for each of these webinars and more information about each, appear below:
Supercharge Your Search 2015 — Tuesday, May 5, 2-3 PM EDT
In the age of Google and Web 3.0, everyone is a researcher but the amount of information can be overwhelming. How do you minimize the amount of time spent looking at useless results just to get the right high quality information that you need? This webinar is designed to give you the strategies, techniques, and tools to help you find and organize twice the amount of high quality information in half the time. Click here to register.
Searching For and Finding RFPs (Government Contracting Opportunities): A Systematic Research Based Approach — Tuesday, May 12, 2-3 PM EDT
Searching for and finding RFPs (request for proposals) from government agencies can be a frustrating and time-consuming process. This webinar is designed to equip you with the strategies, techniques, and tools to help you more quickly and efficiently search for and locate federal, state, tribal and local government contracting opportunities. Click here to register.
Finding High Quality Health (Evidence Based) Resources on a Deadline and a Budget — Tuesday, May 19, 2-3 PM EDT
With increasing frequency, project sponsors are asking grantees to use high quality health resources in their proposals. Locating high quality health-related information can be a challenge – especially when you have an impending deadline and/or have a limited budget. This webinar covers strategies, resources and tools to quickly and effectively locate high quality low-cost health information with an emphasis on evidence-based medical (ebm) resources. Click here to register.
Each session costs $40 dollars, which covers all the standard webinar material AND also includes a 30 minute personalized post webinar session with Matt, where you can discuss your particular research needs.
About the presenter:
Matthew Von Hendy MA/MLS has more than 20 years of experience as a professional research librarian and has worked at EPA, NASA, and the National Academies of Science. In 2012, he established his own research and information consulting firm, Green Heron Information Services, and has been working closely with evaluation and other research professionals ever since. He frequently presents at local, regional, and national evaluation conferences.
Have questions or need more information? Contact Matt Von Hendy – firstname.lastname@example.org
You will recall that in an earlier post we discussed the importance of learning from program “failure.” (“Fail Forward: What We Can Learn from Program Failure”)
Below are some films and other resources about the value of risk and failure in helping us to learn and improve.
“I have not failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison
“By understanding how and why programs don’t achieve the results they intend, we can design and execute improved programs in the future. It is important to note that psychological research has shown that individuals learn more from failure than they do from success. Our goals should be to learn from our defeats and to surmount them—especially in programs that address critical social, educational, and human service needs. Learning from the challenges that confront these kinds of programs can have a powerful impact of the success of future programming.” To learn about how we take advantage of feedback visit our Feedback & Continuous improvement page.
Before beginning an evaluation, it may be helpful to consider the following questions:
1. Why is the evaluation being conducted? What is/are the purpose(s) of the evaluation?
Common reasons for conducting an evaluation are to:
- monitor progress of program implementation and provide formative feedback to designers and program managers (i.e., a formative evaluation seeks to discover what is happening and why, for the purpose of program improvement and refinement.)
- measure final outcomes or effects produced by the program (i.e., a summative evaluation);
- provide evidence of a program’s achievements to current or future funders;
- convince skeptics or opponents of the value of the program;
- elucidate important lessons and contribute to public knowledge;
- tell a meaningful and important story;
- provide information on program efficiency;
- neutrally and impartially document the changes produced in clients or systems;
- fulfill contractual obligations;
- advocate for the expansion or reduction of a program with current and/or additional funders.
Evaluations may simultaneously serve many purposes. For the purpose of clarity and to ensure that evaluation findings meet the client’s and stakeholders’ needs, the client and evaluator may want to identify and rank the top two or three reasons for conducting the evaluation. Clarifying the purpose(s) of the evaluation early in the process will maximize the usefulness of the
2. What is the “it” that is being evaluated? (A program, initiative, organization, network, set of processes or relationships, services, activities?) There are many things that may be evaluated in any given program or intervention. It may be best to start with a few (2-4) key questions and concerns (See #4, below ). Also, for purposes of clarity, it may be useful to discuss what isn’t being evaluated.
3. What are the intended outcomes that the program or intervention intends to produce? What is the program meant to achieve? What changes or differences does the program hope to produce, and in whom? What will be different as the result of the program or intervention? Please note that changes can occur in individuals, organizations, communities, and other social environments. While evaluations often look for changes in persons, changes need not be restricted to alterations in individuals’ behavior, attitudes, or knowledge, but can extend to larger units of analysis, like changes in organizations, networks of organizations, and communities. For collective groups or institutions, changes may occur in: policies, positions, vision/mission, collective actions, communication, overall effectiveness, public perception, etc. For individuals: changes may occur in behaviors, attitudes, skills, ideas, competencies, etc. If you to learn more about out evaluation practice visit our Impact & Assessment reporting page.
How do programs know what they should be doing— which target populations require services, the types of services programs should provide, the amounts of services, which kinds of services will be most effective, etc.? Needs assessments are the best way to determine the needs of individuals, communities, and other populations. A needs assessment is a systematic process for identifying and determining such needs. Like program evaluations, needs assessments draw on a range of social science methods—from surveys and observations, to focus groups and individual interviews.
Needs assessments assume a clear definition of “a need”. As James Altschuld and Ryan Watkins point out in New Directions in Evaluation, No. 144, Winter, 2014 “A need, in the simplest sense, is a measurable gap between two conditions: what currently is, and what should be….This requires ascertaining what the circumstances are at a point in time, what is desired in the future, and a comparison of the two.” Needs assessments don’t just exclusively focus on what is and should be, but also on gathering and synthesizing data about how to narrow the gap between the existing state and the desired state. Needs assessments also prioritize needs so that users of the assessment can address specified needs in a reasonable order, and devote appropriate resources to meeting identified needs.
By gathering data from a range of stakeholders, needs assessments are able to determine the best means to achieve the desired results. To be effective, however, needs assessments must not simply focus on deficits in individuals and communities, but must also explore existing strengths, capacities, and assets. Too narrow a focus on “what’s missing” can blind researchers and program designers to the existing assets on which effective programming can be built. Effective needs assessments therefore, ask questions about: 1) on-going needs, 2) current strengths/assets/ capacities, 3) and desired states.
Needs assessments may differ in their design, but regardless of design, most needs assessment follow these phases:
- Explore and gather data about the current condition/state of affairs (including existing assets)
- Explore/identify desired or optimal condition/state of affairs
- Analyze data to understand the difference or “gap” between the current condition and desired condition.
- Prioritize identified needs and “gaps.”
- With needs (and assets) in mind, design program to address (diminish or eliminate) the gap between existing needs and desired state.
When conducted in a timely and thoughtful way, needs assessments can be of substantial utility in helping programs to effectively deliver services to those who most need them. To find out more about our assessment methods visit our Impact & Assessment reporting page.
“Needs Assessment: Trends and a View Toward the Future,” New Directions in Evaluation, No. 144, Winter, 2014 James W. Altschuld and Ryan Watkins (eds.)
Definition of Needs Assessment
Comprehensive Needs Assessment
Methods for Conducting an Educational Needs Assessment
Program evaluations are conducted for a variety of reasons. Purposes can range from a mechanical compliance with a funder’s reporting requirements, to the genuine desire by program managers and stakeholder to learn “Are we making a difference?” and if so, “What kind of difference are we making?” The different purposes of, and motivations for, conducting evaluations determine the different types of evaluations. Below, I briefly discuss the variety of evaluation types.
Formative, Summative, Process, Impact and Outcome Evaluations
Formative evaluations are evaluations whose primary purpose is to gather information that can be used to improve or strengthen the implementation of a program. Formative evaluations typically are conducted in the early- to mid-period of a program’s implementation. Summative evaluations are conducted near, or at, the end of a program or program cycle, and are intended to show whether or not the program has achieved its intended outcomes (i.e., intended effects on individuals, organizations, or communities) and to indicate the ultimate value, merit and worth of the program. Summative evaluations seek to determine whether the program should be continued, replicated or curtailed, whereas formative evaluations are intended to help program designers, managers, and implementers to address challenges to the program’s effectiveness.
Process evaluations, like formative evaluations, are conducted during the program’s early and mid-cycle phases of implementation. Typically process evaluations seek data with which to understand what’s actually going on in a program (what the program actually is and does), and whether intended service recipients are receiving the services they need. Process evaluations are, as the name implies, about the processes involved in delivering the program.
Impact evaluations sometimes alternatively called “outcome evaluations,” gather and analyze data to show the ultimate, often broader range, and longer lasting, effects of a program. An impact evaluation determines the causal effects of the program. This involves trying to measure if the program has achieved its intended outcomes. (see, for example) The International Initiative for Impact Evaluation (3ie) defines rigorous impact evaluations as: ”Analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context” (see, for example) Impact (and outcome evaluations) are primarily concerned with determining whether the effects of the program are the result of the program, or the result of some other extraneous factor(s). Ultimately, outcome evaluations want to answer the question, “What effect(s) did the program have on its participants (e.g., changes in knowledge, attitudes, behaviors, skills, practices) and were these effects the result of the program?
Although the different types of evaluation described above differ in their intended purposes and times of implementation, it is important to keep in mind that every program evaluation should be guided by good evaluation research questions. (See our earlier post, Questions Before Methods) Program evaluation, like any effective research project depends upon asking important and insight-producing questions. Ultimately, the different types of evaluations discussed above support the general definition of “program evaluation —a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency.” (see, for example) To learn more about our evaluation methods visit our Data collection & Outcome measurement page.
Understanding the Causes of Outcomes and Impacts—An AEA video webinar (18 minutes):
Introduction to Evaluation:
What is Program Evaluation:
What are Nonprofit Organizations?
Organizations are social entities that have a collective purpose and that interact with larger environments (the economy, society, other organizations, etc.) Nonprofit organizations are a sub-type of organization that use “surplus revenues” (i.e. revenues beyond those used for operation and sustenance) to achieve desirable social ends, rather than to produce profits or dividends. In essence, nonprofits use their financial resources (provided by individual donations, foundation and government grants, etc.) to ameliorate the lives and conditions of community members. In the US, there are a range of nonprofit organizations, including hospitals, charities, educational organizations, social welfare organizations, foundations, community organizations, etc. According to the National Center for Charitable Statistics, there are approximately 1.5 million nonprofits in the US.
Organizations vs. Programs
Many nonprofit organizations mobilize resources in the form of organizedprograms that provide activities and products that are designed to improve the lives of program participants. Carter MacNamara writes that, “A program is a collection of resources in an organization and is geared to accomplish a certain goal or set of goals. Programs are one major aspect of the non-profit’s structure. The typical non-profit organizational structure is built around programs, that is, the non-profit provides certain major services, each of which is usually formalized into a program.” (http://literacy.kent.edu/Oasis/grants/overviewprogplan.html) In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, and materials, to positively affect the lives of those they serve.
Program Evaluation Meets Organization Development
In order to assess the effectiveness and efficiency of programs, nonprofits will often conduct program evaluations. Program evaluations are customarily guided by a set of evaluation research questions, e.g., What are the effects of the program on participants? What challenges did participants encounter while in the program? Did the program make a difference in the lives of those it was intended to serve? Did the program cause the observed changed in program participants? etc. (for more examples of evaluation questions, see our previous posts “Questions Before Methods” and “Approaching an Evaluation: 10 Issues to Consider” ) Although program evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations often collect, synthesize, and report information that can useful in improving the broader operation and health of the organization that hosts the program. Program evaluation thus can contribute to organization development, “the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency and/or to enable the organization to achieve its strategic goals.” (wikipedia)
In fact, findings from program evaluations often have important implications for the development and sustainability of the entire host organization. This is especially true in the case of small-to-medium sized nonprofit organizations, whose core programs often comprise the bulk of the organization’s structure and raison d’être. Consequently, information from program evaluations—especially formative evaluations which focus on strengthening program effectiveness—can be used to clarify the organization’s goals and objectives, to identify key organizational challenges and ways to address these, and to strengthen the overall effectiveness of the organization’s efforts. Additionally, program evaluations can offer an ideal opportunity for an organization to reflect on its practices and purposes, to rethink ways to achieve the organization’s mission, and to identify new data-based strategies for enhancing the organization’s long-term viability and well-being. Ultimately, program evaluation can, and in many cases should, be an integral component of organization development. If you want to learn more about our methods for program development and evaluation visit our Program development & Funding page.
About Nonprofit Organizations:
Types of Non Profit Organizations in the US:
Overview of Nonprofit Program Planning by Carter McNamara:
Basic Guide to Nonprofit Program Design and Marketing:
What is Organization Development?
When discussing with clients potential sources of data about a program’s operations and effects, it has often been said to me, “But we just have anecdotal evidence.” It’s as if anecdotal data don’t count. Too often anecdotes are dismissed as unscientific and valueless—as if they are just stories. In point of fact, anecdotes (qualitative accounts, “word-based” data) can be a valuable source of information and offer powerful insights about how a program works and the effects it produces. When carefully collected and systematically analyzed, especially when combined with other sources of quantitative data, anecdotes can be a powerful “window” on a program.
In a recent blog post, (see link below) the evaluator Michael Quinn Patton reflects on the value and utility of anecdotal information. Patton shows that, when collected in sufficient quantity, compared (or “triangulated’) with other kinds of data, and systematically and sensibly analyzed, anecdotes can provide important information about the character and meaning of a given phenomenon. Furthermore, anecdotes are often the starting place for hypotheses and experiments that ultimately produce quantitative evidence of phenomena. William Trochim underscores the importance of word-based, qualitative data (of which anecdotes are a specific type) when he points out. “All quantitative data is based on qualitative judgment. Numbers in and of themselves can’t be interpreted without understanding the assumptions which underlie them…”(David Foster Wallace made a similar point from an entirely different vantage point, in Consider the Lobster: “You can’t escape language. Language is everything and everywhere. It’s what lets us have anything to do with one another. p. 70.)
Trochim goes on to say,
“All numerical information involves numerous judgments about what the number means. The bottom line here is that quantitative and qualitative data are, at some level, virtually inseparable. Neither exists in a vacuum or can be considered totally devoid of the other. To ask which is “better” or more “valid” or has greater “verisimilitude” or whatever ignores the intimate connection between them. To do good research we need to use both the qualitative and the quantitative.”
Patton reminds us of the importance of anecdotes when he quotes N.G. Carr, author of The Shallows: What the Internet is doing to our brains. W. W. Norton(2010). New York:
“We live anecdotally, proceeding from birth to death through a series of incidents, but scientists can be quick to dismiss the value of anecdotes. “Anecdotal” has become something of a curse word, at least when applied to research and other explorations of the real.. . . . The empirical, if it’s to provide anything like a full picture, needs to make room for both the statistical and the anecdotal.
The danger in scorning the anecdotal is that science gets too far removed from the actual experience of life, that it loses sight of the fact that mathematical averages and other such measures are always abstractions.”
I believe that it is important to use multiple kinds of information to understand what programs do, and what their outcomes are. Quantitative data is essential for understanding abstract trends and at getting at the “larger picture.” That said, it is nearly impossible to make sense of quantitative data without using language to reveal assumption, implications, explanations, and meaning of quantitative data. Anecdotal data, as one kind of qualitative data, is critical to effective program evaluation research. To learn more about how we utilize both quantitative and qualitative data visit our Data collection & Outcome measurement page.
Michael Quinn Patton, “Anecdote as Epithet – Rumination #1 from Qualitative Research and Evaluation Methods”
William Trochim, “The Qualitative Debate”
“The Qualitative- Quantitative Debate”
Video about Qualitative, Quantitative, and Mixed-Methods Research
A Table Summarizing Qualitative versus Quantitative Research: Key Points in a Classic Debate
Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research
Qualitative research interviews are a critical component of program evaluations. In-person and telephone interviews are especially valuable because they allow the evaluator to participate in direct conversations with program participants, program staff, community members, and other stakeholders. These conversations enable the evaluator to learn in a rich conversational venue about interviewees’ experiences, perspectives, attitudes, and knowledge. Unlike questionnaires and surveys, which typically require structured, categorical responses to standardized written questions so that data can be quantified, qualitative interviews allow for deeper probing of interviewees and the use of clarifying follow-up questions which can surface information that often remains unrevealed in survey/questionnaire formats.
Although research interviews are guided by a pre-determined, written protocol which contains guiding questions, excellent interviews require a nimble and improvisational interviewer who can thoughtfully and swiftly respond to interviewees’ observations and reflections. Qualitative research interviews also require that the interviewer be a skilled listener and thoughtful interpreter of verbally presented data. Interviewers must listen carefully both to the denotative narrative “text” of the interviewee, and to the connotative subtext (the implied intent, tacit sub-themes and connotations) that the interviewee presents.
The most productive qualitative interviews are those that approximate a good conversation. This requires the interviewer to establish a comfortable atmosphere; ask interesting and germane questions; display respect for the interviewee; and create a sense of equality, candor, and reciprocity between the interviewer and the interviewee. Good interviews not only are a source of rich and informative data for the interviewer, they can also be a reflective learning opportunity for the interviewee. Like every good conversation, both parties should benefit. To learn more about our qualitative evaluation methods visit our Data collection & Outcome measurement page.
Discussion of interview basics:
Tip Sheet for Qualitative Interviewing:
Advantages and disadvantages of different research methods (personal interviews, telephone surveys, mail surveys, etc.):
Interviews: An Introduction to Qualitative Research Interviewing, by Steinar Kvale, Sage Publications, Thousand Oaks California, 1996:
About research interviewing:
Interviewing in qualitative research
Interviewing in educational research:
In a recent New York Times article, “Why You Hate Work”
Tony Schwartz and Christin Porath discuss why employees’ experience of work has increasingly become an experience of depletion and “burnout.” The factors are many, including: 1) demands on employee time that far exceed employee capacity to meet demands; 2) a leaner and less populous workforce, and therefore more work distributed to fewer workers; and 3) technology-driven expectations for immediate response to requests for employees’ attention and commitment (think here of answering e-mails at 1:00 AM). The authors cite both national and international studies that indicate that workers at all levels of various kinds of organizations feel less engaged, less satisfied, and less fulfilled by their experience at work. Schwartz and Porath argue however, that when companies better address the physical, mental, emotional, and spiritual dimensions of their workers, they not only produce more engaged and fulfilled workers, but more productive and profitable organizations. Organizations can begin to do this by instituting simple changes like mandating meetings that last no longer than 90 minutes; rewarding managers that display empathy, care, and humility; and providing regular and frequent breaks so that employees can ‘recharge ‘and work more creatively. Successful companies provide opportunities for employee renewal, focus, emotional support, and sense of purpose. When companies provide such opportunities, companies, investors, and employees benefit.
How might program evaluation add to non-profit organizations’ efforts to create what Schwartz and Porath call “truly human-centered organizations” (which) put their people first….because they recognize that they are the key to creating long-term value.” While program evaluation, alone, cannot prevent employee burnout, timely and well-designed formative evaluations can add to non-profit organizations’ capacities to implement effective programming by providing insight into the unintended features of programs that often ‘get in the way’ of staff (and program participants’) sense of efficacy and purposefulness. By conducting formative evaluations—evaluations that focus on program strengthening and effectiveness-maximization— program evaluation can help organizations and funders to create programs in which staff and participants don’t have to “spin their wheels,” i.e., programs where both staff and participants can achieve a greater sense of effectiveness, purpose, and satisfaction.
Because Brad Rose Consulting, Inc. often works at the intersection of program evaluation and organization development (OD), we work with clients to collect data, understand the characteristics of programs, and provide evidence-based insights into how programs, and the organizations that support them, can become maximally effective. We make concrete recommendations that help our clients adjust their modes of operation and thereby increase staff engagement and better serve their participants/clients. While the latter are the reason programs exist, the former are often the under-recognized key to programs’ success. To learn more about our adaptive approach to program evaluation visit our Impact & Assessment reporting page.
In a recent, themed issue devoted to the topic of validity in program evaluation, the journal New Directions in Evaluation (No. 142, Summer, 2014) revisited and commented on Ernest House’s influential 1980 book, Evaluating with Validity. House then argued that validity in evaluation must not just be limited to classic scientific conceptions of the valid (i.e. empirically describing things as they are), but that it must also include an expanded dimension of argumentative validity, in which an evaluation “must be true, coherent, and just.” Paying particular attention to social context, House argued that “there is more to validity than getting the facts right.” He wrote that “…the validity of an evaluation depends upon whether the evaluation is true, credible, and normatively correct.” House ultimately argued that evaluations must make compelling and persuasive arguments about what is true (about a program) and thereby bring “truth, beauty, and justice” to the evaluation enterprise.
In the same issue of New Directions in Evaluation, in her essay, “How ‘Beauty’ Can Bring Truth and Justice to Life,” E. Jane Davidson argues that the process of creating a clear, compelling, and coherent evaluative story ( i.e., a “beautiful” narrative account) is the key to unlocking “validity (truth) and fairness (justice).” To briefly summarize, Davidson argues that a coherent evaluation story weaves together quantitate evidence, qualitative evidence, and clear evaluative reasoning to produce an account of what happened and the value of what happened. She says that an effective evaluation—one that is truly accessible, assumption-unearthing, and values-explicit—enables evaluators to “arrive at robust conclusions about not just what has happened, but how good, valuable, and important it is.” (P.31)
House’s book and Davidson’s essay highlight how effective evaluations—ones that allow us to clearly see and understand what has happened in a program— rely on strong narrative accounts that tell a coherent and revealing story. Evaluations are not just tables and data—although these are necessary parts of any evaluation narrative—they are true, compelling, and fact-based stories about what happened, why things happened the way they did, and what the value (and meaning) is of the things that happened.
“When I reflect on what has improved the quality of my own work in recent years, it has been a relentless push toward succinctness and crystal clarity while grappling with some quite complex, and difficult material. For me this means striving to produce simple direct and clear answers to evaluation questions and being utterly transparent in the reasoning I have used to get to those conclusions.” (p.39)
Davidson further observes that evaluation reports often are often plagued by confusing, long-winded, and academic jargon, that make them not only difficult to read, but obfuscate the often muddled and ill-reasoned thinking behind the evaluation process itself. She argues that evaluation reporting must be clear, accessible, and simple—which does not mean that reports need to be simplistic, but that they must be coherent and comprehensible. I am reminded of a statement by the philosopher John Searle, ‘If you can’t say it clearly, you don’t understand it yourself’.
Reflecting on Davidson’s article, I realize that the best evaluation reports are the product of well thought-out and effectively conducted evaluation research, presented in a clear and cogent way. The findings from such research may be complex, but they need not be obscure or enigmatic. On the contrary, clear evaluation reports must be true stories, well told. To learn more about our evaluation reporting visit our Impact & Assessment reporting page.
I recently participated in a workshop at Brandeis University for graduate students who were considering non-academic careers in the social sciences. During the workshop, one of the students asked about the difference between program evaluation and other kinds of social research. This is a valuable and important question to which I responded that program evaluation is a type of applied social research that is conducted with “a value, or set of values, in its denominator.” I further explained that I meant that evaluation research is always conducted with an eye to whether the outcomes, or results, of a program were achieved, especially when these outcomes are compared to a desired and valued standard or criterion. At the heart of program evaluation is the idea that outcomes, or changes, are valuable and desired. Evaluators conduct evaluation research to find out if these valuable changes (often expressed as program goals or objectives) are, in fact, achieved by the program.
Evaluation research shares many of the same methods and approaches as other social sciences, and indeed, natural sciences. Evaluators draw upon a range of evaluation designs (e.g. experimental design, quasi-experimental desing, non-experimental design) and a range of methodologies (e.g. case studies, observational studies, interviews, etc.) to learn what the effects of a given intervention have been. Did, for example, 8th grade students who received an enriched STEM curriculum do better on tests, than did their otherwise similar peers who didn’t receive the enriched curriculum? Do homeless women who receive career readiness workshops succeed at obtaining employment at greater rates than do other similar homeless women who don’t participate in such workshops? (For more on these types of outcome evaluations, see our previous blog post, “What You Need to Know About Outcome Evaluations: The Basics,”) While not all program evaluations are outcome evaluations, all evaluations gather systematic data with which judgments about the program can be made.
Evaluation’s Differences From Other Kinds of Social Research
Evaluation research is distinct from other forms of applied social research in so far as it:
- seeks to determine the merit, value, and/or worth of a program’s activities and results.
- entails the systematic collection of empirical data that is used to measure the processes and/or outcomes of a program, with the goal of furthering the program’s development and improvement.
- provides actionable information for decision-makers and program stakeholders, so that, based on objective data, a program can be strengthened or curtailed.
- focuses on particular knowledge (usually about a program and its outcomes), rather than seeks widely generalizable and universal knowledge.
While evaluators share many of the same methods and approaches as other researchers, program evaluators must employ an explicit set of values against which to judge the findings of their empirical research. The means that evaluators must both be competent social scientists and exercise value-based judgments and interpretations about the meaning of data. To learn more about our evaluation methods visit our Data collection & Outcome measurement page.
Research vs. Evaluation
Differences Between Research and Evaluation
Harvard Family Research Project’s “Ask an Expert” series.
See “Michael Scriven on the Differences Between Evaluation and Social Science Research,”
Office of Educational Assessment
Sandra Mathison’s “What is the Difference Between Evaluation and Research, and Why Do We Care?”
“Distinguishing Evaluation from Research”
In her book Evaluation (2nd Edition) Carol Weiss writes, “Outcomes define what the program intends to achieve.” (p.117) Outcomes are the results or changes that occur, either in individual participants, or targeted communities. Outcomes occur because a program marshals resources and mobilizes human effort to address a specified social problem. Outcomes, then, are what the program is all about; they are the reason the program exists.
In order to assess which outcomes are achieved, program evaluators design and conduct outcome evaluations. These evaluations are intended to indicate, or measure, the kinds and levels of change that occur for those affected by the program or treatment. “Outcome evaluations measure how clients and their circumstances change, and whether the treatment experience (or program) has been a factor in causing this change. In other words, outcome evaluations aim to assess treatment effectiveness. (World Health Organization)
Outcome evaluations, like other kinds of evaluations, may employ a logic model, or theory of change, which can help evaluators and their clients to identify the short-, medium-, and long-term changes that a program seeks to produce. (See our blog post “Using a Logic Model” ) Once intended changes are identified in the logic model, it is critical for the evaluator to further identify valid and effective measures of said changes, so that these changes are correctly documented. It is preferable to identify desired outcomes before the program begins operation, so that these outcomes can be tracked throughout the program’s life-span.
Most outcome evaluations employ instruments that contain measures of attitudes, behaviors, values, knowledge, and skills. Such instruments may be either standardized, often validated, or they may be uniquely designed, special- purpose instruments, (e.g., a survey designed specifically for this particular program.) Additionally, the measures contained in an instrument can be either “objective,” i.e., they don’t rely on individuals’ self-reports, or conversely, they can be “subjective,” i.e., based on informants’ self-estimates of effect. Ideally, outcome evaluations try to use objective measures, whenever possible. In many instances, however, it is desirable to use instruments that rely on participants’ self-reported changes and reports of program benefits.
It is important to note that outcomes (i.e. changes or results) can occur at different points in time in the life span of a program. Although outcome evaluations are often associated with “summative,” or end-of-program-cycle evaluations, because program outcomes can occur in the early or middle stages of a program’s operation, outcomes may be measured before the final stage of the program. It may even be useful for some evaluations to look at both short- and long-term outcomes, and therefore to be implemented at different points in time (i.e., early and late.)
Another issue relevant to outcome evaluation is dealing with unintended outcomes of a program. As you know, programs can have a range of intended goals. Some outcomes or results, however, may not be a part of the intended goals of the program. They nonetheless occur. It is critical for evaluations to try to capture the unintended consequences of programs’ operation as well as the intended outcomes.
Ultimately, outcome evaluations are the way that evaluators and their clients know if the program is making a difference, which differences it’s making, and if the differences it’s making, are the result of the program. To learn more about our evaluation and outcome assessment methods visit our Data collection & Outcome measurement page.
World Health Organization, Workbook 7
Measuring Program Outcomes: A Practical Approach (1996) United Way of America’s
Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources
Evaluation Methodology Basics, The Nuts and Bolts of Sound Evaluation,
E. Jane Davidson, Sage, 2005.
Evaluation (2nd Edition) Carol Weiss, Prentice Hall, 1998.
Pioneered by market researchers and mid-20th century sociologists, focus groups are a qualitative research method that involves small groups of people in guided discussions about their attitudes, beliefs, experiences, and opinions about a selected topic or issue. Often used by marketers who obtain feedback from consumers about a product or service, focus groups have also become an effective and widely recognized social science research tool that enables researchers to explore participants’ views, and to reveal rich data that often remain under-reported by other kinds of data collection strategies (e.g., surveys, questionnaires, etc. ).
Organized around a set of guiding questions, focus groups typically are composed of 6-10 people and a moderator who poses open-ended questions that allow participants to address questions. Focus groups usually include people who are somewhat similar in characteristics or social roles. Participants are selected for their knowledge, reflectiveness, and willingness to engage topics or questions. Ideally—although not always possible—it is best to involve participants who don’t previously know one another.
Focus group conversations enable participants to offer observations, define issues, pose and refine questions, and create informative debate/discussions. Focus group moderators must: be attentive, pose useful and creative questions, create a welcoming and non-judgmental atmosphere, be sensitive to non-verbal cues and the emotional tenor of participants. Typically, focus group sessions are recorded or videoed so that researchers can later transcribe and analyze participants’ comments. Often an assistant moderator will take notes during the focus group conversation.
Focus groups have advantages over other date collection methods. They often employ group dynamics that help to reveal information that would not emerge from an individual interview or survey: they produce relatively quick, low cost data (they produce an ‘economy of scale’ as compared to individual interviews); allow the moderator to pose appropriate and responsive follow-up questions; enable the moderator to observe non-verbal data; and often produce greater and richer data than a questionnaire or survey.
Focus groups also can have some disadvantages, especially if not conducted by an experienced and skilled moderator: Depending upon their composition, focus groups are not necessarily representative of the general population; respondents may feel social pressure to endorse other group members’ opinions or refrain from voicing their own; group discussions require effective “steering” so that key questions are answered, and participants don’t stray from the questions/topic.
Focus groups are often used in program evaluations. I have had extensive experience conducting focus groups with a wide-range of constituencies. During my 20 years of experience as a program evaluator, I’ve moderated focus groups composed of: homeless persons; disadvantaged youth; university pr ofessors and administrators; k-12 teachers; k-12 and university students, corporate managers; and hospital administrators. In each of these groups I’ve found that it’s been beneficial to: have a non-judgmental attitude, be genuinely curious; exercise a gentle guidance; and respect the opinions, beliefs, and experiences of each focus group member. A sense of humor can also be extremely helpful. (See our previous post: “Interpersonal Skills Enhance Program Evaluation,” Also “Listening to Those Who Matter Most, the Beneficiaries” ) Or if you want to learn more about our qualitative approaches visit our Data collection & Outcome measurement page.
About focus groups:
About focus groups:
How focus groups work:
Focus group interviewing:
‘Focus groups’ at Wikipedia
A needs assessment is a systematic research and planning process for determining the discrepancy between an actual condition or state of affairs, and a future desired condition or state of affairs. Needs assessments are undertaken not only to identify the gap between “what is” and “what should be” but also to identify the programmatic actions and resources that are required to address that gap. Typically, a needs assessment is a part of planning processes that is intended to yield improvements in individuals, education/training, organizations, and/ or communities. (https://en.wikipedia.org/wiki/Needs_assessment ) Ultimately, a needs assessment is “a systematic process whose aim is to acquire an accurate, thorough picture of a system’s strengths and weaknesses, in order to improve it and to meet existing and future challenges.”
( http://dictionary.reference.com/browse/needs+assessment) Needs assessments have a variety of purposes. They can be used to identify and address challenges in a community, to develop training strategies, or to improve the performance of organizations.
There are a variety of conceptual modules of needs assessments. (For a review of various models (See http://ryanrwatkins.com/na/namodels.html ) One of the most popular is the SWOT analysis, in which researchers and action teams conduct a study to determine the strengths, weaknesses, opportunities, and threats involved in a project or business venture. In Planning and Conducting Needs Assessments: A Practical Guide. Thousand Oaks, CA. Sage Publications. (1995) Witkin, and Altschuld, identify a three-stage model of needs assessment, which includes pre-assessment (exploration), assessment (data gathering), and post-assessment (utilization).
Although there are various approaches to needs assessment, most include the following essential components/steps:
- Identify issue/concern
- Conduct a gap analysis (where things are now vs. where they should be)
- Specify methods for collecting information/data
- Perform literature review
- Collect and analyze data
- Develop action plan
- Produce implementation report
- Disseminate report/recommendations to stakeholders.
Why Conduct a Needs Assessment
Needs assessments can be used to identify real-world challenges, to formulate plans to correct inequities, and to involve critical stakeholders in building consensus and mobilizing resources to address identified challenges. For non-profit organizations needs assessments: 1) use data to identify an unaddressed or under-addressed need; 2) help to more effectively utilize resources to address a given problem; 3) make programs measurable, defensible, and fundable; and 4) inform, mobilize, and re-energize stakeholders. Needs assessments can be used with an organizations internal and external stakeholders and constituents.
Brad Rose Consulting Inc. has extensive experience in designing and implementing needs assessments for non-profit organizations, educational institutions, and health and human service programs. We’d welcome a chance to speak with you and your colleagues about how we can help you to conduct a needs assessment. To learn more about our assessment methods visit our Data collection & Outcome measurement page.
Pyramid Model of Needs Assessment
Needs Assessment: Strategies for Community Groups and Organizations
Needs Assessment 101
U.S. Department of Education
Needs Assessment: A User’s Guide
As mentioned in a previous blog post, program evaluation can play an important role in an organization’s strategic planning initiatives. This is especially true in non-profit organizations, human service agencies, k-12, and higher education institutions, all of whom must rely on non-market data for evidence of program effects. Evaluation can help these organizations to identify, gather, and analyze data with which to judge the impact of their activities and to strengthen current, or redirect future, efforts. Only with clear and accurate information can a non-profit organization take stock of its effectiveness and make informed choices about needed changes in direction. As Heather Tunis and Maura Harrington, note “An evaluation plan helps refine data collection and assessment practices so that the information is most useful to advancing the organization’s mission and the objectives of the program being evaluated. Evaluation is a key component of being a learning organization.” (Non-profits: Strategic Planning and Future Program Evaluation) As Mark Fulop observes, “Indeed nonprofits that embrace evaluation as strategy will be driven by internal excellence rather than an external locus of control. Nonprofits that embrace evaluation as strategy will strengthen not only their organizational core but the centrality of their place in solving social needs.” (“The Roles of Strategic Evaluation in Non-profits” ) To learn more about our evaluation methods visit our Data collection & Outcome measurement page.
Also, here are a few links to resources about strategic planning and program evaluation:
From CDC, “Using Program Evaluation to Improve Programs: Strategic Planning” http://www.cdc.gov/HealthyYouth/evaluation/pdf/sp_kit/sp_toolkit.pdf
“Strategic Planning Resources for Non-Profits” http://nonprofitanswerguide.org/faq/strategic-planning/
“Why Strategic Planning for Non-profits is Important,” http://www.event360.com/blog/why-nonprofit-strategic-planning-is-important/
“What is strategic planning, and why should all schools have a strategic plan?” http://www.strategicplanning4schools.com/overview.html
From the National Alliance for Media Arts and Culture, “Basic Steps to a Strategic Planning Process,”http://www.namac.org/strategic-planning-steps
From the World Bank, “Strategic Planning- A Ten-Step Guide” http://siteresources.worldbank.org/INTAFRREGTOPTEIA/Resources/mosaica_10_steps.pdf
Occasionally, I encounter a resource that I think may be useful to clients and colleagues. I recently had the pleasure of enlisting the help of Green Heron Information services who helped me conduct a literature review for a project that I was working on. I’d like to share with you some of the central ideas about doing literature reviews—ideas that may be helpful to both evaluators and grant seekers— and encourage you to connect with Matt Von Hendy who is the president of Green Heron Information Services. (240) 401-7433 (email at email@example.com or visit www.greenheroninfo.com)
If you are like most people you probably have not thought about literature reviews since college or graduate school until you need to write one for a contract report, journal article or grant proposal. Just a quick review: A literature review is a piece of work that provides an overview of published information on a particular topic or subject usually within a specific period of time and discusses critical points of the current state of knowledge in the field including major findings as well as theoretical and methodological contributions. It will generally seek to present a summary of the important works but also provide a synthesis of this information as well.
Literature reviews matter for a number of reasons: they demonstrate a strong knowledge of the current state of research in the field or topic; they show what issues are being discussed or debated and where research is headed; and they provide excellent background information for placing a program, initiative or grant proposal in context. In short, a well-written literature review can provide a ‘mental road map’ of the past, present and future of research in a particular field.
Literature reviews can take many different types and forms but typically good ones share certain characteristics such as:
- Follows an organizational pattern that combines summary and synthesis
- Tracks the intellectual progression of a thought or a field of study
- Contains a conclusion that offer suggestions for future research
- Is well-researched
- Uses a wide variety of high quality resources including journal articles, conference papers, books and reports
Many evaluation and grant professionals when doing research for literature reviews use some combination of Google and/or other professionals as their primary information sources. While these resources are a great place to start, they both have limitations which make them not so good places to end your research. For example, search engines such as Google filter results based on a number of factors and very few experts can keep up to date with the amount of information that is being published. Fortunately, many high quality tools such as citation databases and subject specific databases exist that make going beyond Google relatively easy. Many evaluation professionals and proposal writers are motivated to do their own research but there are times such as working in new areas or tight deadlines where hiring an information professional to consult, research or write a literature review can be helpful.
You may think while this all sounds good in theory but wonder how would it work in practice? Let me offer a very quick case study of conducting research for a literature review in evaluating programs that attempt to improve mental health outcomes for teenagers in the United States. I would first start a list of sources by consulting experts and searching on Google. My next step would be to look at the two major citation databases, Scopus and Web of Science, and find out which journal articles and conference papers are most cited. I would then search in the subject specific databases that cover the health and psychology areas such as PubMed, Medline and PsycINFO. Finally, I would examine resources such as academic and non-profit think tanks just to make sure I was not missing anything important.
A well-researched and written literature review offers a number of benefits for evaluation professionals, grant-seekers and even funders and grantors: it can show an excellent understanding of the research in a subject area; it can demonstrate about what current issues or topics are being debated and suggest directions for future research; it can also provide an excellent way to place a program, initiative or proposal into context within the larger picture of what is happening in an area. If you have questions with getting started on a literature review, we are always glad to offer suggestions.
Green Heron Information Services offers consulting, research and writing services in support of literature review efforts. www.greenheroninfo.com
Like other research initiatives, each program evaluation should begin with a question, or series of questions that the evaluation seeks to answer. (See my previous bog post “Approaching and Evaluation: Ten Issues to Consider.”). Evaluation questions are what guide the evaluation, give it direction, and express its purpose. Identifying guiding questions is essential to the success of any evaluation research effort.
Of course, evaluation stakeholders can have various interests, and therefore, can have various kinds of questions about a program. For example, funders often want to know if the program worked, if it was a good investment, and if the desired changes/outcomes were achieved (i.e., outcome/summative questions). During the program’s implementation, program managers and implementers, may want to know what’s working and what’s not working, so they can refine the program to ensure that it is more likely to produce the desired outcomes (i.e., formative questions). Program managers and implementers may also want to know which part of the program has been implemented, and if recipients of services are indeed, being reached and receiving program services (i.e., process questions). Participants, community stakeholders, funders, and program implementers may all want to know if the program makes the intended difference in the lives of those who the program is designed to serve (i.e., outcome/summative questions).
While program evaluations seldom serve only one purpose, or ask only one type of question, it may be useful to examine the kinds of questions that pertain to the different types of evaluations. Establishing clarity about the questions an evaluation will answer will maximize the evaluation’s effectiveness, and ensure that stakeholders find evaluation results useful and meaningful.
Types of Evaluation Questions
Although the list below is not exhaustive, it is illustrative of the kinds of questions that each type of evaluation seeks to answer.
▪ Process Evaluation Questions
- Were the services, products, and resources of the program, in fact, produced and delivered to stakeholders and users?
- Did the program’s services, products, and resources reach their intended audiences and users?
- Were services, products, and resources made available to intended audiences and users in a timely manner?
- What kinds of challenges did the program encounter in developing, disseminating, and providing its services, products, and resources?
- What steps were taken by the program to address these challenges?
▪ Formative Evaluation Questions
- How do program stakeholders rate the quality, relevance, and utility, of the program’s activities, products and services?
- How can the activities, products, and services of the program be refined and strengthened during project implementation, so that they better meet the needs of participants and stakeholders?
- What suggestions do participants and stakeholders have for improving the quality, relevance, and utility, of the program?
- Which elements of the program do participants find most beneficial, and which least beneficial?
▪ Outcome/Summative Evaluation Questions
- What effect(s) did the program have on its participants and stakeholders (e.g., changes in knowledge, attitudes, behavior, skills and practices)?
- Did the activities, actions, and services (i.e., outputs) of the program provide high quality services and resources to stakeholders?
- Did the activities, actions, and services of the program raise the awareness and provide new and useful knowledge to participants?
- What is the ultimate worth, merit, and value of the program?
- Should the program be continued or curtailed?
The process of identifying which questions program sponsors want the evaluation to answer thus becomes a means for identifying the kinds of methods that an evaluation will use. If, ultimately, we want to know if a program is causing a specific outcome, then the best method (the “gold standard”) is to design a randomized control experiment (RCT). Often, however, we are interested not just in knowing if a program causes a particular outcome, but why it does so and how it does so. In that case, it will be essential to use a mixed methods approach that draws not just on quantitative outcome data that compare the outcomes of treatment and control groups, but also to use qualitative methods (e.g., interviews, focus groups, direct observation of program functioning, document review, etc.) that can help elucidate why what happens, happens, and what program participants experience.
Robust and useful program evaluations begin with the questions that stakeholders want answered, and then identifies the best methods to gather data to answer these questions. To learn more about our evaluation methods visit our Data collection & Outcome measurement page.
Evaluator competencies—the skills, knowledge and attitudes— required to be an effective program evaluator have been much discussed. (See, for example, The International Board of Standards for Training, Performance and Instruction Evaluator Competencies, and the CDC’s “Finding the Right People for Your Program Evaluation Team: Evaluator and Planning Team Job Descriptions.” )
A good evaluator must, of course, be able to develop a research design, carry out research in the field, analyze data, and report findings. These technical/methodological skills, although of critical importance, are however, not the only skills that evaluators need. Effective evaluations also depend upon a range of interpersonal or relational skills that make effective and responsive interpersonal interaction possible.
I recently posted to the American Evaluation Associations listserv a query about the importance and role of the interpersonal skills in evaluation. I asked for AEA members’ opinions about the importance of interpersonal skills in conducting successful evaluations. A number of evaluators responded to my inquiry. The central theme of those responses was that successful evaluators and successful evaluation engagements require that evaluators possess and employ key interpersonal skills, and that without these, evaluation engagements are unlikely to be successful.
Among the most prominent reasons that my AEA colleagues reported for the importance of interpersonal skills were: 1) the importance of building strong, candid, and constructive relationships, on which effective data collection depends; and 2) the importance of establishing trusting and collaborative relationships between evaluators and stakeholders in order to help to ensure that evaluation findings will be utilized by clients and stakeholders. Additionally, some colleagues commented that the self-evident reason for utilizing strong interpersonal skills in evaluation engagements: these skills enhance the probability that clients and stakeholders will share information and provide insights about the program. Thus effective evaluation necessarily entails trusting, open, and amicable relationships that make access to program knowledge and information possible .
Reflecting on my 25 years of professional experience, which includes observing the work of many evaluators, I think that key interpersonal characteristics, include the abilities to:
- Build rapport and trust with clients, evaluands, and stakeholders
- Act with personal integrity
- Display a genuine curiosity and ask good questions
- Make oneself vulnerable in order to learn (see my earlier blog post on the role of vulnerability in learning and creativity at https://bradroseconsulting.com/secret-innovation-creativity-change/
- Actively listen
- Be empathic
- Be both socially aware and self-aware— i.e., be aware of, and manage, both one’s own and other’s emotions (including the features of emotional intelligence, i.e, capacities to accurately perceive emotions, use emotions to facilitate thinking, to understand emotional meanings, and to manage emotions).
- Treat each person with respect
- Manage conflict and galvanize collaboration
- Problem solve
- Facilitate collective (group) learning
- These interpersonal skills are central to successful program evaluations. Attention to these characteristics– both by program evaluators and by those seeking to engage a program evaluator (i.e. evaluation clients)— will greatly maximize the probability of successful evaluation projects.
To learn more about our evaluation methods visit our Data collection & Outcome measurement page.
Interactive Evaluation Practice: Mastering the Interpersonal Dynamics of Program Evaluation J.A. King and L. Stevahn (Sage).
Working with Emotional Intelligence. Daniel Goleman (Bantam)
This past March Brad Rose was selected as Fellow by the National Center for Innovation and Excellence. Brad will join a group of 13 Fellows all of whom work with the Center to further their mission across the United States.
About the Center:
“The National Center for Innovation and Excellence is a dynamic community dedicated to developing youth, changing communities, growing economies, and improving the lives of others by working with organizations, foundations, governments, and communities to design, implement, and test outcomes for large scale youth development strategies and community change initiatives.”
To learn more visit http://www.ncfie.org/
Programs are seldom implemented under pristine laboratory conditions. Instead, they occur in the real world, in real time. They unfold in complex environments, with ever-changing circumstances and unforeseeable developments. Consequently, program evaluations need to be adaptive, aware of the reality of programs’ often tumultuous contexts, and capable of suppleness and flexibility. This is especially true for evaluations that seek to assess the impact of innovative initiatives whose goals are often not standardized and pre-determined, but are evolving and emergent.
Over the last 20 years, Developmental Evaluation has emerged as an important evaluation approach for meeting the evaluation needs of innovative initiatives. As Michael Quinn Patton, a noted theorist and practitioner of Developmental Evaluation has noted,
“Developmental evaluation (DE) is especially appropriate for innovative initiatives or organizations in dynamic and complex environments where participants, conditions, interventions, and context are turbulent, pathways for achieving desired outcomes are uncertain, and conflicts about what to do are high. DE supports reality-testing, innovation, and adaptation in complex dynamic systems where relationships among critical elements are nonlinear and emergent. Evaluation use in such environments focuses on continuous and ongoing adaptation, intensive reflective practice, and rapid, real-time feedback.” (http://comm.eval.org/viewdocument/?DocumentKey=95f16941-7e8a-4785-907a-42615d919d7a )
Developmental Evaluation Serves Innovative Programs
While Developmental Evaluation is appropriate for many programs and organizations, it is especially useful for programs that aspire to continuous learning, that value adaptation, and that seek innovative means to address emerging (vs. “known”) issues. Such programs are typically found in the social philanthropic and non-profit sectors. Evaluators who practice Developmental Evaluation transcend the typical role of a traditional evaluator—they don’t just design formative or summative evaluations. Developmental evaluators work closely with decision makers, program designers and staff to ask key questions about program design and logic, to collect data—sometimes in rapid time frames—to inform real-time program implementation and refinement, and to ensure that programs consistently employ the principles of learning and continuous improvement. Patton observed in his book Utilization Focused Evaluation (3rd Edition):
“Developmental Evaluation refers to evaluation processes undertaken for the purpose of supporting program, project, staff and/or organizational development, including asking evaluative questions and applying evaluation logic for development purposes. The evaluator is part of a team whose members collaborate to conceptualize, design, and test new approaches in a long-term, on-going process of continuous improvement, adaptation and intentional change. The evaluator’s primary function is to elucidate them discussions with evaluative questions, data and logic, and to facilitate data-based decision-making…”
Brad Rose Consulting, Inc. utilizes the principles and insights of Developmental Evaluation. Our 20+ years of experience working with social entrepreneurs and innovative non-profit organizations has taught us that even seemingly “standard” program designs can benefit from a nuanced, responsive, context-sensitive, evaluation approach, one that draws on the practices of Developmental Evaluation. Additionally, innovative programs whose outcomes are not fully predictable nor exclusively pre-determined, will find that Developmental Evaluation provides the iterative feedback necessary to strengthen the program and to achieve enhanced outcomes. Because Developmental Evaluation is essentially consultative, integrative, and built on a constructive and supportive relationship between the evaluator and the organization’s staff, it offers programs designers, managers, and implementers superior insights into the complex, often rapidly changing conditions in which genuine innovations occur. To learn more about our program development methods visit our Program development & Funding page.
A Developmental Evaluation Primer, at J.W. McConnell Family Foundation
Video Michael Quinn Patton on Developmental Evaluation
Link to Michael Quinn Patton, Developmental Evaluation
A conversation with Michal Quinn Patton
“Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.” Guilford Press New York. 2011
In our last blog post, Evaluation Serving Community Foundations and Donors, we discussed the value of program evaluation to community foundations, especially as a means for demonstrating to donors and other foundation stakeholders that program and project funds are used to achieve the outcomes that donors desire. In this blog post, I’d like to discuss an additional benefit of program evaluations.
Program Evaluation for Strengthening Grantee Effectiveness
Because well-designed formative evaluations can provide useful information about how programs are working, especially during programs’ early and intermediate stages, timely and well-designed formative evaluations can play a significant role in helping grantees, i.e., program managers and implementers, with critical information with which to refine and strengthen programming. Community Foundations, therefore, can use formative evaluations to help grantees better achieve their programmatic goals and objectives. By conducting formative evaluations, not just summative evaluations, Community Foundations can ensure that grantees are optimally prepared to achieve the desired program results.
Formative evaluations typically ask the following kinds of questions:
a.. As the project is developed and launched, what are the project’s strengths and vulnerabilities?
b.. In regard to the programs implementation, what’s working and what’s not?
c.. What kinds of implementation problems emerge and how are these being addressed?
d.. What’s happening that wasn’t expected?
e.. How do participants, staff, and stakeholders perceive the program’s initial and mid-cycle effectiveness?
f.. What new ideas are emerging from project implementation that can be tested to strengthen the program’s effectiveness?
g.. How can the program be improved to maximize program outcomes and impacts?
By implementing formative evaluations, Community Foundations can:
a.. Effectively monitor the progress grantees make toward program goals.
b.. Develop grantees’ awareness of, and facility with program evaluation.
c.. Ensure that programs are optimally positioned to achieve intended outcomes.
d.. Develop the indigenous capacities of grantee organizations to monitor and evaluate.
e.. Assure that organizations supported by Community Foundations make their best efforts to refine programming and thus, better serve their stakeholders and service recipients.
Brad Rose Consulting, Inc. has extensive experience designing formative evaluations and program monitoring initiatives. We work with clients in the Community Foundation and Non-Profit sectors to ensure that programs are effectively working towards their goals and making timely, data-informed adjustments to improve program quality. Ultimately we are concerned both with measuring programs’ effects AND with maximizing programs’ effectiveness. We can help Community Foundations to achieve these dual goals. To learn more about our work with community foundations visit our Community service page.
The Importance of Demonstrating to Community Foundation Funders Programs’ Impacts
As donors seek to strengthen and improve the lives of their communities, they are increasingly interested in understanding the effects of their social investments. Consequently, community foundations are now frequently asked to show that the programs and organizations they support are having the effects that donors would like to see. Program evaluations of community foundation-supported programs are an extremely useful way to document the effectiveness of programs. Evaluations can provide critical information about the effects of programs, show donors that their expectations are being met, and provide critical information with which to ensure that critical community needs are being effectively addressed. A recent article in the New Yorker noted, “Measuring the impact of giving is more important to donors aged 21-40 than it was to previous generations.” (New Yorker December, 2013)
Brad Rose Consulting, Inc. has evaluated a wide-range of philanthropic programs, ranging from human services and arts education programs, to youth development and domestic violence prevention programs. We work with foundations to ensure both effective and sensitive program evaluations that show the effects of donor- supported programming. Our evaluations are especially helpful to programs supported by designated funds. While our experience includes working with national-level foundations, our approach to tailored evaluation initiatives is especially geared to helping community-level foundations who are increasingly asked to demonstrate the effectiveness of the programs their donor funds support.
Program Evaluation to Support Planning
Although our evaluations gather data to show the effects of existing programs, our evaluation approach can be very helpful as funders and donors plan for future initiatives. By showing both the achievements and challenges of current programs, our evaluations provide to program planners and funders helpful insight into a variety of program area “best practices”. We also work with foundation grantees to help them develop their own capacities for evaluating the effects of their efforts. By educating grant recipients about the need for, and fundamental elements of, program evaluation, we prepare grantees to better plan and execute programs that demonstrate results.
If you are a community foundation staff member, we would welcome and opportunity to discuss with you the ways that Brad Rose Consulting, Inc. can help you and your grantees to strengthen the programs you support so that they better serve the goals and expectations of your donors. Similarly, if you are an organization that receives foundation support, we would welcome a chance to discuss with you how our program evaluations can help you strengthen your programmatic efforts and help to secure future foundation support. Please feel free to contact me, click here for contact information. Or to learn more about our work with community foundations visit our Community service page.
Typically, we work with clients from the early stages of program development in order to understand their organization’s needs and the needs of program funders and other stakeholders. Following initial consultations with program managers and program staff, we work collaboratively to identify key evaluation questions, and to design a strategy for collecting and analyzing data that will provide meaningful and useful information to all stakeholders.
Depending upon the specific initiative, we implement a range of evaluation tools (e.g., interview protocols, web-based surveys, focus groups, quantitative measures, etc.) that allow us to collect, analyze, and interpret data about the activities and outcomes of the specified program. Periodic debriefings with program staff and stakeholders allow us to communicate preliminary findings, and to offer program managers timely opportunities to refine programming so that they can better achieve intended goals.
Our collaborative approach to working with clients allows us to actively support program managers, staff, and funders to make data-informed judgments about programs’ effectiveness and value. At the appropriate time(s) in the program’s implementation, we write a report(s) that details findings from program evaluation activities and that makes data-based suggestions for program improvement. To learn more about our approach to evaluation visit our Data collection & Outcome measurement page.
Most organizations conduct evaluations because they want to determine if they are making a difference in the lives of the people that they serve. Determining program effectiveness, showing the specific effects of programming, and using data to strengthen programs, are all important and laudable reasons for carrying out an evaluation.
In recent years, however “accountability” has become a driving force for many organizations (schools, government agencies, and non-profits) to conduct evaluations. “Accountability” has become a watchword—especially in the educational and non-profit sectors. While accountability has its legitimate purposes (i.e., demonstrating to stakeholders that an organization or program is responsible, ethical, committed to achieving its goals, etc.) too often the desire to demonstrate accountability, especially legal compliance, overshadows the use of evaluations to enhance program effectiveness and strengthen program outcomes.
Brad Rose Consulting, Inc. is committed to evaluations that provide an evidence- based account of a program’s effects. Equally importantly, however, we are committed to designing and conducting evaluations that help to strengthen a program (and its host organization) so that it can better achieve its desired outcomes. We work with organizations to objectively find out what’s working and what needs to be strengthened.
When, for example, we work with educators (superintendents, principals, teachers and school staff) and school systems to evaluate their educational programs, we design evaluations that BOTH show program effectiveness AND provide data-based insights that help strengthen future outcomes. We understand that educators need BOTH to know if students are learning AND how to enhance future student achievement. Such evaluations require not merely collecting static and tiresome data, but implementing evaluations that richly show how educational initiatives can be made more effective. Often such evaluations transcend the collection merely of student test scores and look at the multiple factors that influence student achievement. We deliberately work to make our evaluations constructive opportunities for strengthening instruction and enhancing student achievement. To learn more about our work in education visit our Higher education & K-12 page.
More information about accountability:
Last week, I was working with a new client, and we were sketching out a logic model for one, among a variety, of the programs that my client operates. As we talked about the inputs, outputs, and the short-, medium-, and long-term, outcomes their program produces, it dawned on me that we were unintentionally sketching many of the elements that might contribute to a strategic plan for the overall organization. This experience prompted me to think about how developing logic models for specific initiatives and programs, can also assist organizations to consider and reflect upon their broader goals, operations, and results. Although a logic model typically charts the logic of a particular program, it shares many of the features and illustrates many of the features of the organization that runs the program.
What is Strategic Planning?
“Simply put, strategic planning determines where an organization is going over the next year or more, how it’s going to get there and how it’ll know if it got there or not. The focus of a strategic plan is usually on the entire organization…” (See the Free Management Library at http://managementhelp.org/strategicplanning/index.htm#anchor1234) Balanced Scorecard says, “Strategic planning is an organizational management activity that is used to set priorities, focus energy and resources, strengthen operations, ensure that employees and other stakeholders are working toward common goals, establish agreement around intended outcomes/results, and assess and adjust the organization’s direction in response to a changing environment. It is a disciplined effort that produces fundamental decisions and actions that shape and guide what an organization is, who it serves, what it does, and why it does it, with a focus on the future. Resource.
Features Common to Both Logic Models and Strategic Plans
My experience working with clients has shown me that logic models raise many of the same questions that strategic plans do: What are our assumptions about how a program works? What is the environment (context) in which a program operates? What are we trying to achieve (goals and objectives)? What investments (inputs) do we make? What activities (outputs) do we engage in? What are the results, changes, impacts (outcomes) that we want to, and in fact, DO, produce? How do we measure our effects and achievements (measures/metrics)?
Although logic modeling can’t do all of the things a strategic plan can, it can become – especially when it includes an organization’s many stakeholders – an important contributor to the process through which an organization reflects upon where it is and where it wants to go. The collective learning that accompanies the process of building a logic model for a specific program, can also inform the organization’s efforts to develop a broader strategic plan.
Read Brad’s current whitepaper “Logic Modeling”
What is a Strategic Plan?
What is the Balanced Scorecard?
The Basics of Strategic Planning and Strategic Management
What a Strategic Plan Is and Isn’t
Ten Keys to Successful Strategic Planning for Nonprofit and Foundation Leaders
Types of Strategic Planning
Understanding Strategic Planning
Steps to a Strategic Plan
Five Steps to a Strategic Plan
Strategic Planning for Non-Profits
Strategic Planning for Non-Profits
What is the best way to do strategic planning for a nonprofit?
Videos About Strategic Planning
University of Arizona
Introduction to Strategic Planning
A logic model is a schematic representation of the elements of a program and the program’s resulting effects. A logic model (also known, as a “theory of change”) is a useful tool for understanding the way a program intends to produce the outcomes (i.e. changes) it hopes to produce. Logic models typically consist of a flowchart schematic that shows the logical connection between a program’s “inputs” (i.e. invested resources), “outputs” (program activities and actions), “short-term outcomes” (changes), “medium-term outcomes” (changes), and “long range impacts” (changes).
When developing a logic model many evaluators and program staff rightly focus on inputs, outputs, and program outcomes (the core of the program). However, it is critical to also include in the logic model the implicit assumptions that underlie the program’s operation, the needs that the program aspires to address, and the program’s environment, or context. Assumptions, needs, and context are crucial factors in understanding how the program does what it intends to do. Ultimately these are crucial to understanding the causal mechanisms that produce the intended changes of any program.
Without clearly understanding the causal mechanisms at work in a program, program staff may work ineffectively, placing emphasis on the wrong or inefective activities—and ultimately fail to correctly address the challenges the program intends to address. Similarly, without a clear understanding of the causal mechanisms that enable the program to achieve its outcomes, the program evaluation may not measure the proper outcomes or fail to see the changes the program, in fact, brings about.
Brad Rose Consulting, Inc. works with clients to develop simple, yet robust, logic models that explicitly document the causal mechanisms that are at work in a program. By discussing, and explicitly identifying the often implicit causal assumptions, as well as highlighting the needs for the program and the social context of a program, we not only ensure that the evaluation is properly designed and executed, we also help program implementers to ensure that they are activating the causal processes/mechanisms that yield the changes that the program strives to achieve.
Read Brad’s current whitepaper “Logic Modeling”
In recent years, “social entrepreneur” has become a prominent term in the not-for-profit, foundation, and NGO worlds. But what exactly is a “social entrepreneur?” While social entrepreneurs share many of the characteristics ascribed to for-profit entrepreneurs, Roger L. Martin & Sally Osberg observe in their article in the Stanford Innovation Review Social Entrepreneurship: The Case for Definition that “the social entrepreneur aims for value in the form of large-scale, transformational benefit that accrues either to a significant segment of society or to society at large.” They also note that “social entrepreneurship….is as vital to the progress of societies as is entrepreneurship to the progress of economies, and it merits more rigorous, serious attention than it has attracted so far.”
PBS (http://www.pbs.org/opb/thenewheroes/whatis/ ) notes “A social entrepreneur identifies and solves social problems on a large scale. Just as business entrepreneurs create and transform whole industries, social entrepreneurs act as the change agents for society, seizing opportunities others miss in order to improve systems, invent and disseminate new approaches and advance sustainable solutions that create social value. Unlike traditional business entrepreneurs, social entrepreneurs primarily seek to generate “social value” rather than profits. And unlike the majority of non-profit organizations, their work is targeted not only towards immediate, small-scale effects, but sweeping, long-term change.”
The Ashoka Foundation similarly stresses the large-scale effects that social entrepreneurs seek to make. “Social entrepreneurs are individuals with innovative solutions to society’s most pressing social problems. They are ambitious and persistent, tackling major social issues and offering new ideas for wide-scale change. Rather than leaving societal needs to the government or business sectors, social entrepreneurs find what is not working and solve the problem by changing the system, spreading the solution, and persuading entire societies to take new leaps.” (See https://www.ashoka.org/social_entrepreneur )
While the above sources highlight the often ambitious, indeed global, goals of social entrepreneurs, most social entrepreneurship, in fact, involves developing and sustaining specific organizations and, in turn, operating discrete programs. These programmatic efforts can benefit from program evaluations that gather information to show that they are making the differences that their founders hope to make. Evaluations of social entrepreneur-sponsored initiatives not only provide critical evidence of impact, but equally importantly, provide objective information gathered directly from program recipients and others program stakeholders about the ways that such efforts might be strengthened. (See our previous post, “Listening to Those Who Matter Most: Beneficiaries.”) Evaluations of social entrepreneur-sponsored initiatives are especially important because program participants, service recipients, and other beneficiaries are seldom in a position to directly provide feedback to the innovators who are responsible for the programming. Transformative initiatives–no less than smaller scale programs—can substantially benefit from program evaluations. To learn more about our work with Non-Profits visit our Non-Profits page.
See also “Advancing Evaluation Practices in Philanthropy” at The Stanford Innovation Review http://www.ssireview.org/supplement/advancing_evaluation_practices_in_philanthropy
The other day, I conducted a focus group with disadvantaged youth. On behalf of a local workforce investment board, I interviewed a group of 16-24 year-olds about their use of cell phone and other hand-held technologies, in order to learn whether it would be possible to reach youth with career development programming via cell phone and electronic modalities. (In my 20+ years as a professional evaluator, I’ve conducted between 50 and 60 focus groups, with participants who range across the socioeconomic spectrum—from homeless women to college presidents.) As this focus group proceeded, I became aware of a two things. Many of the youth saw me, understandably, as an authority figure to whom they had to give guarded responses—at least initially— and whose trust I needed to earn. Additionally, I, too, felt vulnerable before the group of young people, who I feared might think I was uninformed about cyber culture and the prevailing circumstances of their age group. Each of us, in our own ways, felt “vulnerable.”
It occurred to me that focus group members’ sense of vulnerability would yield only if I myself became more open and vulnerable. Consequently, I abandoned my interview protocol, and began improvising candid and spontaneous questions. I also confessed my lack of knowledge about the technologies that young people often use so comfortably, as if it is an extension of themselves. I also redoubled my efforts to enlist the opinions of each member—especially those who seemed, at first, reticent to share their experience. As the focus group continued, I noticed that some of the initial reticence and reserve of my interlocutors began to dissolve, and even those who had not initially offered their opinions and experience, began to fully participate in the group. I also noticed that as I further expressed my genuine interest in learning about their experience, the sense of who possessed the authority shifted from me—the question-asker— to the youth in the group, who became experts on the subject I was interviewing them about.
The Necessity of Vulnerability in Education
Although all of us necessarily spend a lot of our lives shielding ourselves from various forms of vulnerability (economic, social, emotional, etc.), research is beginning to show that psychological vulnerability and the willingness to risk social shame and embarrassment, is essential for genuine learning, creativity and path-breaking innovation. In a recent TED presentation by Brene Brown a research professor at the University of Houston Graduate School of Social Work, who has spent the last decade studying vulnerability, courage, authenticity, and shame (Click here to listen). Ms. Brown discusses the importance of making mistakes and enduring potential embarrassment, in order to learn new things and make new connections. Brown highlights the significance of making ourselves vulnerable (i.e. taking risks, enduring uncertainty, handling emotional exposure) so that we can genuinely connect with others, and learn from them. Fear of failure and fear of vulnerability (especially fear of social shame) she says, too often get in the way of our learning from others. Moreover, we are often deathly afraid of making mistakes (See our recent post “Fail Forward: What We Can Learn from Program ”Failure”. You can also listen to the entire NPR TED Hour on the importance of mistakes to the process of learning, here. Ultimately, we must embrace, rather than deny vulnerability, if we are to connect, and thereby learn.
I’ve conducted research for most of my professional life. As I reflected on my professional experience, I realize that I’ve learned the most from people and situations when I’ve been willing to make myself vulnerable, to be fully present, and to authentically engage others. As in the above-mentioned focus group, and in many proceeding that, I recognize that when I’ve allowed myself to be open and available—to be unconcerned with knowing all the right answers, in advance— indeed, when I’ve made myself vulnerable and present—is precisely when I’ve learned the most important lessons and gained the most insight into a given phenomenon.
Successful program evaluations require effective, constant, and adaptive learning—often in fluid, uncertain, and continually evolving contexts. Genuine learning occurs when we make ourselves vulnerable enough to sincerely engage others, to connect with them, and to acknowledge that what we don’t know is the first step toward gaining knowledge, toward genuine knowledge. To learn more about our adaptive approach to evaluation visit our Feedback & Continuous improvement page.
“Listening to Those Who Matter Most, the Beneficiaries,” (Spring, 2013 Stanford Social Innovation Review) highlights the importance of incorporating the perspectives of program beneficiaries (participants, clients, service recipients, etc.) into program evaluations. The authors note that non-profit organizations, unlike their counterparts in health care, education, and business, are often not as effective in gathering feedback and input from those who they serve. Although extremely important, the collection of opinions and perspectives from program participants has three fundamental challenges: 1) It can be expensive and time intensive. 2) It is often difficult to collect data, especially with disadvantaged and minimally literate populations. 3) Honest feedback can make us (i.e., program funders and program implementers) uncomfortable, especially if program beneficiaries don’t think that programs are working the way they are supposed to.
As the authors point out, feedback from participants is important for two fundamental reasons: It provides a voice to those who are served. As Bridgespan Group partner, Daniel Stid, notes, “Beneficiaries aren’t buying your service; rather a third party is paying you to provide it to them. Hence the focus shifts more toward the requirements of who is paying, versus the unmet needs and aspirations of those meant to benefit.” Equally importantly, gathering and analyzing the perspectives and opinions of beneficiaries can help program implementers to refine programming and make it more effective.
The authors of “Listening to Those Who Matter Most, the Beneficiaries,” make a strong case for systematically collecting and utilizing beneficiary input. “Beneficiary Feedback isn’t just the right thing to do, it is the smart thing to do.”
Our experience in designing and conducting program evaluations has shown the value of soliciting the views, perspectives, and narrative experiences of program beneficiaries. Beneficiary feedback is a fundamental component of our program evaluations—whether we are evaluating programs that serve homeless mothers, or programs that serve college students. We’ve had 20 years of experience conducting interviews, focus groups, and surveys, which are designed to efficiently gather and productively use information from program participants. While the authors of “Listening to Those Who Matter Most, the Beneficiaries,” suggest that such efforts can be resource intensive, and indeed they can be, we’ve developed strategies for maximizing the effectiveness of these techniques while minimizing the cost of their implementation. To learn more about our evaluation methods visit our Data collection & Outcome measurement page.
Not long ago I was meeting with a prospective client. It was our first meeting, and shortly after our initial conversation had begun—but long before we had a chance to discuss either the purposes of the evaluation, the questions the evaluation would address, or the methods that would be used— the client began imagining the many marketing uses for the evaluation’s findings. Eager to dissuade my colleague from prematurely celebrating her program’s successes, I observed that while it may turn out that the evaluation would reveal important information about the program’s achievements and benefits, it might also find that the program had, in fact, not achieved some of the goals it had set out to realize. I cautioned my colleague, “My experience tells me that we will want to wait to see what the evaluation shows, before making plans to use evaluation findings for marketing purposes.” In essence, I was making a case for discovering the truth of the program before launching an advertising campaign.
We live in a period where demands for accountability and systematic documentation of program achievements are pervasive. Understandably, grantees and program managers are eager to demonstrate the benefits of their programs. Indeed, many organizations rely upon evaluation findings to demonstrate to current and future funders that they are making a difference and that their programs are worthy of continued funding. Despite these pressures, it is very important that program evaluations be conducted with the utmost integrity and objectivity so that findings are accurate and useful to all stakeholders.
The integrity of the evaluation is critical, indeed, paramount for all stakeholders. Reliable, robust, and unbiased evaluation findings are important not just to funders who want to know whether their financial resources were used wisely, but also to the program implementers, who need to know whether they are making the difference(s) they intend to make. Without objective data about the outcomes that a program produces, no one can know with any certainty whether a program is a success or a “failure” (take a look out our blog post Fail Forward, that examines what we can learn from “failure”) i.e., if it needs refining and strengthening.
As a member of the American Evaluation Association, Brad Rose Consulting, Inc. is committed to upholding the AEA’s “Guiding Principles for Evaluators”, the Principles state:
“Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process. Evaluators should be explicit about their own, their clients’, and other stakeholders’ interests and values concerning the conduct and outcomes of an evaluation (and)… should not misrepresent their procedures, data or findings. Within reasonable limits, they should attempt to prevent or correct misuse of their work by others.”
In each engagement, Brad Rose Consulting, Inc. adheres to the “Guiding Principles for Evaluators” because we are committed to ensuring that the findings of its evaluations are clearly and honestly represented to all stakeholders. This commitment is critical not just to program sponsors—the people who pay for programs— but also to program managers and implementers, who also are in need of unbiased and dispassionate information about the results of the programs they operate. To learn about the evaluation methods we offer visit our Data collection & Outcome measurement page.
On March 26, at Bryant University, in Smithfield Rhode Island, Brad Rose presented an introductory workshop about program evaluation to members of Rhode Island’s non-profit and foundation communities. Attendees included the state humanities council, local foundations, historic preservation councils, anti-violence programs, mentoring programs for children of prisoners, dance programs, youth leadership programs, libraries, and others.
After outlining the fundamentals of program evaluation, Brad discussed a variety of approaches to evaluating programs. Brad stressed that program evaluations work best when they are driven by key questions, rather than by pre-determined methodologies. He told attendees that just as form follow function, evaluation design should be appropriate to the kinds of evaluation questions that need to be answered. In an interactive session, attendees completed draft logic models of their respective programs, and shared these with the group (to view an example of a logic model click here). Following the workshop, feedback from Dr. Sandra Enos, the organizer of the workshop, indicated that attendees were very pleased with the workshop, finding that it provided a practical, experienced-based perspective that was especially useful to the non-profit and foundation communities Following the meeting, Dr. Enos said, “We would love to host Brad again in a working session to support the agencies in the detailed development of their evaluation designs.”
Click here to download Brad’s presentation.
Periodically, I discover and like to share links to resources related to program evaluation. I think these links can be useful for colleagues in the non-profit, foundation, education, and government sectors. Here are some links that may be of interest.
Links to typical program outcomes and indicators are available from the Urban Institute. These links include outcomes for a variety of programs, including arts programs, youth mentoring programs, and advocacy programs. All in all, this site has outcomes and indicators for 14 specific program areas.
The perspective of program participants is a very important source of data about the effects of program. Yet this perspective is often overlooked. An article from the Stanford Social Innovation Review entitled “Listening to Those Who Matter Most, the Beneficiaries” provides good insight into why program participant perspective is so valuable.
The Foundation Center has a database of over 150 tools for assessing the social impact of programs. While there are dozens and dozens of useful tools here for you to browse, take a look at “A Guide to Actionable Measurement,” from the Gates Foundation and “Framework for Evaluation” from the CDC.
The Annie E. Casey Foundation A HANDBOOK OF DATA COLLECTION TOOLS: COMPANION TO “A GUIDE TO MEASURING ADVOCACY AND POLICY” may be helpful for organizations seeking to effect changes in public perceptions and public policy.
In up-coming posts, I will share additional links to tools and resources.
In his recent article in the New York Times, “What Data Can’t Do” (February 18, 2013, visit here ), David Brooks discusses some of the limits of “data.”
Brooks writes that we now live in a world that is saturated with gargantuan data collection capabilities, and that today’s powerful computers are able to handle huge data sets which “can now make sense of mind-bogglingly complex situations.” Despite these analytical capacities, there are a number of things that data can’t do very well. Brooks remarks that data is unable to fully understand the social world; often fails to integrate and deal with the quality (vs. quantity) of social interactions; and struggles to make sense of the “context,” i.e., the real environments, in which human decisions and human interactions are inevitably embedded. (See our earlier blog post “Context Is Critical”.)
Brooks insightfully notes that data often “obscures values,” by which he means that data often conceals the implicit assumptions, perspectives, and theories on which they are based. “Data is never ‘raw,’ it’s always structured according to somebody’s predispositions and values.” Data is always a selection of information. What counts as data depends upon what kinds of information the researcher values and thinks is important.
Program evaluations necessarily depend on the collection and analysis of data because data constitutes important measures and indicators of a program’s operation and results. While evaluations require data, it is important to note that data alone while, necessary, is insufficient for telling the complete story about a program and its effects. To get at the truth of a program, it is necessary to 1) discuss both the benefits and limitations of what constitutes “the data”—to understand what counts as evidence; 2) to use multiple kinds of data—both quantitative and qualitative; and 3) to employ experience- based judgment when interpreting the meaning of data.
Brad Rose Consulting, Inc., addresses the limitations pointed out by David Brooks, by working with clients and program stakeholders to identify what counts as “data,” and by collecting and analyzing multiple forms of data. We typically use a multi-method evaluation strategy, one which relies on both quantitative and qualitative measures. Most importantly, we bring to each evaluation project our experience-based judgment when interpreting the meaning of data, because we know that to fully understand what a program achieves (or doesn’t achieve), evaluators need robust experience so that they can transform mere information into genuine, useable knowledge. To learn about our diverse evaluation methods visit our Data collection & Outcome measurement page.
Although there are a number of perfectly good reasons that an organization may choose to create and maintain an internal program evaluation capacity, there are number of very good reasons, indeed, advantages, associated with the use of an external evaluator.
Breadth of Experience
External evaluators bring a breadth of experience evaluating a range of programs. Such eclectic and wide-ranging experience can be especially useful when evaluating innovative programs that seek to creatively serve their target populations. Evaluators who have worked in a variety of program contexts and who have worked with a diversity of program stakeholders can draw on their experience to inform current evaluation initiatives. External evaluators have often “seen” an abundance of programs, and the resulting knowledge can be a substantial asset to the organization that engages their professional services.
External evaluators are often more disinterested and objective in their view of a program and its outcomes. External evaluators are less susceptible to the internal politics of organizations, have less of an economic ‘stake’ in the success or failure of a program, and therefore are better positioned to provide an unbiased eye with which to conduct a program evaluation. Objectivity is critical for discovering whether a program really works or not and is essential if program stakeholders are to know whether a program achieves its intended results.
While internal evaluators may be highly skilled, external evaluators often mobilize a range of expertise and technical skills that internal evaluators under-develop. Professional evaluators specialize in developing their skills, and apply their expertise to a variety of programs, they often have a superior ‘quiver’ of evaluation knowledge and wisdom. Professional expertise is the province of specialization, and career program evaluators necessarily develop rich and deep evaluation expertise.
An external evaluator can be very cost effective, especially for smaller and mid-sized organizations (local non-profits, community-based organizations, school districts, family and community foundations, colleges, etc.) that may not have sufficient resources with which to fund and maintain an internal evaluation capacity. External evaluators are able to contain and reduce infrastructure costs, and therefore are comparatively inexpensive for clients.
With 20 years of experience in providing program evaluation services to its clients in the non-profit, foundation, education, health, and community service sectors, Brad Rose Consulting, Inc. is able to provide cost effective, customized, and high-value, program evaluations to its clients. More information about the kinds of program evaluation services we provide is available here.
What is Action Research?
Action Research is a method of applied, often organization-based, research whose fundamental tenet is that we learn through action, learn through doing and reflecting. Action research is used in “real world” situations, rather than in ideal, experimental conditions. It focuses on solving real world problems.
Although there are a number of strands of Action Research (AR) including: participatory action research, emancipatory research, co-operative inquiry, appreciative inquiry, all share a commitment to positively changing a concrete organizational or social challenge through a deliberate process of taking action, and reflecting on cycles of emergent learning. Kurt Lewin, one of the original theorists of AR said that: “If you want truly to understand something, try to change it.” Ultimately, Action Research is about learning though doing, indeed, learning through changing.
Collaboration and Co-Learning
Although Action Research uses many of the same methodologies as positivist empirical science (observation, collection of data, etc.), AR typically involves collaborating with, and gathering input from, the people who are likely to be affected by the research. As Gilmore, Krantz, and Ramirez point out in their article “Action Based Modes of Inquiry and the Host-Researcher Relationship,” Consultation 5.3 (Fall 1986): 161, “… there is a dual commitment in action research to study a system and concurrently to collaborate with members of the system in changing it in what is together regarded as a desirable direction. Accomplishing this twin goal requires the active collaboration of researcher and client, and thus it stresses the importance of co-learning as a primary aspect of the research process.”
(Retrieved from http://www.web.ca/robrien/papers/arfinal.html#_edn1)
Collaboration, Stakeholder Involvement, and Constructive Judgment for Program Strengthening
Brad Rose Consulting draws on the key ideas Action Research–collaboration and stakeholder involvement— to ensure that its evaluations are grounded in, and reflect the experience of program stakeholders. Because we work at the intersection of program evaluation (i.e. Does a program produce its intended results?) and organization development (i.e. How can an organization’s performance be enhanced, and how can we ensure that it better achieves it goals and purposes?), we know that the success of program evaluations depend in large part upon the involvement of all program stakeholders.
This means that we work with program stakeholders, (e.g. program managers, program staff, program participants, community members, etc.) to understand the intentions, processes, and experiences of each group. Brad Rose Consulting, Inc., begins each evaluation engagement by listening to clients and participants—including listening to their aspirations, their needs, their understanding of program objectives, and their experience (both positive and negative) participating in programs. Furthermore, we engage clients not as passive recipients of “expert knowledge,” but rather as co-learners who seek both to understand if a program is working, and how a program can be strengthened to better achieve its goals.
Ultimately we make evaluative judgments about the effectiveness of a program, but our approach to making such judgments is guided by our commitment to constructive judgments that help clients to achieve both intended programmatic outcomes (program results) and desired organizational goals. To learn more about our adaptive approach to evaluation visit our Feedback & Continuous improvement page.
“Context matters,” Debra Rog, past President of the American Evaluation Association reported in her address to the 2012 meeting of ASA, and recently wrote in her insightful article, “When Background Becomes Foreground: Toward Context-Sensitive Evaluation Practice (New Directions for Evaluation No.135, Fall, 2012)
Indeed, as Rog correctly points out, program evaluators (and program sponsors) often ignore at their own peril, program context. Too often evaluations begin with attention focused on methodology, then later—often when it’s too late–discover that the results, meanings, and uses, of evaluation findings have been hampered by insufficiently considering the context in which an evaluation is conducted.
Five Aspects of Context
Rog says there are 5 aspects of context that directly and indirectly affect the selection, design, and ultimate success of an evaluation:
1. the nature of the problem that a program or intervention seeks to address
2. the nature of the intervention—how the structure, complexity, and dynamics of the program (including program life cycle) affect the selection and implementation of the evaluation approach
3. the broader environment (or setting) in which the program is situated and operates (for example, the availability of affordable housing may profoundly affect the outcomes and successes of a program intended to assist homeless persons to accesses housing.)
4. the parameters of the evaluation, including the evaluation budget, allotted time for implementation of evaluation activities, and the availability of data
5. the decision-making context for the evaluation—who are the decision-makers that will use evaluation findings, which types of decisions do they need to make, and whcih standards of rigor and levels of confidence do decisions-makers require
Context-Sensitive Evaluations – What to Look For
Rog underscores the importance of conducting “context-sensitive” evaluations—evaluations that first consider the various aspects of the context in which programs operate and in which program evaluation activities will occur. She makes a plea to evaluators and evaluation sponsors to refrain from engaging in a “method’s first approach” which too often fetishizes methodologies at the cost of conducting appropriate evaluations that can be of maximum value and use to all program stakeholders.
In our experience, the most effective context-sensitive evaluations address:
- Who program stakeholders are (funders, program managers, program participants, community members, etc.)
- What stakeholders need to learn about a program’s operation and outcomes
- The social and economic context of the program’s operation
- Key questions to guide the evaluation research
- Research methodologies that provide robust and cost-effective findings
- A logic model that clearly specifies the program activities and results
- A wide range of evaluation research methodologies and data collection strategies to ensure that program results are systematically and rigorously measured.
- Clear, accessible reports so that all stakeholders benefit from evaluation findings
- Detailed recommendations for how sponsors and program managers can strengthen further efforts
When it comes to program evaluation, not only does context matter, it is on the critical path for getting the best results. To find out about how we consider context in our evaluation methods visit our Data collection & Outcome measurement page.
Vilfredo Pareto, an Italian economist, sociologist, and philosopher, observed: “Give me a fruitful error any time, full of seeds, bursting with its own corrections. You can keep your sterile truth for yourself.” Programs frequently achieve some of their original goals, while missing others. In some cases, they achieve unintended results, both desirable and undesirable. In other cases, programs fail to achieve any of the original outcomes (i.e., changes, results) they intended at all.
Discovering New Information in Failure
Rather than identifying a program’s unachieved results as merely program failure, we need to rethink what we can learn from failures. What do program failures tell us about what we do that’s ineffective or otherwise misses the mark? By understanding how and why programs don’t achieve the results they intend, we can design and execute improved programs in the future. It is important to note that psychological research has shown that individuals learn more from failure than they do from success. Our goals should be to learn from our defeats and to surmount them—especially in programs that address critical social, educational, and human service needs. Learning from the challenges that confront these kinds of programs can have a powerful impact of the success of future programming.
Brad Rose Consulting, Inc. is committed to conducting program evaluations that help program managers, funders, and stakeholders to ensure successful program design, accurately measure results, and make timely adjustments in order to maximize positive program impacts. To learn more about our commitment to learning from failure visit our Feedback & Continuous improvement page.
Samuel Becket wrote, “Ever tried. Ever Failed. No matter. Try again. Fail again. Fail better.”
See: “Embracing Failure,” an article at the Asian Development Bank
The Mellon Mays Undergraduate Fellowship is an initiative to to reduce the serious under-representation of minorities in the faculties of higher education. MMUF commissioned Brad Rose Consulting to do an evaluation of their success over the past 10 years. Here are Brad Rose’s findings. The report is a fantastic example of what you can expect from a commissioned evaluation.
To learn more about our work in education visit our Higher education & K-12 page.
A logic model/theory of change can be a very useful learning tool, which, when generated collaboratively with clients, helps evaluators and clients to clearly understand the workings and goals of a program. It can also be extremely helpful in developing the most accurate measures of a program’s intended results. Click on the thumbnail to learn more about the easy-to-understand logic model/theory of change tool that Brad Rose Consulting uses to illustrate the way a program works and how it achieves the results/changes that it seeks to realize.
Before beginning an evaluation, it may be helpful to consider the following questions:
1. Why is the evaluation being conducted? What is/are the purpose(s) of the evaluation?
Common reasons for conducting an evaluation are to:
- monitor progress of program implementation and provide formative feedback to designers and program managers (i.e., a formative evaluation seeks to discover what is happening and why, for the purpose of program improvement and refinement.)
- measure final outcomes or effects produced by the program (i.e., a summative evaluation);
- provide evidence of a program’s achievements to current or future funders;
- convince skeptics or opponents of the value of the program;
- elucidate important lessons and contribute to public knowledge;
- tell a meaningful and important story;
- provide information on program efficiency;
- neutrally and impartially document the changes produced in clients or systems;
- fulfill contractual obligations;
- advocate for the expansion or reduction of a program with current and/or additional funders.
Evaluations may simultaneously serve many purposes. For the purpose of clarity and to ensure that evaluation findings meet the client’s and stakeholders’ needs, the client and evaluator may want to identify and rank the top two or three reasons for conducting the evaluation. Clarifying the purpose(s) of the evaluation early in the process will maximize the usefulness of the
2. What is the “it” that is being evaluated? (A program, initiative, organization, network, set of processes or relationships, services, activities?) There are many things that may be evaluated in any given program or intervention. It may be best to start with a few (2-4) key questions and concerns (See #4, below ). Also, for purposes of clarity, it may be useful to discuss what isn’t being evaluated.
3. What are the intended outcomes that the program or intervention intends to produce? What is the program meant to achieve? What changes or differences does the program hope to produce, and in whom? What will be different as the result of the program or intervention? Please note that changes can occur in individuals, organizations, communities, and other social environments. While evaluations often look for changes in persons, changes need not be restricted to alterations in individuals’ behavior, attitudes, or knowledge, but can extend to larger units of analysis, like changes in organizations, networks of organizations, and communities. For collective groups or institutions, changes may occur in: policies, positions, vision/mission, collective actions, communication, overall effectiveness, public perception, etc. For individuals: changes may occur in behaviors, attitudes, skills, ideas, competencies, etc.
4. Every evaluation should have some basic questions that it seeks to answer. What are the key questions to be answered by the evaluation? What do clients want to be able to say (report) about the program to key stakeholders? By collaboratively defining key questions, the evaluator and the client will sharpen the focus the evaluation and maximize the clarity and usefulness of the evaluation findings.
5. Who is the evaluation for? Who are the major stakeholders or interested parties for evaluation results? Who wants to know? Who are the various “end users” of the evaluation findings?
6. How will evaluation findings be used? (To improve the program; to make judgements about the economic or social value of the program—it’s costs and benefits–; to document and publicize efforts; to expand or curtail the program?)
7. What information will stakeholders find useful and how will they use the evaluation findings?
8. What will be the “product” of the evaluation? What form will findings take? How are findings to be disseminated? (Written report, periodic briefings, analytic memos, a briefing paper, public presentation, etc.?)
9. What are the potential sources of information/data? (Interviews, program documents, surveys, quantitative/statistical data, comparison with other/similar programs, field observations, testimony of experts?) What are the most accessible and cost effective sources of information for the client?
10. What is the optimal design for the evaluation? Which Methods will yield the most valid, accurate, and persuasive evaluation conclusions? If interested in indicating a cause and effect relationship, is it possible (and desirable) to expend the resources necessary to conduct an experimental (i.e., “control group study”), or quasi-experimental design study? Experimental designs can be resource intensive and therefore, more costly. If resources are not substantial, the client and the evaluator will want to discuss other kinds of evaluation designs that will provide stakeholders with the most substantial, valid, and persuasive evaluative information.
To learn more about our evaluation methods visit our Data collection & Outcome measurement page.