Program evaluation services that increase program value for grantors, grantees, and communities. x

Archive for category: Resources

November 3, 2020
03 Nov 2020

The Pandemic and Nonprofits


There are few aspects of society that the pandemic is not affecting. Accordingly, COVID-19 is having a substantial effect on the operation and sustenance of nonprofits. A June, 2020 article in the Stanford Social Innovation Review, which summarizes a national survey of 750 members of primarily US based nonprofits, reported that nearly 75% of survey respondents stated their organizations had experienced a drop in revenues, and over 80% had moved all or some of their programs and services to an online format. (See “The Continuing Impact of COVID-19 on the Social Sector”) Eighty percent of surveyed nonprofits are shifting to work from home, many are expecting or already have made reductions in staff, and many are considering mergers with other non-profits. (See also the reported effects on 110 nonprofits in  “The Impact of COVID-19 on Large and Mid-Sized Nonprofits,” June 15, 2020, Independent Sector which reports that 71% of surveyed large and medium sized nonprofits have reduced services.)

In the Nonprofit Quarterly article, “Nonprofits Struggle to Stay Alive amid COVID-19” one nonprofit CEO says “The impact of COVID-19 on the nonprofit community is unprecedented. It has affected the capacity and sustainability of every nonprofit—from education to the environment, affordable housing to mental health services, animal welfare to the arts—no organization will emerge unscathed.” This article goes on to say that in Arizona, “Fundraising and program cancellations as a result of COVID-19 have cost Arizona nonprofits an estimated $53 million in lost revenue (as of June 11). In that same survey, 25 percent of nonprofits indicated that they’ve had to lay off or furlough employees, and 69 percent report a loss of critical program volunteers.”

In a more recent, October, 2020  article “Second Wave of Virus Looms, Some US Nonprofits Running Out of Road”, Ruth McCambridge writes “Nearly every aspect of operations has been shaken. Organizations have had to find new ways to provide their services while staying as close as they can to their stakeholders. Revenues shrank, but expenses did not go away. As communities are reopening and trying—perhaps too quickly—to return to “normal,” we are also learning that no matter how important an organization’s mission might be, things are harder and more expensive in a world with COVID-19 still undefeated.”


How Important Are Nonprofits?

It’s vital to keep in mind not only the social role, but the economic importance of the nonprofit sector. The National Council on Nonprofits reports that, “Nonprofits employ 12.3 million people, with payrolls exceeding those of most other U.S. industries, including construction, transportation, and finance. A substantial portion of the nearly $2 trillion nonprofits spend annually is the more than $826 billion they spend on salaries, benefits, and payroll taxes every year. Also, nonprofit staff members pay taxes on their salaries, as well as sales taxes on their purchases and property taxes on what they own.”  (See: “Economic Impact“) The Urban Institute says that “Approximately 1.54 million nonprofits were registered with the Internal Revenue Service (IRS) in 2016, an increase of 4.5 percent from 2006.”  (See “The Nonprofit Sector in Brief” Urban Institute) “ The United States non-profit sector alone would rank as the 17th largest economy in the world.” (See “The Economic Impact of Nonprofit Organizations —Part One” by Andrew Paniello)



While the 2020 COVID-19 pandemic brings multiple challenges, might there be things that nonprofits can do to adapt and sustain themselves? In “Three Things Nonprofits Should Prioritize in the Wake of COVID-19” by Amy Celep, Megan Coolidge & Lori Bartczak, in the April, 30, 2020 Stanford Social Innovation Review, the authors say that nonprofits may choose to revisit their purposes and value in the current environment, in order to rally donors, engage partners, and motivate staff. They also suggest that nonprofits: 1) assess their financial situations, including understanding how they depend on various revenue streams; 2) create various financial scenarios for best, worst, and most likely financial scenarios; and 3) have frank conversations with donors and stakeholder about their plans and intentions so that these nonprofits can get some sense of the reliability of contributed revenue streams. They note that “Many organizations are finding success in asking customers or members to donate pre-paid fees for canceled services and events to help programs and services they care about continue in the future.”

Although there are certainly new opportunities for nonprofits to rethink their operations and to create new approaches to fundraising, it is likely that the nonprofit sector will continue to face rather steep challenges in the months and years ahead. In fact, many of the challenges confronting nonprofits began before the onset of the pandemic. The ameliorative goals of many non-profits had already been challenged by the operation of the “normal” economy, which favored the well-positioned and affluent, and created ever greater needs among ever larger swaths of the American population for the many services that nonprofits have delivered.

For additional resources and ideas that will help nonprofits to continue to serve their communities see “Nonprofits and Corona virus, COVID-19,” National Council of Nonprofits.



COVID-19’s Impact on Nonprofits’ Revenues, Digitization, and Mergers,” by David La Piana, Stanford Social Innovation Review Jun. 4, 2020
As Second Wave of Virus Looms, Some US Nonprofits Running out of Road” by Ruth McCambridge, Oct 20, 2020, Nonprofit 

Nonprofits Struggle to Stay Alive amid COVID-19” by Martin Levine, June 23, 2020,  NonProfit Quarterly

Three Things Nonprofits Should Prioritize in the Wake of COVID-19” by Amy Celep, Megan Coolidge & Lori Bartczak   Apr. 30, 2020 Stanford Social Innovation Review.

The Impact of COVID-19 on Large and Mid-Sized Nonprofits,” June 15, 2020, Independent Sector

Nonprofits and Coronavirus, COVID-19,” National Council of Nonprofits

Economic Impact,” National Council of Nonprofits.
September 24, 2019
24 Sep 2019

It’s Not Just Your Credit Card Score – The Erosion of Privacy

What is Privacy Good For?

The right to privacy is a much-cherished value in America. As we noted in an earlier article, “Transparent as a Jellyfish? Why Privacy is Important” privacy is crucial to the development of a person’s autonomy and subjectivity. When privacy is reduced by surveillance or restrictive interference—either by governments or corporations—such interference may not just affect our social and political freedoms, but undermine the preconditions for the fundamental development and sustenance of the self.

Danial Solove, Professor of Law at George Washington University Law School, lists ten important reasons for privacy including: limiting the power of government and corporations over individuals; the need to establish important social boundaries; creating trust; and as a precondition for freedom of speech and thought. Solove also notes, “Privacy enables people to manage their reputations. How we are judged by others affects our opportunities, friendships, and overall well-being.” (See “Ten Reasons Why Privacy Matters” Danial Solove). Julie Cohen, in “What is Privacy For?” Harvard Law Review, Vol. 126, 2013 writes: “Privacy shelters dynamic, emergent subjectivity from the efforts of commercial and government actors to render individuals and communities fixed, transparent, and predictable. It protects the situated practices of boundary management through which self-definition and the capacity for self-reflection develop.”

Strains on Privacy

Privacy, of course, is under continual strain. In his recent article, “Uh-oh: Silicon Valley is building a Chinese-style social credit system,” (Fast Company, August 8, 2019) Mike Elgan notes that China is not alone in seeking to create a “social credit” system—a system that monitors and rewards/punishes citizen behavior.

China’s state-run system would seem to be extreme (e.g., it rewards and punishes for such things as failure to pay debts, excessive video gaming, criticizing the government, late payments, failing to sweep the sidewalk in front of your store or house, smoking or playing loud music on trains, jaywalking, etc. It also publishes lists of citizens’ social credit ratings, and uses public shaming as a means to enforce desired behavior.) Elgan notes that Silicon Valley has similar designs on monitoring and motivating what it deems as “desirable and undesirable” behavior. The outlines of an ever-evolving corporate-sponsored, technology-based “social credit” system now include:

  • Life insurance companies can base premiums on what they find in your social media posts
  • Airbnb—now has more than 6 million listings in its system, and the company can ban customers and limit their travel/accommodation choices. Airbnb can disable your account for life for any reason it chooses, and it reserves the right to not tell you the reason.
  • PatronScan, an ID-reading service helps restaurants and bars to spot fake IDs—and troublemakers. The company maintains a list of objectionable customers which is designed to protect venues from people previously removed for “fighting, sexual assault, drugs, theft, and other bad behavior,” A “public” list is shared among all PatronScan customers.Under a new policy Uber announced in May: If your average rating is “significantly below average,” Uber will ban you from the service.
  • WhatsApp is, in much of the world today, the main form of electronic communication. Users can be blocked if too many other users block you. Not being allowed to use WhatsApp in some countries is as punishing as not being allowed to use the telephone system in America.

The Consequences

While no one wants to endorse “bad behavior,” ceding the power to corporations and technology giants to determine which behavior counts as undesirable and punishable may not be the most just or democratic way to ensure societal norms and expectations. As Elgan observes, “The most disturbing attribute of a social credit system is not that it’s invasive, but that it’s extra-legal. Crimes are punished outside the legal system, which means no presumption of innocence, no legal representation, no judge, no jury, and often no appeal. In other words, it’s an alternative legal system where the accused have fewer rights.” Even more ominously, as Julie Cohen writes, “Conditions of diminished privacy shrink the capacity (of self government), because they impair both the capacity and the scope for the practice of citizenship. But a liberal democratic society cannot sustain itself without citizens who possess the capacity for democratic self-government. A society that permits the unchecked ascendancy of surveillance infrastructures cannot hope to remain a liberal democracy.”



“America Isn’t Far Off from China’s ‘Social Credit Score’” Anthony Davenport, Observer, February 19, 2018.

“How the West Got China’s Social Credit System Wrong,”  Lousse Matsakis, Wired, July 29. 2019

“Ten Reasons Why Privacy Matters” Daniel Solove

“What Privacy Is For?” Julie Cohen, Harvard Law Review, Vol. 126, 2013

“The Spy in Your Wallet: Credit Cards Have a Privacy Problem,” Geoffrey A. Fowler, The Washington Post, August 26, 2019.


September 3, 2019
03 Sep 2019

What is “Normal”?

Why do so many of us aspire to be “normal?” Who decides what’s normal and abnormal? What happens to our self- and social- worth when we discover that we aren’t “normal?” In a recent article, “How Did We Come Up with What Counts as Normal,” Jonathan Mooney discusses the rise of an idea that has acquired substantial power in modern society. Mooney notes that “normal’ entered the English language only in the mid-19th century and has its roots in the Latin “norma” which refers to the carpenter’s T-Square. It originally meant simply “perpendicular.” Right-angles however, are considered mathematically “good” and “normal” soon came to be associated not just with their description of the orthogonal angle, but also with the normative notion of something that is desirable or socially expected. Mooney argues that it is this ambiguity as both a descriptive word and as a normative ideal, that makes “normal” so appealing and powerful.

“Normal” was first used in the academic disciplines of comparative anatomy and physiology. For academics in these and other fields, “normal” soon evolved to describe bodies and organs that were “perfect” or “ideal” and also was used to name certain states as “natural”. Eventually, thanks largely to the field of statistics, ideas about the normal soon conflated the average with the ideal or perfect. In the 19th century, for example, Adolphe Quetelet, a deep believer in the power of statistics, advanced the idea of the “average man” and argued that “the normal” (i.e., average) was perfect and beautiful. Quetelet characterized that which was not “normal” not simply as “abnormal,” or non-average, but as something potentially monstrous. “In 1870, in a series of essays on “deformities” in children, he juxtaposed children with disabilities to the normal proportions of other human bodies, which he calculated using averages.” Thus, averages soon became the aspirant ideal.

Mooney also describes how the statistician Francis Galton, who was Charles Darwin’s cousin, “…was both the first person to develop a properly statistical theory of the normal . . . . and also the first person to suggest that it be applied as a practice of social and biological normalization.” “By the early twentieth century, the concept of a normal man took hold. Soon, the emerging field of public health embraced the idea of the normal; schools, with rows of desks and a one-size-fits-all approach to learning, were designed for the mythical middle; and the industrial economy sought standardization, which was brought about by the application of averages, standards, and norms to industrial production. Moreover, eugenics, an offshoot of genetics created by Galton, was committed to ridding the world of human “defectives.”

The ensuing predominance (some might say “domination”) of “the normal” became firmly established by the mid-20th century. Mooney points out however, that the normal was not so much “discovered” as it was invented, largely by statistics and statisticians, and promulgated by the social sciences and moralists. “Alain Desrosières, a renowned historian of statistics wrote, “With the power deployed by statistical thought, the diversity inherent in living creatures was reduced to an inessential spread of “errors” and the average was held up as the normal—as a literal, moral, and intellectual ideal.”


“How Did We Come Up with What Counts as Normal,” Jonathan Mooney, Literary Hub August 16, 2019

Normal Sucks: How to Live, Learn, and Thrive Outside the Lines, Jonathan Mooney, Henry Holt and Co., 2019

“Ranking, Rating, and Measuring Everything”

For information on social norms (formal and informal norms, morays, folkways, etc.) see and “What is a Norm?”

May 7, 2019
07 May 2019

Meritocracy: Who Deserves to Succeed?

Meritocracy is a system in which skills, ability, talent, and knowledge are thought to be the best basis for promoting people to positions of power and social standing. Advancement in a meritocratic system is based on performance, typically as measured through examination, or otherwise demonstrated achievement. Meritocracies can be found as far back as 6th century BC China, where an administrative meritocracy was based on civil service examinations, rather than inherited offices. In contemporary England, there is a Meritocratic political party which believes, among other things, that there should be a 100% inheritance tax, so that the super-rich can’t pass on their wealth to a select few (their privileged children) and that every child should get an equal chance to succeed in life. Needless to say, a fully realized meritocracy could go a long way to ending elite dynasties and hereditary monarchy.

Ironically, the term ‘meritocracy’ was coined as a satirical slur in a dystopic novel, The Rise of the Meritocracy, 1870–2033, published in 1958, by the British sociologist and Labor party politician Michael Young. The Rise of the Meritocracy imagines a world in which social class and inherited position has been replaced by a system that promotes those to the top those who have advanced educationally as evidenced via rigid testing and objective standards. The book, however, argues that meritocracy doesn’t eliminate ruling elites, but simply ends up recreating a new class system by means of education and testing. As the conservative commentator, Toby Young, (the son of the author of Rise of the Meritocracy) recently observed “(there is) the tendency within meritocracies for the cognitive elite to become a self-perpetuating oligarchy.”

For many years, the US has been thought to be predominately a meritocracy–one in which the social power of inheritance and privilege had been superseded by a system in which leaders and socially prominent persons are those who possess superior knowledge and talent. A spate of recent articles, some of which were precipitated by the recent college-admissions scandal (See, for example. “A History of College Admissions Schemes, From Encoded Pencils to Paid Stand-Ins,” Adeel Hassan, March 15, 2019, New York Times) calls into question many of the assumptions about the benefits of meritocracy, and suggests that meritocracy has not yet been realized.

There are, of course, a number of criticisms of meritocracy and the concept of “merit” on which it is based. Among these:

  • What counts as meritorious and who decides which qualities, skills, and knowledge are worthy of merit?
  • In educational systems, do standardized tests and other measures of merit accurately and thoroughly indicate merit/worth?
  • Is meritocracy a kind of “social Darwinism,” in which the survival (and promotion) of the physically “fittest,” is replaced by the survival of the cognitively “smartest” (i.e. the best test takers)?
  • Does wealth and inheritance affect individuals’ ability to obtain the educational credentials by which meritocracy is demonstrated? (For example, does the level of education required for a person to become competitive in a meritocracy discriminate against those who are unable to afford the often expensive and time-consuming “markers” that an education affords?)
  • Does meritocracy, despite its original anti-elitist intentions, merely recreate another kind of permanent elite?

The ultimate question is whether a meritocracy is the best we can do? While meritocracy is problematic, is there a fairer system to replace it? (See for example Richard Dawkins brief discussion, “Democracy or Meritocracy: Which is the Government of Reason?”)


“Meritocracy: Real or Myth?”

“A History of College Admissions Schemes, From Encoded Pencils to Paid Stand-Ins,” Adeel Hassan, March 15, 2019, New York Times

“A ‘Meritocracy’ Is Not What People Think It Is.” Ben Zimmer, The Atlantic March 14, 2019

“The Scandals of Meritocracy,” Ross Douthat, New York Times, March 16, 2019

The Big Test: The Secret History of the American Meritocracy, Nicholas Lemann, Farrar, Straus and Giroux; 1st edition (October 1, 1999)

“Meritocracy” at Wikipedia

“What’s (still) wrong with meritocracy” Toby Young, The Spectator

“College Admission Scandal,” various authors, New York Times

April 23, 2019
23 Apr 2019

Talking Back to Foundations

Philanthropies play an important role in contemporary society. They are, by their very nature, focused on supporting the public good and human welfare. While philanthropies, each year, channel vast sums to the achievement of laudable goals, their social and economic power has raised questions about their unalloyed reputation for ‘doing good’. See, for example, our previous blogpost, “Philanthrocapitalism?”

In an attempt to provide a voice to those who have worked with foundations, a new website, GrantAdvisor, offers grantees of foundations a safe way to anonymously give feedback about the grantmaking/receiving process. GrantAdvisor also offers foundations an opportunity to learn about grantees’ experience working with foundation staff.

GrantAdvisor effectively serves as a kind “Yelp” to those in the non-profit community. Here for example is a link to grantees experience working with the Wal-Mart Foundation.

You can view GrantAdvisor here to learn more about how the site works and to review the reviews of a number of important national and local foundations.


“A Place Where You Can Speak Your Mind to That Foundation,” Amy Costello. Non Profit Quarterly

“Benchmarking Foundation Evaluation Practices,” Center for Effective Philanthropy

“Helping Community Foundations Strengthen Grantees’ Effectiveness”

March 19, 2019
19 Mar 2019

Glitches in Philanthropy?

Over the years, Brad Rose Consulting has provided evaluation services to philanthropies and community service organizations. These clients are dedicated to making the world a better place, often through philanthropic work with disadvantaged populations. While the work of philanthropies is generally perceived as laudable, there are a number of potential objections to the rise of philanthropic largesse.

In a BBC article, “The Problems with Charity” the authors survey a number of potential objections to, and liabilities associated with, charitable and philanthropic work. These include:
  • Charities often target symptoms, not causes- Charity helps the recipient with their problem, but it doesn’t do much to deal with the causes of that problem.
  • Charity may become a substitute for real justice- The idea that charity is wrong when it’s used to patch up the effects of the fundamental injustices that are built into the structure and values of a society.
  • Charity may not provide the best solution to a problem- Charitable giving may not be the most effective way of solving world poverty. Indeed, charitable giving may even distract from finding the best solution – which might involve a complex rethink of the way the world organizes its economic relationships, and large-scale government initiatives to change people’s conditions.
  • Charity may benefit the state rather than the needy- If the charity sector increases spending in an area also funded by government then there is a risk that government will choose to spend less in that area.
  • Charities are often accountable to the givers not the receivers- because the recipients of charity are often unorganized and the charity doesn’t know their individual identities, it’s often easier for charities to make their performance reports to the givers.

In his article “The Downside of Doing Good,” David Campbell examines recent critiques of philanthropy.  Following Anand Giridharadas (Winners Take All: The Elite Charade of Changing the World) Campbell argues that “wealthy philanthropists and other prominent social change leaders often co-exist in a parallel universe (called) “MarketWorld,” where the best solutions to society’s problems require the same knowhow used in corporate boardrooms. That is because MarketWorld ignores the underlying causes for problems like poverty and hunger.

He further observes that efforts at educational reform funded by such philanthropic luminaries as Bill and Melinda Gates, via support for charter schools, often fail to understand the underlying inequalities that make schools resourced in vastly different ways. “As long as school systems are funded locally, based on property values, students in wealthy communities will have advantages over those residing in poorer ones. However, creating a more equal system to pay for schools would take tax dollars and advantages away from the rich. The wealthy would lose, and the disadvantaged would win. So it’s possible to see the nearly $500 million that billionaires and other rich people have pumped into charter schools and other education reform efforts over the past dozen years, as a way to dodge this problem.”

While philanthropy is likely to remain with us for the foreseeable future, the books and articles mentioned here underline some of the problematic assumptions that haunt philanthropic attempts to ameliorate conditions whose causes are often the deeper, underlying dynamics associated with the societies and economies these philanthropies seek to amend.


Winners Take All: The Elite Charade of Changing the World, by Anand Giridharadas

Just Giving Why Philanthropy Is Failing Democracy and How It Can Do Better, by Rob Reich

“The Problem with Philanthropy,”  David Sirota, In These Times, June 13, 2014
February 19, 2019
19 Feb 2019

Power in Organizations

Most of us spend a good portion of our lives in organizations or indirectly relating to organizations (businesses, non-profits, civic and legal organizations, religious organizations, military and criminal justice organizations, etc.). One might say that in the modern world, we “live in” an environment composed largely of organizations. (See our previous blogposts “Organization Development: What Is It & How Can Evaluation Help?”  and “What’s the Difference? 10 Things You Should Know About Organizations vs. Programs”)

Organizations contain, utilize, and deploy various kinds of social power. Such power is the capacity of individuals and groups to affect, control, or influence outcomes (i.e. changes). Power doesn’t exist in isolation, but in relationships to other individuals and/or groups. If we want to accomplish goals at work—whether these goals are about producing widgets, or making the world a better place—we need to draw on and negotiate various kinds of formal and informal power. Sometimes it may be useful to think of various resources as sources of power. Tangible resources include, money, machinery, physical infrastructure, etc. Less tangible, but no less important, resources may include, authority, social status/prestige, social networks, individuals’ intelligence, professional experience, even social attractiveness and charisma. Both tangible and intangible resources are used as sources of power with which organizations achieve objectives and goals.

In her article, “Types of Powers in Organizations,” Diana Dahl summarizes 7 types of power in organizations. These include:

  1. Coercive Power— a person or group is able to punish others for not following orders has coercive power.
  2. Connection Power— connection power is gained by knowing and being listened to by influential people. Increasing connections and mastering political networking lead to a greater potential for connection power.
  3. Reward Power— the ability to give rewards to other employees. Rewards are not always monetary, such as improved work hours and words of praise.
  4. Legitimate Power (also known as legitimate authority)— when employees believe a person can give orders based on his position within the organization, such as when a manager orders staff members to complete a task and they comply because the orders came from their superior.
  5. Referent Power— people who are liked, respected, and are viewed by other employees as worth emulating. Supervisors who lead by example, treat employees with respect, seek their collaboration and gain the trust of their employees possess referent power.
  6. Informational Power— access to valued information. This power can be quickly fleeting because once the needed information is shared, the person’s power is gone.
  7. Expert Power— the greater a person’s knowledge or specialized skill set, the greater her potential for expert power.

People in organizations must get things accomplished. Having a clear idea of what constitutes organizational power and the kinds of resources that need to mobilized to reach goals, may help us to better navigate what are often complicated and contentious organizations.


Power in Organizations: Structures, Processes and Outcomes, by Richard Hall and Pamela Tolbert Pearson/Prentice Hall, 2005, 9th edition

“Types of Powers in Organizations,” by Diana Dahl ; Updated September 26, 2017

Links to and summaries of “Theories of Organizational Power” at

“Power and Politics in Organizational Life,” Abraham Zaleznik, Harvard Business Review

“Power in Organizations: A Way of Thinking About What You’ve Got, and How to Use It,” Roelf Woldring

Summary of tactics to gain organizational power 

October 24, 2018
24 Oct 2018

Do Work Teams Work?

“Collaboration” and “teamwork” are the catchphrases of the contemporary workplace. Since the 1980s in the U.S., work teams have been hailed as the solution to assembly line workers’ alienation and disaffection, and white-collar workers’ isolation and disconnection. Work teams have been associated with increased productivity, innovation, employee satisfaction, and reduced turnover. Additionally, teams at work are said to have beneficial effects on employee learning, problem-solving, communication, company loyalty, and organizational cohesiveness. Teams are now found throughout the for-profit, non-profit, and governmental sectors, and much of the work of the field of organization development (OD) is devoted to fostering and sustaining teams at work.

In his recent article “Stop Wasting Money on Team Building,” Harvard Business Review, September 11, 2018, Carlos Valdes-Dapena, argues that teams are less effective than many believe them to be. Based on research conducted at Mars, Inc. “a 35 billion dollar global corporation with a commitment to collaboration,” Valdes-Dapena argues that while employees like the idea of teams and team work, employees don’t, in fact, much collaborate in teams. After conducting 125 interviews and administering questionnaires with team members, he writes “If there was one dominant theme from the interviews, it is summarized in this remarkable sentiment: “I really like and value my teammates. And I know we should collaborate more. We just don’t.”

Valdes-Dapena reports that employees “…felt the most clarity about their individual objectives, and felt a strong sense of ownership for the work they were accountable for.” He also shows that “Mars was full of people who loved to get busy on tasks and responsibilities that had their names next to them. It was work they could do exceedingly well, producing results without collaborating. On top of that, they were being affirmed for those results by their bosses and the performance rating system.” Essentially, Valdes-Dapena, argues, teams may sound good in theory, but it is probably better to tap individual self-interest, if you really want to get the job done.

In “3 Types of Dysfunctional Teams and How To Fix Them,” Patty McManus says that there are different types of dysfunctional work teams. She characterizes these different team types as: “The War Zone,” “The Love Fest,” and “The Unteam.” In “War Zone” teams, competition and factionalism among members obscure or derail the potential benefits of teamwork. In the “Love Fest” team, there is a focus on muting disagreements, highlighting areas of agreement, and avoidance of tough issues in the interest of maintaining good feelings. “The Unteam” is characterized by meetings that are used for top-down communication and status updates, and fail to build shared perspective about the organization. In the “Unteam” members may get along as individuals, but they have little connection to one another or a larger purpose they all share.

McManus claims that the problems of teams may be overcome by what she terms “ecosystems teams,” i.e., teams that surface and manage differences, build healthy inter-dependence among members, and engage the organization—beyond the mere confines of the team.

Matthew Swyers also sees problems in teams at work. In “7 Reasons Good Teams Become Dysfunctional,” (Inc. September 27, 2012,) Swyers writes that there are seven types of problems that teams may experience:

  • absence of a strong and competent leader
  • team members more interested in individual glory than achieving team objectives
  • failure to define team goals and desired outcomes
  • disproportionately place too much of the team’s work on a few of its members’ shoulders
  • lack focus and endless debate, without moving toward an ultimate goal
  • lack of accountability and postponed timetables
  • failure of decisiveness.

Each of these writers highlight the vulnerabilities of teams at work. Although the work of these writers doesn’t foreclose the positive possibilities of team organization at work, they raise important questions about both the enthusiasm for, and the effectiveness of, teams. Additionally, each author suggests that with enlightened modifications, organizations can overcome the liabilities of teams and begin to reap the benefits of team-based employee collaboration. That said, none of these writers, and few among the other U.S. based writers who have engaged this topic, treat the underlying assumptions of workplace reform—that work can be made more habitable and humane without the independent organizations that have traditionally represented workers’/employees’ interests in the workplace. For discussion of models of workplace reform that genuinely represent workers’ interest in more humane, collaborative, and ultimately, productive working environments, we will need to look elsewhere.


Workgroups vs. Teams

“Importance of Teamwork at Work,” Tim Zimmer

“Importance of Teamwork in Organizations,” Aaron Marquis

“What Makes Teams Work?” Regina Fazio Maruca, Fast Company

“Stop Wasting Money on Team Building,” Carlos Valdes-Dapena, Harvard Business Review, September 11, 2018

“3 Types Of Dysfunctional Teams And How To Fix Them,” Patty McManus, Fast Company

“When Is Teamwork Really Necessary?” Michael D. Watkins, Harvard Business Review, August 16, 2018

“7 Reasons Good Teams Become Dysfunctional,” Matthew Swyers , Inc. Sept 27 2012

“Why Teams Don’t Work,” Diane Coutu, Harvard Business Review, May 2009

July 31, 2018
31 Jul 2018

Are There Any Questions?

Asking questions is a critical aspect of learning. We’ve previously written about the importance of questions in our blog post “Evaluation Research Interviews: Just Like Good Conversations.” In a recent article, “The Surprising Power of Questions,” which appears in the Harvard Business Review, May-June, 2018, authors Alison Wood Brooks and Leslie K. John offer suggestions for asking better questions.

As Brooks and John report, we often don’t ask enough questions during our conversations. Too often we talk rather than listen. Brooks and John, however, note that recent research shows that by asking good questions and genuinely listening to the answers, we are more likely to achieve both genuine information exchange and effective self-presentation. “Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding.”

Although asking more questions in our conversations is important, the authors show that asking follow-up questions is critical. Follow-up questions “…signal to your conversation partner that you are listening, care, and want to know more. People interacting with a partner who asks lots of follow-up questions tend to feel respected and heard.”

Another critical component of a question-asking is to be sure that we ask open-ended questions, not simply categorial (yes/no) questions. “Open-ended questions …can be particularly useful in uncovering information or learning something new. Indeed, they are wellsprings of innovation—which is often the result of finding the hidden, unexpected answer that no one has thought of before.”

Asking effective questions depends, of course, on the purpose and context of conversations. That said, it is vital to ask questions in an appropriate sequence. Counterintuitively, asking tougher questions first, and leaving easier questions until later “…can make your conversational partner more willing to open up.” On the other hand, asking tough questions too early in the conversation, can seem intrusive and sometimes offensive. If the ultimate goal of the conversation is to build a strong relationship with your interlocutor, especially with someone who you don’t know, or don’t know well, it may be better opening with less sensitive questions and escalate slowly. Tone and attitude are also important: “People are more forthcoming when you ask questions in a casual way, rather than in a buttoned-up, official tone.”

While question-asking is a necessary component of learning, the authors remind us that “The wellspring of all questions is wonder and curiosity and a capacity for delight. We pose and respond to queries in the belief that the magic of a conversation will produce a whole that is greater than the sum of its parts. Sustained personal engagement and motivation—in our lives as well as our work—require that we are always mindful of the transformative joy of asking and answering questions.”


The Surprising Power of Questions,” Alison Wood Brooks and Leslie K. John. Harvard Business Review, May–June 2018 (pp.60–67)

Using Qualitative Interviews in Program Evaluations

July 24, 2018
24 Jul 2018

Learning to Learn

In a recent article in the May 2, 2018 Harvard Business Review, “Learning Is a Learned Behavior. Here’s How to Get Better at It,” Ulrich Boser rejects the idea that our capacities for learning are innate and immutable. He argues, instead, that a growing body of research shows that learners are not born, but made. Boser says that we can all get better at learning how to learn, and that improving our knowledge-acquisition skills is a matter of practicing some basic strategies.

Learning how to learn is a matter of:

  1. setting clear and achievable targets about what we want to learn
  2. developing our metacognition skills (“metacognition” is a fancy way to say thinking about thinking) so that as we learn, we ask ourselves questions like, Could I explain this to a friend? Do I need to get more background knowledge? etc.
  3. reflecting on what we are learning by taking time to “step away” from our deliberate learning activities so that during periods of calm and even mind-wondering, new insights emerge

Boser says that research shows we’re more committed, if we develop a learning plan with clear objectives, and that periodic reflection on the skills and concepts we’re trying to master, i.e., utilizing metacognition, makes each of us a better learner.

You can read more about strategies for learning in Boser’s article and his book
June 14, 2018
14 Jun 2018

The Tyranny of Metrics

In a recent article “Against Metrics: How Measuring Performance by Numbers Backfires,” Jerry Z Muller argues that companies, educational institutions, government agencies, and philanthropies are now in the grip of what he calls “metric fixation,” “…the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardized data (metrics).”

In this brief and important article, Muller critiques the growing phenomenon of paying employees for performance. He points out that such schemes often lead to a narrowing measure of what is desirable for the organization, leads members of an organization to “game the system”, often undermines organizations ability to think more broadly about their purposes, and most importantly, impedes innovation.

Looking at the unintended outcomes of metric fixation, he writes:

“When reward is tied to measured performance, metric fixation invites just this sort of gaming. But metric fixation also leads to a variety of more subtle unintended negative consequences. These include goal displacement, which comes in many varieties: when performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a pay rise or raising the stock price at the time that stock options are vested), people focus on satisfying those measures – often at the expense of other, more important organizational goals that are not measured. The best-known example is ‘teaching to the test’, a widespread phenomenon that has distorted primary and secondary education in the United States since the adoption of the No Child Left Behind Act of 2001.”

Pay for performance schemes, however, are not alone in eliciting a narrowing of goals, and a tendency to game the system. Metric fixation (or what I term the “tyranny of measurement”) can be a risk for a range of non-profit organizations and educational institutions who often feel that demands for accountability can be addressed by merely counting the number of participants who receive services, or the number of students who score well on reading tests. While it is important to have clear goals, and to be able to indicate if these goals are met, organizations, in their rush to address demands from funders and other stakeholders for accountability, must be careful not to reduce their goals—indeed their organizations’ vision— to only a few countable variables. “What can and does get measured is not always worth measuring, may not be what we really want to know, and may draw effort away from the things we care about” (Muller). As Albert Einstein observed, “Not everything that counts can be counted, and not everything that can be counted, counts.”


“Against Metrics: How Measuring Performance by Numbers Backfires,” Aeon, April 24, 2018

The Tyranny of MetricsJerry Z. Muller, Princeton University Press, 2018

May 8, 2018
08 May 2018

Transparent as a Jellyfish? Why Privacy is Important

Recent revelations about Facebook and Cambridge Analytica’s use of personal data have raised serious concerns about internet privacy. It would appear that we inhabit a world in which privacy is increasingly under assault—not just from leering governments, but also from panoptic corporations.

Although the right to privacy in the US is not explicitly protected by the Constitution, constitutional amendments and case law have provided some protections to what has become a foundational assumption of American citizens. The “right to privacy” (what Supreme Court Justice Louis Brandis once called “the right to be left alone,”) is a widely held value, both in the U.S. and throughout the world. But why is privacy important?

In “Ten Reasons Why Privacy Matters” Danial Solove, Professor of Law at George Washington University Law School, lists ten important reasons including: limiting the power of government and corporations over individuals; the need to establish important social boundaries; creating trust; and as a precondition for freedom of speech and thought. Solove also notes, “Privacy enables people to manage their reputations. How we are judged by others affects our opportunities, friendships, and overall well-being.”

Julie E. Cohen, of Georgetown argues that privacy is not just a protection, but an irreducible environment in which individuals are free to develop who they are and who they will be. “Privacy is shorthand for breathing room to engage in the process of … self-development. What Cohen means is that since life and contexts are always changing, privacy cannot be reductively conceived as one specific type of thing. It is better understood as an important buffer that gives us space to develop an identity that is somewhat separate from the surveillance, judgment, and values of our society and culture.” (See “Why Does Privacy Matter? One Scholar’s Answer” Jathan Sadowski, The Atlantic, Feb 26) In the Harvard Law Review, (“What Privacy Is For” Julie E. Cohen, Harvard Law Review, Vol. 126, 2013) Cohen writes, “Privacy shelters dynamic, emergent subjectivity from the efforts of commercial and government actors to render individuals and communities fixed, transparent, and predictable. It protects the situated practices of boundary management through which self-definition and the capacity for self-reflection develop.”

Cohen’s argument that privacy is a pre-condition for the development of an autonomous and thriving self is a critical and often overlooked point. If individuals are to develop, individuate, and thrive, they need room to do so, without interference or unwanted surveillance. Such conditions are also necessary for the maintenance of individual freedom vs. slavery.  As Orlando Patterson argued in his book,  Freedom Vol.1 Freedom in the Making of Western Culture (Basic Books, 1991) freedom historically developed in the West as a long struggle against chattel slavery.  Slavery, of course, entails the subjugation of the individual/person, and depends upon the thwarting of autonomy. While slavery may not fully eradicate the full and healthy development of the “self,” it may deform and distort that development. Autonomous selves are both the product of and the condition of social freedom.

Privacy, which is crucial to the development of a person’s autonomy and subjectivity, when reduced by surveillance or restrictive interference—either by governments or corporations who gather and sell our private information—may interfere not just with social and political freedom, but with the development and sustenance of the self.  “Transparency” (especially when applied to personal information) may seem like an important feature to those who gather “Big Data,” but it may also represent an intrusion and an attempt to whittle away the environment of privacy that the self depends upon for its full and healthy development. As Cohen observes, “Efforts to repackage pervasive surveillance as innovation — under the moniker “Big Data” — are better understood as efforts to enshrine the methods and values of the modulated society at the heart of our system of knowledge production. In short, privacy incursions harm individuals, but not only individuals. Privacy incursions in the name of progress, innovation, and ordered liberty jeopardize the continuing vitality of the political and intellectual culture that we say we value.” (See “What Privacy Is For” Julie E. Cohen, Harvard Law Review, Vol. 126, 2013)

Privacy is not just important to the protection of individuals from governments and commercial interests, it is also essential for the development of full, autonomous, and healthy selves.


“Ten Reasons Why Privacy Matters” Daniel Solove

“Why Does Privacy Matter? One Scholar’s Answer” Jathan Sadowski, The Atlantic, Feb 26, 2013

“What Privacy Is For” Julie E. Cohen, Harvard Law Review, Vol. 126, 2013

Orlando Patterson argued in his book,  Freedom Vol.1 Freedom in the Making of Western Culture (Basic Books, 1991)

“Facebook and Cambridge Analytica, What You Need to Know as Fallout Widens” Kevin Granville, New York Times, Mar 19, 2018

“I Downloaded the Information that Facebook Has on Me. Yikes” Brian Chen, New York Times, Apr 11, 2018

“Right to Privacy: Constitutional Rights & Privacy Laws” Tim Sharp, Livescience, June 12, 2013

Surveillance Capitalism, Shoshone Zuboff
Listen to “Facebook and the Reign of Surveillance Capitalism” Radio Open Source
Read a review of Surveillance Capitalism

“How to Save Your Privacy from the Internet’s Clutches” Natasha Lomas, Romain Dillet, TC, Apr 14, 2018

April 25, 2018
25 Apr 2018

Everybody Lies

Although the author of Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are, (Seth Stephens-Davidowitz, Dey St., 2017,) hesitates to specify precisely what ‘big data’ is, he is confident that we are living through an era in which there is an explosion in the amount and quality of data—especially internet-imbedded data—that can tell us things about humans that previous data sources and data analysis methods have been unable to reveal. In fact, Stephens-Davidowitz argues in this quick and easy to read book that, “I am now convinced that Google searches are the most important data set ever collected on the human psyche…and I am convinced that new data increasingly available in our digital age will expand our understanding of humankind.”

Stephens-Davidowitz argues that unlike previous, predominately survey-based data, the emergence of big data—primarily data provided by Google and other on-line searches, makes insights into humans’ deepest interests, desires, behaviors, and values much more transparent and accessible.  Whereas traditional survey research has a number of vulnerabilities (e.g., people are not candid, and in fact lie, they provide socially desirable answers, they both exaggerate or underestimate behaviors and characteristics, etc.) analysis of internet data together with the use of new analytical tools (e.g., Google trends), now makes available immensely more accurate information about what people actually think, believe, and fear.

Stephens-Davidowitz illustrates the insights that collection and analysis of internet-based data now make possible. He shows, for example, how analysis of Google searches about race revealed voters’ real (vs. survey-reported) attitudes toward race even in otherwise seemingly liberal precincts. These attitudes—largely hidden from analysts who used traditional kinds of survey methods, made possible the surprising election of a figure like Donald Trump, who mobilized anti-immigrant sentiment and racist allusions to win the 2016 presidential election. “Surveys and conventional wisdom placed modern racism predominantly in the South and mostly among Republicans.  But the places with the highest racist search rates included upstate New York, western Pennsylvania, eastern Ohio, and rural Illinois…” (p. 7) “The Google searches revealed a darkness and hatred among a meaningful number of Americans that pundits for many years missed. Search data revealed that we live in a very different society from the one academics and journalists, relying on polls, thought that we live in. It revealed a nasty, scary, and wider-spread rage that was waiting for a candidate to give voice to.” (p. 12)

Everybody Lies…examines the ways that new methods of analyzing internet data can yield accurate insights about what people are really concerned with and thinking about. In a chapter titled “Digital Truth Serum,” the author surveys a number of topics including gender bias and sexism, America’s Nazi sympathizers, and the underreported rise of child abuse during economic recessions. He repeatedly demonstrates how internet data reveal accurate and often counter-intuitive findings. In one instance, the author shows that, in fact, internet data reveal that we are more likely to interact with someone with opposing political ideas on the internet than in real life, and that in many instances—and counter to what is widely believed—liberals and conservatives often visit and draw upon the same news websites. (It appears that fascist sympathizers and liberals both rely on

Everybody Lies…makes an argument for a ‘revolution’ in social science research. Stephens-Davidowitz believes that the collection and careful analysis of internet-based data promises a much more rigorous and penetrating approach to answering questions about peoples’ genuine attitudes, behaviors, and political dispositions. Along the way to demonstrating the superiority of such research, Stephens-Davidowitz touches on some important, but taboo and previously difficult-to-answer questions: What percentage of American males are gay? Is Freud’s theory of sexual symbols in dreams really accurate? At what age are political attitudes first established?  How racist are most Americans?

Stephens-Davidowitz writes, “Frankly, the overwhelming majority of academics have ignored the data explosion caused by the digital age. The world’s most famous sex researches stick with the tried and true. They ask a few hundred subjects about their desires; they don’t ask sites like PornHub for their data. The world’s most famous linguists analyze individual tests; they largely ignore the patterns revealed in billions of books. The methodologies taught to graduate students in psychology, political science, and sociology have been, for the most part, untouched by the digital revolution. The broad, mostly unexplored terrain opened by the data explosion has been left to a small number of forward-thinking professors, rebellious grad students and hobbyists. That will change.” (p. 274)


Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are, Seth Stephens-Davidowitz, Dey St., 2017

March 27, 2018
27 Mar 2018

Two Cheers for Bureaucracy in Education

Rationalization and Bureaucracy

In the early years of the 20th century, the German sociologist Max Weber argued that modern society increasingly relies upon the “rationalization” of social organizations and institutions. He maintained that Western society is increasingly reliant upon reason, efficiency, predictability, and means/ends calculation. He further believed that modern society is highly dependent upon both public and private bureaucracies (e.g., the nation state and the modern corporation) as a way to achieve important societal goals (education, social welfare, medical care, business administration, governance, etc.) Bureaucracies are, “Highly organized networks of hierarchy and command structures (which are) necessary to run any ordered society – especially ones large in scope.” (See “Max Weber’s Theory of Rationalization: What it Can Tell Us of Modernity,” ) As one form of social organization, bureaucracy is distinguished by its: (1) clear hierarchy of authority, (2) rigid division of labor, (3) written and inflexible rules, regulations, and procedures, and (4) impersonal relationships. (For this and additional definitions, see the BusinessDictionary)

Weber and subsequent social theorists saw the process of rationalization and bureaucratization as replacing traditional modes of life, traditional values, and religious orientations with a society characterized by growing calculability, pursuit of individuals’ self-interest, efficiency, and ordered control. (Weber termed the loss of tradition that accompanied the increasing rationalization of Western society as the “disenchantment” of society.) As modernity transforms traditional social forms and social values, rationality and bureaucracy come to dominate the various spheres of contemporary society. Moreover, as society becomes ever more rationalized, it increasingly depends upon bureaucratic regimes of governance and management by impersonal rules and the exercise of technical knowledge by experts. Today, various social arenas—ranging from government to corporate organizations, from healthcare to public education—have become suffused with the ethos of bureaucracy and rationality. “…rationalization means a historical drive towards a world in which ‘one can, in principle, master all things by calculation.’” (See, Max Weber, in Stanford Encyclopedia of Philosophy)

Advantages and Disadvantages of Bureaucracy

Although “bureaucracy’ is often thought of as a pejorative term, bureaucracy has some advantages over other forms of social organization.  Bureaucracy creates and utilizes rules and laws (vs. fiat decisions by a powerful notable, such as a king), mobilizes the knowledge of educated experts, promotes meritocracy, delineates and sets boundaries for the exercise of social power, establishes a formal chain of command and specifies organizational authority, and provides a technically efficient form of organization for dealing with routine matters that concern large numbers of persons. These advantages however, are accompanied by what are thought by many to be substantial disadvantages, including, compelling officials to conform to fixed rules and detailed procedures, sponsoring bureaucrats’ focus on narrow objectives, and supporting bureaucrats to become defensive, rigid, and unresponsive to the urgent individual needs and concerns of private citizens. “…individual officials working under bureaucratic incentive systems frequently find it to be in their own best interests to adhere rigidly to internal rules and formalities in a ritualistic fashion, behaving as if “proper procedure” were more important than the larger goals for serving their clients or the general public that they are supposedly designed to accomplish (i.e., the “red tape” phenomenon).” (See Bureaucracy)

Education and Bureaucracy 

If we look at public education in contemporary society, we see many features associated with bureaucracy. State education agencies, districts, and schools: 1) are run by trained experts (e.g., credentialed teachers and administrators), 2) feature rigid hierarchies of authority, 3) have a strict division of labor, 4) depend upon and are run by formal and impersonal rules of administration and control, and 5) credential students by relying on impersonal and standardized methods for assessing student achievement. Additionally, students are routinely segregated into age-specified categories (classes) and are subjected not to individually tailored curricula, but to routine and standardized curricula that attempt to teach students en masse.

Does Standardized Testing Support Educational Bureaucracy?

Standardized testing of student achievement is one of the bureaucratic characteristics of modern public education. Although assessment is thought to be a necessary means for measuring student learning, it is also a means by which educational organizations categorize students, assign them social statues, and allocate them to various social trajectories (“life chances” to use Weber’s terminology). Standardized testing regimes also assist the educational bureaucracy by creating different categories of clientele (i.e., students) who can then be served en masse by large-scale routinized educational programs and mass-produced textbooks. Some would even argue that students are made to fit school as much as schools are made to fit the student. (For a summary of the problems associated with standardized testing, see “What’s Wrong with Standardized Tests.”)

While students are subject to the rule of bureaucracy, so too are faculty and administrators. Like the students they teach and oversee, faculty and administrators are subject to formal structures of authority, adhere to a strict division of labor, follow formal rules and regulations, and must be credentialed and certified. Like their students, teachers are also subject to assessment and review. (See our previous blog post “Too Much Assessment in Higher Education,” for an example of the effects of assessment on higher education.) Schools are also reviewed and rated by State Departments of Education.

While some feel that the stultifying aspects of bureaucracy may be ameliorated, the original theorist of rationalization and bureaucracy, Max Weber, was pessimistic about the reform of bureaucracy. As he surveyed the early 20th century and considered the likely developmental direction of Western society, he said that citizens of society were likely to find themselves increasingly entrapped in what he termed the “iron cage of bondage,” which continued to be cemented by the growth of rationalization and bureaucracy. Whether this dark prognosis is generally true for Western society is still very much debatable. That said, it is difficult to imagine large-scale public education without many of the features of bureaucracy that Weber first described —including standardized student testing. (For examples of reform efforts as they apply to standardized tests, see The National Center for Fair & Open Testing)

March 6, 2018
06 Mar 2018

Too Much Assessment in Higher Education?

Too Much Assessment in Higher Education?In an important article, “The Misguided Drive to Measure ‘Learning Outcomes’”, New York Times, Feb 23, 2018, Molly Worthen argues that the growth of, and seeming obsession with, the assessment of learning outcomes in higher education has profoundly shaped curricula and instruction, undercut the unique capacities of colleges and universities to foster genuine critical thinking, and proven to be both bureaucracy bloating and extremely expensive. Worthen shows that, driven largely by the interests of accrediting agencies and for-profit tech and consulting companies, higher education’s rush to demonstrate student learning and skill-acquisition disproportionately affects non-elite schools, and often compels these under-resourced institutions to devote scarce dollars to obtaining evidence of instructional impact, “…more and more university administrators want campus-wide, quantifiable data that reveal what skills students are learning. Their desire has fed a bureaucratic behemoth known as learning outcomes assessment.”

In order to show that students graduate with job-ready skills and attitudes, Worthen argues that higher education institutions’ focus on assessment obscures the “real crisis” in higher education: “the system’s deepening divide into a narrow tier of elite institutions primarily serving the rich and a vast landscape of glorified trade schools for everyone else.” She notes the cruel irony in the mania for assessment is that there is little evidence beyond occasional anecdotes, that regimes of assessment actually improve student learning. Moreover, more selective (i.e., elite) institutions, themselves, don’t utilize assessment of learning outcomes at the same rate as less prestigious institutions. “Research indicates that the more selective a university, the less likely it is to embrace assessment.”

Perhaps the greatest irony, Worthen writes, is that assessment regimes subvert the unique purposes and capacities of higher education. “The value of universities to a capitalist society depends on their ability to resist capitalism, to carve out space for intellectual endeavors that don’t have obvious metrics or market value.” She further observes, “Producing thoughtful, talented graduates is not a matter of focusing on market-ready skills. It’s about giving students an opportunity that most of them will never have again in their lives: the chance for serious exploration of complicated intellectual problems, the gift of time in an institution where curiosity and discovery are the source of meaning.”


The Misguided Drive to Measure ‘Learning Outcomes’”, Molly Worthen, New York Times, 23 February 2018

January 3, 2018
03 Jan 2018

Evaluation in Complex and Evolving Environments

Program evaluations seldom occur in stable, scientifically controlled environments. Often programs are implemented in complex and rapidly evolving settings that make traditional evaluation research approaches—which depend upon the stability of the “treatment” and the designation of predetermined outcomes—difficult to utilize.

Michael Quinn Patton, one of the originators of Developmental Evaluation, says that “Developmental evaluation processes include asking evaluative questions and gathering information to provide feedback and support developmental decision-making and course corrections along the emergent path. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation, and intentional change. The evaluator’s primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and to facilitate data-based assessments and decision-making in the unfolding and developmental processes of innovation.”

In their paper, “A Practitioners Guide to Developmental Evaluation,” Dozios and her colleagues note, “Developmental Evaluation differs from traditional forms of evaluation in several key ways:”

  • The primary focus is on adaptive learning rather than accountability to an external authority.
  • The purpose is to provide real-time feedback and generate learnings to inform development.
  • The evaluator is embedded in the initiative as a member of the team.
  • The DE role extends well beyond data collection and analysis; the evaluator actively intervenes to shape the course of development, helping to inform decision-making and facilitate learning.
  • The evaluation is designed to capture system dynamics and surface innovative strategies and ideas.
  • The approach is flexible, with new measures and monitoring mechanisms evolving as understanding of the situation deepens and the initiative’s goals emerge

Development evaluation is especially useful for social innovators who often find themselves inventing the program as it is implemented, and who often don’t have a stable and unchanging set of anticipated outcomes. Following Patton, Dozois, Langlois, and Blanchet-Cohen observe that Developmental Evaluation is especially well suited to situations that are:

  • Highly emergent and volatile (e.g., the environment is always changing)
  • Difficult to plan or predict because the variables are interdependent and non-linear
  • Socially complex— requiring collaboration among stakeholders from different organizations, systems, and/or sectors
  • Innovative, requiring real-time learning and development

Developmental Evaluation, however, is increasingly appropriate for use in the non-profit world, especially where the stability of programs’ key components including the program’s core treatment and eventual, often evolving, outcomes, are not as certain or firm as program designers might wish.

Brad Rose Consulting is experienced in working with program’s whose environments are volatile and whose iterative program designs are necessarily flexible. We are adept at collecting data that can inform the on-going evolution of a program, and have 20+ years of providing meaningful data to program designers and implementers that help them to adjust to rapidly changing and highly variable environments.


A Practitioner’s Guide to Developmental Evaluation, Elizabeth Dozois, Marc Langlois, and Natasha Blanchet-Cohen

Michael Quinn Patton on Developmental Evaluation

Developmental Evaluation

The Case for Developmental Evaluation

November 8, 2017
08 Nov 2017

Dark Factories

We’ve previously written about the rise of Artificial Intelligence (AI) and robotics, and the impact of these new technologies on employment and the future of work.
(See our previous blog posts: “Humans Need Not Apply: What Happens When There’s No More Work?” and “Will President Trump’s Wall Keep Out the Robots?”) Today we’d like to refer readers to an important article, “Dark Factory”, that recently appeared in The New Yorker. “Dark Factory” explores the growing impact of robotics and AI on the manufacturing and service sectors of the U.S. Economy.

Dark Factory

In “Dark Factory”, Sheelah Kolhatkar discusses her visit to Steelcase, the manufacturer of office furniture. Steelcase, like much of American manufacturing, has had its economic ups and downs over the years. Kolhatkar describes how, in recent years, the company has increasingly employed robotics as a means to improve manufacturing efficiency, and as a result, now relies on fewer workers than it has in the past. Representative of an ever-growing number of manufacturing companies in the U.S., Steelcase employs fewer and fewer high school graduates, and now seeks college educated employees with technological skills, so that these higher skilled workers can supervise an expanding army of manufacturing robots.

As Kolhatkar shows, while efficiency gains are good for Steelcase and other manufacturing companies that employ robots and AI— and even, in some cases, make work more tolerable and less grueling for the remaining employees on the shop floor— the net effect of these technologies is to displace large swaths of the work force and to shift wealth to the owners of companies. Kolhatkar cites research that shows that the use of industrial robots, like those at Steelcase, is directly related to decline in both the number of manufacturing jobs and declining pay for remaining workers.

Kolhatkar also discusses “dark factories”— factories and warehouses whose use of robots and AI are so extensive that they need not turn on the lights because there are so few human workers. While such factories and warehouses are not yet wide-spread, major U.S. corporations are looking to use robotics and AI to run nearly employee-less operations. Although some companies may not be eager to begin utilizing robotic warehouses, competitive pressure is sure to compel U.S. companies to implement fully robotized facilities, or lose competitive battles with other nations that do adopt these technologies.

AI and Robotics Not Limited to Manufacturing

The result of the growing use of robotics and AI in the U.S. is, of course, the declining demand for workers in what where once fairly labor-intensive human-dominated work environments. (Manufacturing now employees only about 10% of the US workforce, and these jobs are under constant threat by new technologies.) Although displaced manufacturing workers often seek jobs in the service sector, this sector is now hardly immune to automation. MacDonald’s, for example, is introducing “digital ordering kiosks” where customers electronically enter their orders and pay for their meals. MacDonald’s is expected to automate in 5500 restaurants by the end of 2018. Uber and Google continue to invest in the development of autonomous driving technologies, and the U.S. trucking industry is eager to adapt autonomous vehicles so that it can reduce substantial labor costs associated with trucking. Amazon has purchased Kiva, a robotics company, and is developing robots that can zoom around Amazon warehouses and fulfill orders. (A Deutsch Bank report estimates that Amazon could save 22 million dollars a year in each of its warehouses, simply by introducing warehouse robots to replace human workers.)

As Kolhatkar’s “Dark Factory” shows, while the future looks increasingly promising for the shareholders of companies who introduce labor-displacing robotics and AI, it doesn’t appear quite so sunny for those humans who must work for a living—especially in the manufacturing and service sectors of the U.S. economy. Like the “dark factories” that promise to displace them, for many workers, the future too, will be dark.


“Welcoming our New Robotic Overlords”, Sheelah Kolhatkar, The New Yorker, October 23 2017

“AI, Robotics, and the Future of Jobs”, Pew Research Center

“Artificial intelligence and employment”, Global Business Outlook

“Advances in Artificial Intelligence Could Lead to Mass Unemployment, Experts Warn”, James Vincent, The Independent, Wednesday 29 January 2014

September 11, 2017
11 Sep 2017

The Importance of Understanding Stakeholders

Every program evaluation is conducted in a context in which there are parties (persons, organizations, etc.) who have an interest, or a “stake,” in the operation and success of the program. In the corporate world, a “stakeholder” is any member of the “groups without whose support the organization would cease to exist.” (see Corporate Stakeholder). More recently, the idea of “a stakeholder” has been broadened to include “any group or individual who is affected by, or who can affect the achievement of, an organization’s objectives.” (The Stakeholder Theory Summary.)
Indeed, in the not-for-profit world, stakeholders may include an array of persons and organizations including funders, community members, program participants, family members, volunteers, staff, government agencies, and the broader public.

Stakeholders in non-profits usually fall into one of three categories of legal statuses:

  1. Constitutional stakeholders such as board members or trustees of the non-profit organization
  2. Contractual stakeholders, including paid staff, or any business, group or individual that has a formal relationship with the organization.
  3. Third-party stakeholders including all the people and groups that may be affected by what the organization does. That includes businesses, the local government, and the citizens who live in the community. (See What is a Stakeholder of a Non-profit.)

Nonprofit stakeholders may range from those who support an organization, to those who oppose an organization. Stakeholder can include advocates, supporters, critics, competitors, and opponents. In its analysis of stakeholders in policy change efforts, the World Bank uses the categories of “promoter,” “defender,” “latents”, and “apathetics.” (See What is a Stakeholder Analysis.)

Conducting a stakeholder analysis is very useful for both evaluation and strategic planning efforts. Identifying various stakeholders’ interests in an organization’s mission and programming can help non-profit leaders and staff to be sure that their efforts and initiatives are achieving desired goals. They can also be useful in ensuring that the needs of those served are being directly met. For both evaluation and strategic planning purposes, a stakeholder analysis is an important process for achieving a shared understanding of each stakeholder’s specific interest in, and relevance to, the work of the non-profit or educational organization.

Brad Rose Consulting has developed a unique approach to stakeholder analysis, one that can be extremely useful as organizations examine the purposes, goals, specific activities, and desired outcomes of their work. We often work with organizations to implement stakeholder analyses. These analyses are helpful both in identifying where an organization is, at any given point in time, and for identifying where it wants to go in the future. You can see our basic stakeholder analysis form here.


Corporate Stakeholder

Stakeholder Theory

Summary of Stakeholder Theory

Stakeholders of a Typical Non-Profit Organization

What is a Stakeholder of a Non-profit

February 17, 2017
17 Feb 2017

A Brief Guide to Evaluation (Part 3): The Uses of Evaluation Findings

a brief guide to evaluationOne of the most important questions to consider before starting an evaluation is, “How will evaluation findings be used?” There are a number of ways the findings from an evaluation can be used. These include:

  • to improve the program (formative evaluation)
  • to make judgements about the ultimate value/benefits of the program to participants and stakeholders (summative evaluation)
  • to sustain and/or expand the program
  • to document and publicize the program’s achievements (outreach and marketing)
  • to curtail the program

By being clear about the intended uses of evaluation findings, organizations can choose the most effective and stakeholder-relevant evaluation design. If, for example, a program funder wants to know whether their investment yielded desired outcomes for program participants, it may not be useful to focus the evaluation on gathering data that primarily indicate ways to improve the program’s delivery. On the other hand, if an organization is curious about how its programming is being implemented, and if such implementation is in fidelity with a research-based program design, the evaluation will want to focus more on implementation processes, rather than ultimate program outcomes.

Needless to say, stipulating the use of evaluation findings is closely related to the question of who are the major audiences for the evaluation. See our previous post, Identifying Evaluation Stakeholders. By clarifying potential audiences for findings, organizations and evaluators can identify the most useful evaluation strategy and design. In the next blog post we will discuss a related question, i.e., “What information will stakeholders find useful and valuable?” And to learn more about our data collection and evaluation click here.


Understanding Different Types of Evaluations

Program Evaluation Essentials for Non-Evaluators 

February 6, 2017
06 Feb 2017

A Brief Guide to Evaluation (Part 2): Identifying Evaluation Stakeholders

When you and your colleagues consider conducting an evaluation, it is critical that you be clear about who the evaluation is for. Your organization is likely to have a number of stakeholders who are interested in learning about whether the programmatic work is achieving its desired results (i.e. outcomes). Some programs are funded by a single funder, others may have multiple funders. Funders are, of course, a very important audience for evaluation findings. They will want to know whether their investment is producing results. However, you will want to consider additional stakeholders as well (see the list below). When you are considering an evaluation—whether it will be conducted by an internal or external evaluator—you will benefit from considering who will be the audience for evaluation findings?
  • Who are the major stakeholders or interested parties for evaluation results?
  • Who wants/needs to know?
  • Who will benefit from knowing about the effectiveness of your program’s efforts?
  • Make a list of key stakeholders and what they might want to know.
Remember, there can be, and usually are, a variety of potential stakeholders for evaluation findings—both external and internal.
  • Current funders
  • Potential funders
  • Community stakeholders
  • Advocacy organizations
  • Government agencies
  • Colleagues/Internal organizational constituencies
  • Peer organizations
  • Program managers and program staff
  • The “field”/general public
  • Political allies
  • Voters
  • Competitors
By being clear about the audiences for evaluation findings, you can better define the kind of evaluation you will want to do. To learn about our evaluation techniques visit our Data collection & Outcome measurement page.
January 24, 2017
24 Jan 2017

A Brief Guide to Evaluation (Part 1)

This is the first in a series of blog posts, that outlines the key elements of an evaluation.

There are many things in the world that can be evaluated. You will want to decide and define the specific object (i.e. the “evaluand”) of the evaluation. One useful approach is for program managers, and program implementers to speak with the evaluator, and together, consider these defining questions about what needs to be evaluated:

Whether you are managing a program, funding an initiative, or about to undertake a policy review, you will want to first consider:

What is the “it” that’s being evaluated?

  • program
  • initiative
  • organization
  • set of processes or relationships
  • campaign
  • services
  • activities
  • product

Once you’ve identified what is being evaluated, it will be useful to consider these additional questions:

  • What’s being done? What are the intentional actions that are being undertaken?
  • Who does what to whom, and when do they do it?
  • Why—for what set of reasons—are these things being done? Describe the need for the initiative or program?
  • Who (which people, and which positions) are doing/carrying out the of the program, initiative, policy?
  • What resources (i.e. inputs) are involved? (not just money and time, but knowledge, cooperation of others, networks of collaborators, etc.)
  • Who (or what) are implementers working to change? What specifically is supposed to change, or be different as a result of the program or initiative doing what it does?
  • Describe who benefits (e.g. program participants, beneficiaries, consumers, etc.) and what the benefits are.

By clearly identifying a determinate entity (i.e., “evaluand”) and the effects (i.e., outcomes) of the operation or implementation of that entity, you will help to ensure a successful evaluation. The clearer you can be in describing what the program, initiative, or policy is, the more effective the evaluation will be. Answering the above questions and sketching out a logic model of the program’s operation (which we’ll discuss in a subsequent blog post), will ensure an effective, accurate, and ultimately highly useful evaluation.

Also, for purposes of clarity, it may be useful for program managers and implementers to consider and discuss with the evaluator what isn’t being evaluated. This conversation can help to define the necessary boundaries around the program and will help to prevent evaluators expending unnecessary and potentially costly efforts to design evaluations of things that exist outside the boundary of the initiative, program or policy. To learn more about our evaluation practices visit our Impact & Assessment page.

April 28, 2016
28 Apr 2016

Evidence-Based Studies

“Evidence-based” – What is it?
“Evidenced-based” has become a common adjectival term for identifying and endorsing the effectiveness of various programs and practices in fields ranging from medicine to education, from psychology to nursing, from criminal justice to psychology. The motivation for marshalling objective evidence in order to guide practices and policies in these diverse fields has been the result of the growing recognition that professional practices—whether they be doctoring or teaching, social work, or nursing—need to be based on something more sound than custom/tradition, practitioners’ habit, professional culture, received wisdom, and hearsay.

What does “evidenced-based” mean?
While definitions of “evidence-based” vary, the most common characteristics of evidence-based research include: objective, empirical research that is valid and replicable, whose findings are based on a strong theoretical foundation, and include high quality data and data collection procedures. The most common definition of Evidence-Based Practice (EBP) is drawn from Dr. David Sackett’s, original (1996) definition of “evidence-based” practices in medicine, i.e., “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research” (Sackett D, 1996). This definition was subsequently amended to, “a systematic approach to clinical problem solving which allows the integration of the best available research evidence with clinical expertise and patient values” (Sackett DL, Strauss SE, Richardson WS, et al. Evidence-based medicine: how to practice and teach EBM. London: Churchill-Livingstone,2000). (See, “Definition of Evidenced-based Medicine”).

An evidenced-based program, whether it be in youth development or education, is comprised of a set of coordinated services/activities that demonstrate effectiveness, as such effectiveness has been established by sound research, preferably, scientifically based research. (See, “Introduction to Evidence-Based Practice“).

In education, evidence-based practices are those practices that are based on sound research that shows that desired outcomes follow from the employment of such practices.  “Evidence-based education is a paradigm by which education stakeholders use empirical evidence to make informed decisions about education interventions (policies, practices, and programs). ‘Evidence-based’ decision making is emphasized over ‘opinion-based’ decision making.” Additionally, “the concept behind evidence-based approaches is that education interventions should be evaluated to prove whether they work, and the results should be fed back to influence practice. Research is connected to day-to-day practice, and individualistic and personal approaches give way to testing and scientific rigor.” (See, “What is Evidence-Based Education?“).

Of course there are different kinds of evidence that can be used to show that practices, programs, and policies are effective.  In a subsequent blog post I will discuss the range of evidence-based studies—from individual case studies and quasi experimental designs, to randomized controlled trials (RCT).  The quality of the evidence as well as the quality of the study in which such evidence appears is a critical factor in deciding whether the practice or program is not just “evidence-based,” but in fact, effective. To learn more about our data collection and measurement click here.


Evidenced-based practice

What are Evidence-Based Interventions (EBI)?

Scientific Research and Evidence-Based Practice

Evidence-based medicine, Florida State College of Medicine

Evidenced-based studies in education

U.S Department of Education, “What Works” Clearinghouse

Defining Evidence-Based Programs

Child and Family Services

Linking Research with Practice in Youth Development – What Works and How Do We Know

Scientifically Based Research vs. Evidence-Based Practices and Instruction

How to Evaluate Evidence-Based or Research-Based Interventions

Issues in Defining and Applying Evidence-Based Practices Criteria for Treatment of Criminal-Justice Involved Clients

Can Randomized Trials Answer the Question of What Works?

February 1, 2016
01 Feb 2016

Using Qualitative Interviews in Program Evaluations

brad rose consultingQualitative interviews can be an important source of program evaluation data. Both in-depth individual interviews and focus group interviews are important methods that provide insights and phenomenologically rich descriptive information that other, numerically oriented data collection methods (e.g., questionnaires and surveys), are often unable to capture. Typically, interviews are either structured, semi-structured, or unstructured. (Structured interviews are, essentially, verbally administered questionnaires, in which a list of predetermined questions is asked. Semi-structured interviews consist of several key questions that help to define the areas to be explored, but allow the interviewer to diverge from the questions in order to pursue or follow up to an idea or response. Unstructured interviews start with an opening question, but don’t employ a detailed interview protocol, favoring instead the interviewer’s spontaneous generation of subsequent follow-up questions. See P. Gill1, K. Stewart2, E. Treasure3 & B. Chadwick, cited below.)

Interviews, especially one-on-one, in-person, interviews, allow an evaluator to participate in direct conversations with program participants, program staff, community members, and other program stakeholders. These conversations enable the evaluator to learn about interviewees’ experiences, perspectives, attitudes, motivations, beliefs, personal history, and knowledge. Interviews can be the source of pertinent information that is often unavailable to other methodological approaches. Qualitative interviews enable evaluators to: elicit direct responses from interviewees, probe and ask follow-up questions, gather rich and detailed descriptive data, offer real-time opportunities to explain and clarify interview questions, observe the affective responses of interviewees, and ultimately, to conduct thorough-going explorations of topics and issues critical to the evaluation initiative.

Although less intimate, focus group interviews are another key source of evaluation data. Organized around a set of guiding questions, focus groups typically are composed of 6-10 people and a moderator who poses open-ended questions to focus group participants. Focus groups usually include people who are somewhat similar in characteristics or social roles. Participants are selected for their knowledge, reflectiveness, and willingness to engage topics or questions. Ideally – although not always possible – it is best to involve participants who don’t previously know one another. Focus groups can be especially useful in clarifying, qualifying, and/or challenging data collected through other methods. Consequently they can be useful as a tool for corroborating findings from other research methodologies.

Regardless of the specific form of qualitative interview—individual or focus group, in-person or telephone—qualitative interviews can be useful in providing data about participants’ experience in, and ultimately the effectiveness of, programs and initiatives. To find out more about our use of qualitative data visit our Data collection & Outcome measurement page.


Learning from Strangers: The Art and Method of Qualitative Interview Studies, Robert S. Weiss, Free Press.

Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences, Irving Seidman. Fourth Edition.

Methods of data collection in qualitative research: interviews and focus groups. P. Gill1, K. Stewart2, E. Treasure3 & B. Chadwick.

Advantages and Disadvantages of Four Interview Techniques in Qualitative Research,  Raymond Opdenakker.

Qualitative Research Methods Overview

Interviewing—The Robert Wood Johnson Foundation

Pros & Cons of Interviewing

Advantages and Disadvantages of Face-to-Face Data Collection

Copyright © 2020 - Brad Rose Consulting