Program evaluation services that increase program value for grantors, grantees, and communities. x

Archive for category: Organization Development

February 4, 2020
04 Feb 2020

Bias, Seeing Things as We Are, Not as They Are

“Bias” is a tendency (either known or unknown) to prefer one thing over another that prevents objectivity, that influences understanding or outcomes in some way. (See the Open Education Sociology Dictionary) Bias is an important phenomenon in social science and in our everyday lives.

In her article, “9 types of research bias and how to avoid them,” Rebecca Sarniak discusses the core kinds of bias in social research. These include both the biases of the researcher, and the biases of the research subject/respondent.

Prevalent kinds of researcher bias include:

  • confirmation bias
  • culture bias
  • question-order bias
  • leading questions/wording bias
  • the halo effect

Respondent biases include:

  • acquiescence bias
  • social desirability bias
  • habituation
  • sponsor bias

In their Scientific American article “How-to-think-about-implicit-bias,”, Keith Payne, Laura Niemi, John M. Doris assure us that bias is not merely rooted in prejudice, but in our tendency to notice patterns and make generalizations. “When is the last time a stereotype popped into your mind? If you are like most people, the authors included, it happens all the time. That doesn’t make you a racist, sexist, or whatever-ist. It just means your brain is working properly, noticing patterns, and making generalizations…. This tendency for stereotype-confirming thoughts to pass spontaneously through our minds is what psychologists call implicit bias. It sets people up to overgeneralize, sometimes leading to discrimination even when people feel they are being fair.”

Of course, bias is not just a phenomenon relevant to social science (and evaluation) research. It affects our everyday activities too. In “10 Cognitive Biases That Distort Your Thinking,” Kendra Cherry explores the following kinds of biases:

In evaluation research, especially when employing qualitative methods, such as interviews and focus groups, unconscious bias can negatively affect evaluation findings. The following types of bias are especially problematic in evaluations:

  • confirmation bias, when a researcher forms a hypothesis or belief and uses respondents’ information to confirm that belief.
  • acquiescence bias, also known as “yea-saying” or “the friendliness bias,” when a respondent demonstrates a tendency to agree with, and be positive about, whatever the interviewer presents.
  • social desirability bias, involves respondents answering questions in a way that they think will lead to being accepted and liked. Some respondents will report inaccurately on sensitive or personal topics to present themselves in the best possible light.
  • sponsor bias, when respondents know – or suspect – the interests and preferences of the sponsor or funder of the research, and modifies their (respondents) answers to questions
  • leading questions/wording bias, elaborating on a respondent’s answer puts words in their mouth because the researcher is trying to confirm a hypothesis, build rapport, or overestimate their understanding of the respondent.

It’s important to strive to eliminate bias in both our personal judgements and in social research. (For an extensive list of cognitive biases, see here.) Awareness of potential biases can alert us to when bias, rather than impartiality, influence our methods and affect our judgments.

Resources:

“How-to-think-about-implicit-bias,” Keith Payne, Laura Niemi, John M. Doris, March 27, 2018, Scientific American
“Bias in Social Research.” M. Hammersley, R. Gomm
January 21, 2020
21 Jan 2020

Did We Achieve What We Intended? Summative Evaluations

A summative evaluation is typically conducted near, or at the end of a program or program cycle. Summative evaluations seek to determine if, over the course of the intervention, the desired outcomes of a program were achieved. An “outcome” is that change, effect, or result which a program or initiative intends to achieve (See “What Counts as an “outcome” and Who Decides” ). Summative evaluations, as their name implies, offer a kind of “summary” of the value or worth of a program. Such an estimation is based on whether, and to what degree, intended outcomes have been achieved. Whereas formative evaluations are conducted near the beginning of a program, summative evaluations are conducted near or at the end of a program. Formative evaluations provide information with which to strengthen the implementation of the program. Conversely, summative evaluations determine whether the program should be continued or discontinued. (See our article “Strengthening Programs and Initiatives through Formative Evaluation”)

Summative evaluations are important because they gather and analyze data that indicate whether a program or initiative has been successful in affecting desired changes. Summative evaluations can be of use in making a case to potential funders and other stakeholders that continued support is a worthwhile investment. A word of caution: While it is important for funders to know that their investments are effective, and that desired changes are happening, summative evaluations may also provide evidence that discontinuation of a program is in order. (See “Fail Forward: What We Can Learn from Program ‘Failure’” )

Resources:

Understanding Different Types of Program Evaluation

“Building Our Understanding: Key Concepts of Evaluation What is it and how do you do it” Center for Disease Control

Evaluation, Second Edition, Carol H Weiss, Prentice Hall

“Types of Evaluation You Need to Know,” by Vipul Nanda

“Making Sense of Summative Evaluation: Three Tips for Making Those “Strings” Work in Your Favor,” by Heather Stombaugh

Just the Facts: Data Collection

January 7, 2020
07 Jan 2020

Strengthening Programs and Initiatives through Formative Evaluation

Formative evaluations are evaluations whose primary purpose is to gather information that can be used to improve or strengthen the implementation of a program or initiative. Formative evaluations typically are conducted in the early-to-mid period of a program’s implementation. Formative evaluations can be contrasted with summative evaluation which are conducted near, or at the end of, a program or program cycle, and are intended to show whether or not the program has achieved its intended outcomes (i.e., intended effects on individuals, organizations, or communities). Summative evaluations are used to indicate the ultimate value, merit, and worth of the program. Their findings can be used to determine whether the program should be continued, replicated, or curtailed.

The goal of formative evaluations is to gather information that can help program designers, managers, and implementers address challenges to the program’s effectiveness. In its paper “Different Types of Evaluation” the CDC notes that formative evaluations are implemented “During the development of a new program (or) when an existing program is being modified or is being used in a new setting or with a new population.” Formative evaluation allows for modifications to be made to the plan before full implementation begins, and helps to maximize the likelihood that the program will succeed.” “Formative evaluations stress engagement with stakeholders when the intervention is being developed and as it is being implemented, to identify when it is not being delivered as planned or not having the intended effects, and to modify the intervention accordingly.” See “Formative Evaluation: Fostering Real-Time Adaptations and Refinements to Improve the Effectiveness of Patient-Centered Medical Home Interventions”.

While there are many potential formative evaluation questions, the core of these consists of gathering information that answers:

  •  Which features of a program or initiative are working and which aren’t working so well?
  • Are there identifiable obstacles, or design features, that “get in the way” of the program working well?
  • Which components of the program do program participants say could be strengthened?
  • Which elements of the program do participants find most beneficial, and which least beneficial?Typically, formative evaluations are used to provide feedback in a timely way, so that the functioning of the program can be modified or adjusted, and the goals of the program better achieved.Brad Rose Consulting has conducted dozens of formative evaluation, each of which has helped program managers to understand ways that their program or initiative can be refined, and program participants better served.
November 5, 2019
05 Nov 2019

We’re All in This Together—The features and dynamics of groups

We—humans—spend a lot of time in groups. Families, workplaces, churches, mosques, and synagogues, political organizations, sports teams, clubs, associations, etc. A “group” is a collection of two or more people that interact, communicate, and influence one another. A crowded elevator or a subway car is not generally considered a group; it’s a crowd. A club or a work-team is a group.

Groups are the settings for a range of behaviors, all of which entail human interaction and influence. Individuals become members of groups in order to achieve goals and to satisfy needs. Groups have shared goals, or agendas, which include their “task agenda”—getting work done; and their “social agenda”—meeting the social-emotional and identity needs of members. Groups assign members to roles that prescribe a set of expectations for each member’s behavior. These roles typically have different statuses, or different levels of prestige associated with each role. There are “in-groups” and “out-groups”; the former are groups with which people identify as members and the latter are groups with which people don’t identify and are often “assigned” by members of other groups. An organization is a kind of group, whose members work together for a shared purpose in a continuing way. Organizations can contain various groups, both formal and informal, within its boundaries.

Groups have different levels of cohesion or incoherence. Both internal competition among group members and external competition with other groups, can affect the degree of cohesion, or solidarity of the group. While cohesion is important to most groups, if excessive, it can be the cause of undesirable factors like “groupthink” which can lower the quality of the group’s decision-making ability, lead to closed-mindedness, prejudice, and exert undue pressure to conform.

These features and dynamics (above) are applicable to most groups. They are especially noticeable at work, where group dynamics are often operative. Status of members, specified roles, pressures towards conformity and “groupthink”, leadership and “followership,” group decision-making, etc., are issues with which we must often deal—both consciously and unconsciously. In the for-profit world and the non-profit world, group dynamics are at play. Awareness of these features can help us to productively deal with them, rather than experience them unconsciously, and at times, adversely.

October 22, 2019
22 Oct 2019

What the Heck are We Evaluating, Anyway?

When you’re thinking about doing an evaluation — either conducting one yourself, or working with an external evaluator to conduct the evaluation — there are a number of issues to consider. (See our earlier article “Approaching an Evaluation—Ten Issues to Consider”)

I’d like to briefly focus on four of those issues:

  • What is the “it” that is being evaluated?
  • What are the questions that you’re seeking to answer?
  • What concepts are to be measured?
  • What are appropriate tools and instruments to measure or indicate the desired change?

1. What is the “it” that is being evaluated?

Every evaluation needs to look at a particular and distinct program, initiative, policy, or effort. It is critical that the evaluator and the client be clear about what the ‘it” is that the evaluation will examine. Most programs or initiatives occur in a particular context, have a history, involve particular persons (e.g. staff and clients/service recipients) and are constituted by a set of specific actions and practices (e.g., trainings, educational efforts, activities, etc.) Moreover, each program or initiative has particular changes (i.e. outcomes) that it seeks to produce. Such changes can be manifold or singular. Typically, programs and initiatives seek to produce changes in attitudes, behavior, knowledge, capacities, etc. Changes can occur in individuals and/or collectivities (e.g. communities, schools, regions, populations, etc.)

2. What are the questions that you’re seeking to answer?

Evaluations like other investigative or research efforts, involve looking into one or more evaluation questions. For example, does a discreet reading intervention improve students’ reading proficiency, or does a job training program help recipients to find and retain employment? Does a middle school arts program increase students’ appreciation of art? Does a high school math program improve students’ proficiency with algebra problems?

Programs, interventions, and policies are implemented to make valued changes in the targeted group of people that these programs are designed to serve. Every evaluation should have some basic questions that it seeks to answer. By collaboratively defining key questions, the evaluator and the client will sharpen the focus the evaluation and maximize the clarity and usefulness of the evaluation findings. (See “Program Evaluation Methods and Questions: A Discussion”)

3. What concepts are to be measured?

Before launching the evaluation, it is critical to clarify the kinds of changes that are desired, and then to find the appropriate measures for these changes. Programs that seek to improve maternal health, for example, may involve adherence to recommended health screening measures, e.g., pap smears. Evaluation questions for a maternal health program, therefore might include: “Did the patient receive a pap smear in the past year? Two years? Three years?” Ultimately, the question is “Does receipt of such testing improve maternal health?” (Note that this is only one element of maternal health. Other measures might include nutrition, smoking abstinence, wellness, etc.)

4. What are appropriate tools and instruments to measure or indicate the desired change?

Once the concept (e.g. health, reading proficiency, employment, etc.) are clearly identified, then it is possible to identify the measures or indicators of the concept, and to identify appropriate tools that can measure the desired concepts. Ultimately, we want tools that are able either to quantify, or qualitatively indicate, changes in the conceptual phenomenon that programs are designed to affect. In the examples noted above, evaluations would seek to show changes in program participants’ reading proficiency (education), employment, and health.

We have more information on these topics:

“Approaching an Evaluation—Ten Issues to Consider”

“Understanding Different Types of Program Evaluation”

“4 Advantages of an External Evaluator”

October 8, 2019
08 Oct 2019

Why Evaluate Your Program?

Program evaluation is a way to judge the effectiveness of a program. It can also provide valuable information to ensure that the program is maximally capable of achieving its intended results. Some of the common reasons for conducting program evaluation are to:

  • monitor the progress of a program’s implementation and provide feedback to stakeholders about various ways to increase the positive effects of the program
  • measure the outcomes, or effects, produced by a program in order to determine if the program has achieved success and improved the lives of those it is intended to serve or affect
  • provide objective evidence of a program’s achievements to current and/or future funders and policy makers
  • elucidate important lessons and contribute to public knowledge.

There are numerous reasons why a program manager or an organizational leader might chose to conduct an evaluation. Too often however, we don’t do things until we have to. Program evaluation is a way to understand how a program or initiative is doing. Compliance with a funder’s evaluation requirements need not be the only motive for evaluating a program. In fact, learning in a timely way about the achievements of, and challenges to, a program’s implementation—especially in the early-to-mid stages of a program’s implementation—can be a valuable and strategic endeavor for those who oversee programs. Evaluation is a way to learn about and to strengthen programs.

 

Resources:

August 6, 2019
06 Aug 2019

Politics is Not Forbidden. Should Nonprofits Rethink Their Political Agnosticism?

Non-profits have long operated under the assumption that they must remain non-political. A recent article by Bill Shore in the Stanford Social Innovation Review, July 17, 2019, argues that nonprofits need not be constrained by their presumed non-political status. In fact, Shore contends, “Nonprofits need to do much more of exactly what most of them don’t think they can or should do: influence public policy and its execution.”

Shore says that nonprofits can and should be more political than many in the nonprofit community believe they are legally permitted to be. Achieving the goals that many nonprofits pursue depends upon nonprofits becoming more, not less, political. While some activities are prohibited, including working on campaigns, donating to candidates, and engaging in lobbying beyond certain generously defined limits, shore notes that “… a broad range of political work is permitted, appropriate, even essential. (There is also the option of establishing a 501(c)(4) that permits campaign engagement and support, which we haven’t done.)”

Shore asserts getting political is often about educating, not necessarily lobbying or campaigning. He further argues that, “Nonprofits need to build their internal political capacity. Nonprofit political activity is good for nonprofits, good for politics, and good for the people that both aim to serve.” Ultimately, by expanding their political activities within stipulated legal limits, “nonprofits benefit by seeing their programs and services achieve greater scale and reach more people in need, in ways that only politics and public policy can guarantee.”

Resources:

“Getting Political Is Good for Everyone,” Bill Shore, Stanford Social Innovation Review, July 17, 2019

July 23, 2019
23 Jul 2019

Don’t Blame the Robots

We’ve previously written about the impact of Artificial Intelligence (AI) and new computer technologies on employment in the US. (See, for example, “Humans Need Not Apply:
What Happens When There’s No More Work?” “Will President Trump’s Wall Keep Out the Robots? ) Two recent articles further highlight the effects associated with the advance of AI. The first, “Robots Are Not Coming for Your Job. Management Is,” by Brian Merchant, at Gizmodo argues that while the robots-stealing-your-job headlines make for good copy, robots are not to blame for current and anticipated technological displacement. Blaming robots, is to misattribute the origin of the uses of technology. “Robots are not threatening your job. Business-to-business salesmen promising automation solutions to executives are threatening your job. Robots are not killing jobs. The managers who see a cost benefit to replacing a human role with an algorithmic one and choose to make the switch are killing jobs. Robots are not coming for your job. The CEOs who see an opportunity to reap greater profits in machines that will make back their investment in three point seven years and send the savings upstream — they’re the ones coming for your job.” Merchant says that merely technologizing the phenomenon, is to obscure the real sources (and possible solutions to) the problem. “Letting an ambiguous conception of ‘robots’ instead shoulder the blame lets the managerial class evade scrutiny for how it deploys automation, shuts down meaningful discussion about the actual contours of the phenomenon, and prevents us from challenging the march of this manifest “robodestiny” when it should be challenged.”

In “A Machine May Not Take Your Job, but One Could Become Your Boss,” by Kevin Roose, New York Times, June 23, 2019, the author says “…in all of the worry about the potential of artificial intelligence to replace rank-and-file workers, we may have overlooked the possibility it will replace the bosses, too.” Roose observes that one of the goals of AI is to optimize efficiency of humans in the workplace. Thus systems that monitor, guide, and report on employee performance are increasingly seen in white collar workplaces, where employees are being “assisted” to be more productive, more customer friendly, and work more quickly — by “adjunct management,” i.e., artificial intelligence.

Roose scans a variety of workplaces.  In the insurance industry, he reports on AI systems that provide on-screen prompts to call center works — prompting them to be chirpier and more empathetic; he discusses the use of AI and employee tracking in both Amazon warehouses and retail stores. In the later he notes that 7-Eleven “uses in-store sensors to calculate a “true productivity” score for each worker, and rank workers from most to least productive,” and notes “Uber, Lyft and other on-demand platforms have made billions of dollars by outsourcing conventional tasks of human resources — scheduling, payroll, performance reviews — to computers.

Management by algorithm doesn’t just affect call center works, Uber drivers, and warehouse workers. It also bodes less-than-well for managers, whose traditional supervisory and over-site duties are increasingly being handled by “robots”. As we discussed in “Humans Need Not Apply,” AI promises to displace both blue collar, manual laborers and white collar, college-educated professionals — the latter including but not limited to, lawyers, computer programmers, managers, office and retail workers. “A Machine May Not Take Your Job, but One Could Become Your Boss,” hauntingly suggests that management too, is in the cross-hairs of AI.

Resources:

“Robots Are Not Coming for Your Job.  Management Is,” Brian Merchant, Gizmodo

“A Machine May Not Take Your Job, but One Could Become Your Boss,” by Kevin Roose, New York Times, June 23, 2019
July 9, 2019
09 Jul 2019

Ranking, Rating, and Measuring Everything

In an important and brief new book, The Metric Society: on the Quantification of the Social, (Polity, 2019), German sociologist Steffen Mau argues that the historic growth in the availability of data and a seeming societal obsession with quantitatively measuring and ranking everything, is fast making us a “metric society. “A cult of numbers masquerading as rationalization” he says, is having unparalleled impact on how we understand both social and personal value. We are becoming increasingly trapped in a social world where, “The possibilities of life and activity logging are growing apace: consumption patterns, financial transactions, mobility profiles, friendship networks, states of health, education activities, work output, etc.—all this is becoming statistically quantifiable.” Such quantification is far from neutral and scientific, Mau says. It leads to ever greater tendencies, both individual and institutional, to classify, differentiate, and construct social hierarchies. He argues further that these tendencies are paving the way for us to become “an evaluation society,” a society where individuals constantly measure and compare their social worth with others (e.g., dating sites and Facebook “likes”) and where both corporations and the state sort people, based on narrow statistics, into categories that ultimately have differential access to valuable resources.

While the book is filled with examples, Chapter 5, “The Evaluation Cult: Points and Stars,” explores how “‘the evaluation cult’ is binding us to the metrics of measurement, evaluation, and comparison.” Mau scans the proliferation of various tools for evaluation: satisfaction surveys, preference measures, self-assessments, health tracking algorithms, and myriad ranking systems, ranging from Yelp, to publicly available starred reviews of medical providers and lawyers. He shows us how such ratings and rankings—often justified by the claims of providing “transparency,” helpful information, and consumer influence on service providers and products— are upending both markets and the professions, in some cases driving companies to purchase good reviews.

Mau raises questions not just about the validity of measures (after all, what is the difference between a three-star restaurant rating and a four-star rating?), but argues that the growth in the use of such measures is transforming how we view and value ourselves and others. “The universal language of numbers, their lack of ambiguity, and the illusion of commensurability, pave the way for the hegemony of a metrics-based apparatus of comparison.” He says that today, we are witnessing and participating in the emergence of a new “status regime” characterized by quantification and numerical ranking. This “quantitative comparison is frequently translated into a competitive ethos of better versus worse, more versus less.”

Among the other observations Mau offers:

  • growing reliance upon numbers changes our everyday notions of value and social status
  • the availability of quantitative information reinforces the tendency toward social comparison and rivalry
  • quantitative measurement of social phenomena fosters the expansion of competition
  • representations of quantitative data, such as graphs, tables, lists, and scores, change qualitative differences into quantitative inequalities
  • the availability of, and reliance upon, quantitative data leads to further social hierarchization

Ultimately, “…the measurement and quantification of the social realm are not neutral representations of reality. On the contrary, they are representative of specific orders of worth which are invariably based on forgone conclusions as to what can and should be measured and evaluated, and by what means. Metrics may claim to give an objective, accurate, and rational picture of the world as it is, but they also contribute, through the selection, weighting, and linking of information, to the establishment of the normative order.” Essentially. Mau raises a perennial question that is relevant to all evaluative efforts: Do we measure what’s valuable, or is it valuable because we choose to measure it? Please see our previous posts “The Tyranny of Metrics” and “What Counts as an ‘Outcome’—and Who Determines?” Mau argues further that we are becoming a society obsessed with managing our reputations, and ultimately a society of ever greater competition and rivalry.

Resources:

The Metric Society: On the Quantification of the Social, (Polity, 2019),

Heather Douglas, “Facts, Values, and Objectivity”

Max Weber, Objectivity in the Social Sciences 

Max Weber, Methodology of the Social Sciences, Transaction Press, 2011

May 29, 2019
29 May 2019

Being Smart About What You Feel

Emotional Intelligence

In a previous blogpost, “Interpersonal Skills Enhance Program Evaluation,” we discussed the importance of interpersonal and relational skills for program evaluators. These skills make effective and responsive interpersonal interaction possible. “Emotional Intelligence” underlies many of these skills. Emotional Intelligence, first explored by Daniel Goleman, in his book Emotional Intelligence, Why It Matters More Than IQ, Bantam Books, 1995 is the ability to recognize, manage, and utilize both one’s own emotions and the emotions of others. Emotional Intelligence, as summarized by Eric Ravenscraft in his recent article “Emotional Intelligence: The Social Skills You Weren’t Taught in School,” Lifehacker, February 20, 2019, includes the following elements:

  • Self-awareness: Self-awareness involves knowing your own feelings. This includes having an accurate assessment of what you’re capable of, when you need help, and what your emotional triggers are.
  • Self-management: This involves being able to keep your emotions in check when they become disruptive. Self-management involves being able to control outbursts, calmly discussing disagreements, and avoiding activities that undermine you like extended self-pity or panic.
  • Motivation: Everyone is motivated to action by rewards like money or status. Goleman’s model, however, refers to motivation for the sake of personal joy, curiosity, or the satisfaction of being productive.
  • Empathy: While the three previous categories refer to a person’s internal emotions, this one deals with the emotions of others. Empathy is the skill and practice of reading the emotions of others and responding appropriately.
  • Social skills: This category involves the application of empathy as well as negotiating the needs of others with your own. This can include finding common ground with others, managing others in a work environment, and being persuasive.

 

Critiques of Emotional Intelligence

Although Emotional Intelligence (EI) has become an increasingly accepted concept, there are some who question its distinctiveness and validity. Some say that it is difficult to distinguish from regular IQ, that is not really a kind of intelligence but a set of behaviors, and that it is nearly impossible to objectively measure.

A recent article argues that the idea of “reading” the emotions of oneself and of others is itself a problematic conception. In “Emotional Intelligence Needs a Rewrite”, Lisa Feldman Barrett, Nautilus, August 3, 2017, writes that “The traditional foundation of Emotional Intelligence rests on two common-sense assumptions. The first is that it’s possible to detect the emotions of other people accurately. That is, the human face and body are said to broadcast happiness, sadness, anger, fear, and other emotions, and if you observe closely enough, you can read these emotions like words on a page. The second assumption is that emotions are automatically triggered by events in the world, and you can learn to control them through rationality.”

Feldman Barrett argues that neither of these assumptions stands up to scientific investigation. Research shows that faces and bodies alone do not communicate any specific emotion, in any consistent manner, and that since the brain doesn’t have separate regions—one for emotion, and one for cognition—the belief that we can control or manage emotions using our rational brains is fallacious. She argues that although it may sound appealing and reasonable that we can detect emotions in others by observing their faces and bodies, expressions are neither universal nor mono-emotional. “When it comes to detecting emotion in other people, the face and body do not speak for themselves.”  She says that, “Your brain may automatically make sense of someone’s movements in context, allowing you to guess what a person is feeling, but you are always guessing, never detecting.”

The author offers an alternative, neuroscientific view of how the brain works. She says that our brains “create all thoughts, emotions, and perceptions, automatically and on the fly, as needed. This process is completely unconscious. It may seem like you have reflex-like emotional reactions and effortlessly detect emotions in other people, but under the hood, your brain is doing something else entirely.”  Essentially our brains are survival-oriented prediction engines, that produce responses to internal and external stimuli that “become the emotions we experience and the expressions we perceive in other people.” Therefore “Emotional Intelligence requires a brain that can use prediction to manufacture a large, flexible array of different emotions. If you’re in a tricky situation that has called for emotion in the past, your brain will oblige by constructing the emotion that works best.”

Feldman Barrett argues that we don’t so much observe emotions in ourselves and others, as we construct them and predict them.  She further argues that we can give our “constructivist” brains (and their concomitant emotions) a boost by enhancing the granularity of our sensitivity to our feelings and emotional states. One way we can do this is by learning greater vocabularies to describe our own and other’s feeling states, and thereby priming our prediction engines to “guess” what others are feeling, with even more specificity.

Whether our brains construct emotions and predict the emotions of others seems largely irrelevant to the issue of the importance of understanding emotions in ways that help us relate to, and interact with, others.  In the final analysis, the human social world is composed by thinking and feeling beings, and those who can understand (“predict” in Feldman Barrett’s view) and manage emotions will be better prepared to engage in that world.

Resources:

Daniel Goleman, Emotional Intelligence, Why It Matters More Than IQ.   Bantam Books, 1995.

Emotional Intelligence: “The Social Skills You Weren’t Taught in School” Eric Ravenscraft.  Lifehacker, February 20, 2019

“Emotional Intelligence Needs a Rewrite”, Lisa Feldman Barrett, Nautilus, August 3, 2017.

“What is Emotional Intelligence” Michael Akers & Grover Porter,  Oct 8, 2018 Psych Central

“The Benefits of Emotional Intelligence,”  Paula Durlofsky, July 8 2018, Psych Central
May 21, 2019
21 May 2019

Lying at Work

It’s sometimes difficult to tell the truth, especially in arenas like the workplace, where inequalities of power and authority make it difficult to “speak truth to power.” In a recent Harvard Business Review article, “4 Ways Lying Becomes the Norm at a Company” (February 15, 2019) Ron Carucci discusses the results of a substantial, 15 year longitudinal study that examined the systemic (vs. personal) incentives for dishonesty. Carucci says there are a range of incentives, or prompts, for employees to be less than honest at work.  Among these:

  • Inconsistency: An inconsistency between an organization’s stated mission, objectives, and values, and the way it is actually experienced by employees and the marketplace. As one interviewee put it, “Our priorities change by the week. Nobody wants to admit we’re in trouble, so we’re grasping at straws. We don’t know who we are anymore, so we’re just making things up.”
  • Unjust accountability systems, especially when an organization’s processes for measuring employee contributions is perceived as unfair or unjust. Research shows that people are nearly 4 times more likely to withhold or distort information when the system is perceived to be unfair or rigged.
  • Poor organizational governance; for example there is no effective process to gather decision makers into honest conversations about tough issues. Truth is forced underground, leaving the organization to rely on rumors and gossip.
  • Inter-group rivalry, conflict, and competition (what Carlucci terms “weak cross-functional collaboration.”) is a predictor of people withholding information or distorting truthful information. Additionally, Carlucci observes that isolation, fragmentation, and departmental/divisional loyalties often result in dishonesty or a damaging lack of candor.

Because these factors are cumulative, an organization afflicted with all four of these factors is 15 times more likely to end up in an “integrity catastrophe” than those who have none of these four integrity/honesty problems. Carlucci argues however, that these organizational problems are alterable and that a culture of honesty can be achieved by companies and organizations that challenge these issues.

Resources:

“4 Ways Lying Becomes the Norm at a Company,” Ron Carucci, Harvard Business ReviewFebruary 15, 2019

April 23, 2019
23 Apr 2019

Talking Back to Foundations

Philanthropies play an important role in contemporary society. They are, by their very nature, focused on supporting the public good and human welfare. While philanthropies, each year, channel vast sums to the achievement of laudable goals, their social and economic power has raised questions about their unalloyed reputation for ‘doing good’. See, for example, our previous blogpost, “Philanthrocapitalism?”

In an attempt to provide a voice to those who have worked with foundations, a new website, GrantAdvisor, offers grantees of foundations a safe way to anonymously give feedback about the grantmaking/receiving process. GrantAdvisor also offers foundations an opportunity to learn about grantees’ experience working with foundation staff.

GrantAdvisor effectively serves as a kind “Yelp” to those in the non-profit community. Here for example is a link to grantees experience working with the Wal-Mart Foundation.

You can view GrantAdvisor here to learn more about how the site works and to review the reviews of a number of important national and local foundations.

Resources:

“A Place Where You Can Speak Your Mind to That Foundation,” Amy Costello. Non Profit Quarterly

“Benchmarking Foundation Evaluation Practices,” Center for Effective Philanthropy

“Helping Community Foundations Strengthen Grantees’ Effectiveness”

March 19, 2019
19 Mar 2019

Glitches in Philanthropy?

Over the years, Brad Rose Consulting has provided evaluation services to philanthropies and community service organizations. These clients are dedicated to making the world a better place, often through philanthropic work with disadvantaged populations. While the work of philanthropies is generally perceived as laudable, there are a number of potential objections to the rise of philanthropic largesse.

In a BBC article, “The Problems with Charity” the authors survey a number of potential objections to, and liabilities associated with, charitable and philanthropic work. These include:
  • Charities often target symptoms, not causes- Charity helps the recipient with their problem, but it doesn’t do much to deal with the causes of that problem.
  • Charity may become a substitute for real justice- The idea that charity is wrong when it’s used to patch up the effects of the fundamental injustices that are built into the structure and values of a society.
  • Charity may not provide the best solution to a problem- Charitable giving may not be the most effective way of solving world poverty. Indeed, charitable giving may even distract from finding the best solution – which might involve a complex rethink of the way the world organizes its economic relationships, and large-scale government initiatives to change people’s conditions.
  • Charity may benefit the state rather than the needy- If the charity sector increases spending in an area also funded by government then there is a risk that government will choose to spend less in that area.
  • Charities are often accountable to the givers not the receivers- because the recipients of charity are often unorganized and the charity doesn’t know their individual identities, it’s often easier for charities to make their performance reports to the givers.

In his article “The Downside of Doing Good,” David Campbell examines recent critiques of philanthropy.  Following Anand Giridharadas (Winners Take All: The Elite Charade of Changing the World) Campbell argues that “wealthy philanthropists and other prominent social change leaders often co-exist in a parallel universe (called) “MarketWorld,” where the best solutions to society’s problems require the same knowhow used in corporate boardrooms. That is because MarketWorld ignores the underlying causes for problems like poverty and hunger.

He further observes that efforts at educational reform funded by such philanthropic luminaries as Bill and Melinda Gates, via support for charter schools, often fail to understand the underlying inequalities that make schools resourced in vastly different ways. “As long as school systems are funded locally, based on property values, students in wealthy communities will have advantages over those residing in poorer ones. However, creating a more equal system to pay for schools would take tax dollars and advantages away from the rich. The wealthy would lose, and the disadvantaged would win. So it’s possible to see the nearly $500 million that billionaires and other rich people have pumped into charter schools and other education reform efforts over the past dozen years, as a way to dodge this problem.”

While philanthropy is likely to remain with us for the foreseeable future, the books and articles mentioned here underline some of the problematic assumptions that haunt philanthropic attempts to ameliorate conditions whose causes are often the deeper, underlying dynamics associated with the societies and economies these philanthropies seek to amend.

Resources

Winners Take All: The Elite Charade of Changing the World, by Anand Giridharadas

Just Giving Why Philanthropy Is Failing Democracy and How It Can Do Better, by Rob Reich

“The Problem with Philanthropy,”  David Sirota, In These Times, June 13, 2014
February 19, 2019
19 Feb 2019

Power in Organizations

Most of us spend a good portion of our lives in organizations or indirectly relating to organizations (businesses, non-profits, civic and legal organizations, religious organizations, military and criminal justice organizations, etc.). One might say that in the modern world, we “live in” an environment composed largely of organizations. (See our previous blogposts “Organization Development: What Is It & How Can Evaluation Help?”  and “What’s the Difference? 10 Things You Should Know About Organizations vs. Programs”)

Organizations contain, utilize, and deploy various kinds of social power. Such power is the capacity of individuals and groups to affect, control, or influence outcomes (i.e. changes). Power doesn’t exist in isolation, but in relationships to other individuals and/or groups. If we want to accomplish goals at work—whether these goals are about producing widgets, or making the world a better place—we need to draw on and negotiate various kinds of formal and informal power. Sometimes it may be useful to think of various resources as sources of power. Tangible resources include, money, machinery, physical infrastructure, etc. Less tangible, but no less important, resources may include, authority, social status/prestige, social networks, individuals’ intelligence, professional experience, even social attractiveness and charisma. Both tangible and intangible resources are used as sources of power with which organizations achieve objectives and goals.

In her article, “Types of Powers in Organizations,” Diana Dahl summarizes 7 types of power in organizations. These include:

  1. Coercive Power— a person or group is able to punish others for not following orders has coercive power.
  2. Connection Power— connection power is gained by knowing and being listened to by influential people. Increasing connections and mastering political networking lead to a greater potential for connection power.
  3. Reward Power— the ability to give rewards to other employees. Rewards are not always monetary, such as improved work hours and words of praise.
  4. Legitimate Power (also known as legitimate authority)— when employees believe a person can give orders based on his position within the organization, such as when a manager orders staff members to complete a task and they comply because the orders came from their superior.
  5. Referent Power— people who are liked, respected, and are viewed by other employees as worth emulating. Supervisors who lead by example, treat employees with respect, seek their collaboration and gain the trust of their employees possess referent power.
  6. Informational Power— access to valued information. This power can be quickly fleeting because once the needed information is shared, the person’s power is gone.
  7. Expert Power— the greater a person’s knowledge or specialized skill set, the greater her potential for expert power.

People in organizations must get things accomplished. Having a clear idea of what constitutes organizational power and the kinds of resources that need to mobilized to reach goals, may help us to better navigate what are often complicated and contentious organizations.

Resources

Power in Organizations: Structures, Processes and Outcomes, by Richard Hall and Pamela Tolbert Pearson/Prentice Hall, 2005, 9th edition

“Types of Powers in Organizations,” by Diana Dahl ; Updated September 26, 2017

Links to and summaries of “Theories of Organizational Power” at bankofinfo.com/theories-of-organizational-power

“Power and Politics in Organizational Life,” Abraham Zaleznik, Harvard Business Review

“Power in Organizations: A Way of Thinking About What You’ve Got, and How to Use It,” Roelf Woldring

Summary of tactics to gain organizational power 

January 8, 2019
08 Jan 2019

Philanthrocapitalism?

Philanthropy, most of us presume, is a good thing. Philanthropic foundations seek to make the world a better place. In the US, philanthropic foundations have played an important role in funding, designing and “testing” a variety of programs and initiatives that seek to solve the most intransigent social problems of American society, from homelessness and education reform, to health care and access to the arts. As of 2015, there were over 86,000 foundations in the US alone, with total assets of $890,061,214,247.  (See http://data.foundationcenter.org/ )

In their recent article, “The Trouble with Charitable Billionaires,” in the May 28, 2018, Guardian, Carl Rhodes and Peter Bloom argue that although philanthropy appears to be a socially valuable activity, the recent emergence of “philanthrocapitalism” such as that conducted by Mark Zukerberg and his wife, Priscilla Chan, Warren Buffet,  Bill and Melinda Gates, and others, in fact, shifts decision-making power about social change from the public to a stratum of wealthy donors, whose vision for social change is not always innocent or desirable. “Essentially, what we are witnessing is the transfer of responsibility for public goods and services from democratic institutions to the wealthy, to be administered by an executive class.”

Rhodes and Bloom write that, since the 1990s, there has emerged a cohort of billionaires who appear genuinely committed to addressing persistent and seemingly intractable social problems, and who are now investing literally billions of dollars in efforts to tackle these problems. They note that it would seem that many of the world’s richest people simply want to give their money away to good causes, and have thus created a “golden age of philanthropy.” The authors, however, caution that “The golden age of philanthropy is not just about benefits that accrue to individual givers. More broadly, such philanthropy serves to legitimize capitalism, as well as to extend it further and further into all domains of social, cultural and political activity.” Furthermore, “Philanthrocapitalism” they say, “takes the application of management discourses and practices from business corporations and adapts them to charitable work. The focus is on entrepreneurship, market-based approaches and performance metrics…(and these) result, at a practical level, in a philanthropy that is undertaken by CEOs in a manner similar to how they run businesses.”

Rhodes and Bloom problematize this new era of corporate social responsibility and philanthrocapitalism.  “Philanthrocapitalism is commonly presented as the social justice component of an otherwise amoral global free market. At best, corporate charity is a type of voluntary tax paid by the 1% for their role in creating such an economically deprived and unequal world.” Additionally, “Philanthrocapitalism is about much more than the simple act of generosity it pretends to be, instead involving the inculcation of neoliberal values personified by the billionaire CEOs who have led its charge. Philanthropy is recast in the same terms in which a CEO would consider a business venture. Charitable giving is translated into a business model that employs market-based solutions characterized by efficiency and quantified costs and benefits. Ultimately, the authors contend, “…we find a new form of corporate rule, refashioning another dimension of human endeavor (i.e. philanthropy) in its own interests. Such is a society where CEOs are no longer content to do business; they must control public goods as well. In the end, while the Giving Pledge’s website (a philanthropy campaign initiated by Warren Buffett and Bill Gates in 2010 which targets billionaires around the world, encouraging them to give away the majority of their wealth) may feature more and more smiling faces of smug-looking CEOs, the real story is of a world characterized by gross inequality that is getting worse year by year.”

Essentially Rhodes and Bloom question the influence of private fortunes on public issues. While foundations may have the best of intentions, they mobilize private money to address privately defined public issues.  This is hardly a new development.  Rhodes and Bloom however, question whether the more recent importation of business methods, measures, and beliefs are either the best, or the most democratic, way to address deep systemic issues. Their view is that for the many “wounds” experienced by contemporary society, philanthrocapitalism may be merely a self-serving “band-aid.”

Resources

 “The Trouble with Charitable Billionaires,” Carl Rhodes and Peter Bloom, The Guardian

How Liberal Nonprofits Are Failing Those They’re Supposed to Protect,” William C. Anderson,  Verso Blog, October 6, 2017

American Foundations: Roles and Contributions, by Helmut K. Anheier (Editor), David C. Hammack (Editor), The Brookings Institution, 2010

The Self-Help Myth: How Philanthropy Fails to Alleviate Poverty, Erica Kohl-Arenas, University of California Press, 2016

December 4, 2018
04 Dec 2018

Strengthening Program AND Organizational Effectiveness

Program evaluation is seldom simply about making a narrow judgment about the outcomes of a program (i.e., whether the desired changes were, in fact, ultimately produced.) Evaluation is also about helping to provide program implementers and stakeholders with information that will help them strengthen their organization’s efforts, so that desired programmatic goals are more likely to be achieved.

Brad Rose Consulting is strongly committed to translating evaluation data into meaningful and actionable knowledge, so that programs, and the organizations that host programs, can strengthen their efforts and optimize results. Because we are committed not just to measuring program outcomes, but to strengthening the organizations that host and manage programs, we work at the intersection of program evaluation and organization development (OD).

Often challenges facing discrete programs reflect challenges facing the organizations that host programs. (For the difference between “organizations” and “programs” see our previous post “What’s the Difference? 10 Things You Should Know About Organizations vs. Programs,” ) Program evaluations thus present opportunities for host organizations to:
  • engage in the clarification of their goals and purposes
  • enhance understanding of the often implied relationships between a program’s causes and effects
  • articulate for internal stakeholders a collective understanding of the objectives of their programming
  • reflect on alternative concrete strategies to achieve desired outcomes
  • strengthen internal and external communications
  • improve relationships between individuals within in programs and organizations
Although Brad Rose Consulting evaluation projects begin with a focus on discrete programs and initiatives, the answers to the questions that drive our evaluation of programs often provide vital insights into ways to strengthen the effectiveness of the organizations that host, design, and implement those programs. (See “Logic Modeling: Contributing to Strategic Planning” )
Typically, Brad Rose Consulting works with clients to gather data that will help to improve, strengthen, and “nourish” both programs and organizations. For example, our formative evaluations, which are conducted during a project’s implementation, aim to improve a program’s design and performance. (See “Understanding Different Types of Program Evaluation” ) Our evaluation activities provide program managers and implementers with regular, data-based briefings, and with periodic written reports so that programs can make timely adjustments to their operations. Formative feedback, including data-based recommendations for program refinement, can also help to strengthen the broader organization, by identifying opportunities for organizational learning, clarifying the goals of the organization as these are embodied in specific programming, specifying how programs and organizations work to produce results (i.e., articulating cause and effect) and by strengthening systems and processes.

Resources

“What’s the Difference? 10 Things You Should Know About Organizations vs. Programs,”

“Logic Modeling: Contributing to Strategic Planning”

“Understanding Different Types of Program Evaluation”

October 24, 2018
24 Oct 2018

Do Work Teams Work?

“Collaboration” and “teamwork” are the catchphrases of the contemporary workplace. Since the 1980s in the U.S., work teams have been hailed as the solution to assembly line workers’ alienation and disaffection, and white-collar workers’ isolation and disconnection. Work teams have been associated with increased productivity, innovation, employee satisfaction, and reduced turnover. Additionally, teams at work are said to have beneficial effects on employee learning, problem-solving, communication, company loyalty, and organizational cohesiveness. Teams are now found throughout the for-profit, non-profit, and governmental sectors, and much of the work of the field of organization development (OD) is devoted to fostering and sustaining teams at work.

In his recent article “Stop Wasting Money on Team Building,” Harvard Business Review, September 11, 2018, Carlos Valdes-Dapena, argues that teams are less effective than many believe them to be. Based on research conducted at Mars, Inc. “a 35 billion dollar global corporation with a commitment to collaboration,” Valdes-Dapena argues that while employees like the idea of teams and team work, employees don’t, in fact, much collaborate in teams. After conducting 125 interviews and administering questionnaires with team members, he writes “If there was one dominant theme from the interviews, it is summarized in this remarkable sentiment: “I really like and value my teammates. And I know we should collaborate more. We just don’t.”

Valdes-Dapena reports that employees “…felt the most clarity about their individual objectives, and felt a strong sense of ownership for the work they were accountable for.” He also shows that “Mars was full of people who loved to get busy on tasks and responsibilities that had their names next to them. It was work they could do exceedingly well, producing results without collaborating. On top of that, they were being affirmed for those results by their bosses and the performance rating system.” Essentially, Valdes-Dapena, argues, teams may sound good in theory, but it is probably better to tap individual self-interest, if you really want to get the job done.

In “3 Types of Dysfunctional Teams and How To Fix Them,” Patty McManus says that there are different types of dysfunctional work teams. She characterizes these different team types as: “The War Zone,” “The Love Fest,” and “The Unteam.” In “War Zone” teams, competition and factionalism among members obscure or derail the potential benefits of teamwork. In the “Love Fest” team, there is a focus on muting disagreements, highlighting areas of agreement, and avoidance of tough issues in the interest of maintaining good feelings. “The Unteam” is characterized by meetings that are used for top-down communication and status updates, and fail to build shared perspective about the organization. In the “Unteam” members may get along as individuals, but they have little connection to one another or a larger purpose they all share.

McManus claims that the problems of teams may be overcome by what she terms “ecosystems teams,” i.e., teams that surface and manage differences, build healthy inter-dependence among members, and engage the organization—beyond the mere confines of the team.

Matthew Swyers also sees problems in teams at work. In “7 Reasons Good Teams Become Dysfunctional,” (Inc. September 27, 2012,) Swyers writes that there are seven types of problems that teams may experience:

  • absence of a strong and competent leader
  • team members more interested in individual glory than achieving team objectives
  • failure to define team goals and desired outcomes
  • disproportionately place too much of the team’s work on a few of its members’ shoulders
  • lack focus and endless debate, without moving toward an ultimate goal
  • lack of accountability and postponed timetables
  • failure of decisiveness.

Each of these writers highlight the vulnerabilities of teams at work. Although the work of these writers doesn’t foreclose the positive possibilities of team organization at work, they raise important questions about both the enthusiasm for, and the effectiveness of, teams. Additionally, each author suggests that with enlightened modifications, organizations can overcome the liabilities of teams and begin to reap the benefits of team-based employee collaboration. That said, none of these writers, and few among the other U.S. based writers who have engaged this topic, treat the underlying assumptions of workplace reform—that work can be made more habitable and humane without the independent organizations that have traditionally represented workers’/employees’ interests in the workplace. For discussion of models of workplace reform that genuinely represent workers’ interest in more humane, collaborative, and ultimately, productive working environments, we will need to look elsewhere.

Resources:

Workgroups vs. Teams

“Importance of Teamwork at Work,” Tim Zimmer

“Importance of Teamwork in Organizations,” Aaron Marquis

“What Makes Teams Work?” Regina Fazio Maruca, Fast Company

“Stop Wasting Money on Team Building,” Carlos Valdes-Dapena, Harvard Business Review, September 11, 2018

“3 Types Of Dysfunctional Teams And How To Fix Them,” Patty McManus, Fast Company

“When Is Teamwork Really Necessary?” Michael D. Watkins, Harvard Business Review, August 16, 2018

“7 Reasons Good Teams Become Dysfunctional,” Matthew Swyers , Inc. Sept 27 2012

“Why Teams Don’t Work,” Diane Coutu, Harvard Business Review, May 2009

December 11, 2017
11 Dec 2017

What’s the Difference? 10 Things You Should Know About Organizations vs. Programs

Organizations vs. Programs

Organizations are social collectivities that have: members/employees, norms (rules for, and standards of, behavior), ranks of authority, communications systems, and relatively stable boundaries. Organizations exist to achieve purposes (objectives, goals, and missions) and usually exist in a surrounding environment (often composed of other organizations, individuals, and institutions.) Organizations are often able to achieve larger-scale and more long-lasting effects than individuals are able to achieve.  Organizations can take a variety of forms including corporations, non-profits, philanthropies, and military, religious, and educational organizations.

Programs are discreet, organized activities and actions (or sets of activities and actions) that utilize resources to produce desired, typically targeted, outcomes (i.e., changes and results). Programs typically exist within organizations. (It may be useful to think of programs as nested within one or, in some cases, more than one organization.) In seeking to achieve their goals, organizations often design and implement programs that use resources to achieve specific ends for program participants and recipients. Non-profit organizations, for example, implement programs that mobilize resources in the form of activities, services, and products that are intended to improve the lives of program participants/recipients. In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, services, and materials, to positively affect the lives of those they serve.

What is Program Evaluation?

Program evaluation is an applied research process that examines the effects and effectiveness of programs and initiatives. Michael Quinn Patton notes that “Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs in order to make judgements about the program, to improve program effectiveness, and/or to inform decisions about future programming. Program evaluation can be used to look at:  the process of program implementation, the intended and unintended results produced by programs, and the long-term impacts of interventions. Program evaluation employs a variety of social science methodologies–from large-scale surveys and in-depth individual interviews, to focus groups and review of program records.” Although program evaluation is research-based, unlike purely academic research, it is designed to produce actionable and immediately useful information for program designers, managers, funders, stakeholders, and policymakers.

Organization Development, Strategic Planning, and Program Evaluation

Organization Development is a set of processes and practices designed to enhance the ability of organizations to meet their goals and achieve their overall mission. It entails “…a process of continuous diagnosis, action planning, implementation and evaluation, with the goal of transferring (or generating) knowledge and skills so that organizations can improve their capacity for solving-problems and managing future change.” (See: Organizational Development Theory, below) Organization Development deals with a range of features, including organizational climateorganizational culture (i.e., assumptions, values, norms/expectations, patterns of behavior) and organizational strategy. It seeks to strengthen and enhance the long-term “health” and performance of an organization, often by focusing on aligning organizations with their rapidly changing and complex environments through organizational learning, knowledge management, and the specification of organizational norms and values.

Strategic Planning is a tool that supports organization development. Strategic planning is a systematic process of envisioning a desired future for an entire organization (not just a specific program), and translating this vision into broadly defined set of goals, objectives, and a sequence of action steps to achieve these. Strategic planning is an organization’s process of defining its strategy, or direction, and making decisions about allocating its resources to pursue this strategy.

Strategic plans typically identify where and organization is at and where it wants to be in the future. It includes statements about how to “close the gap,” between its current state and its desired, future state. Additionally, strategic planning requires making decisions about allocating resources to pursue an organizations strategy. Strategic planning generally involves not just setting goals and determining actions to achieve the goals, but also mobilizing resources.

Program evaluation is uniquely able to contribute to organization development–the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency. Although evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations collect, synthesize, and report information that can be used to improve the broader operation and health of the organization that hosts the program. Additionally, program evaluation can aid the strategic planning process, by using data about an organization’s programs to indicate whether the organization is successfully realizing its goals and mission through its current programming.

Brad Rose Consulting works at the intersection of evaluation and organization development. While our projects begin with a focus on discrete programs and initiatives, the answers to the questions that drive our evaluation research provide vital insights into the effectiveness of the organizations that host, design, and fund those programs. Findings from our evaluations often have important implications for the development and sustainability of the entire host organization.

Resources:

Organizations: Structures, Processes, and Outcomes, Richard H. Hall and Pamela S Tolbert, Pearson Prentice Hall, 9th edition.

Utilization Focused Evaluation, Michael Quinn Patton, Sage, 3rd edition, 1997

Organization Development: What Is It & How Can Evaluation Help?

Organization Development

Organizational Development Theory

Strategic Planning, Bain and Co. 2017

Strategic Planning

What a Strategic Plan Is and Isn’t
Ten Keys to Successful Strategic Planning for Nonprofit and Foundation Leaders

Elements of a Strategic Plan

Types of Strategic Planning
Understanding Strategic Planning

Five Steps to a Strategic Plan
Five Steps to a Strategic Plan

The Big Lie of Strategic Planning, Roger L. Martin, Harvard Business Review, January-February 2014

December 7, 2016
07 Dec 2016

Organization Development: What Is It & How Can Evaluation Help?

What is ‘Organization Development’?

Organization Development is an intentional set of processes and practices designed to enhance the ability of organizations to meet their goals. It entails “…a process of continuous diagnosis, action planning, implementation and evaluation, with the goal of transferring (or generating) knowledge and skills so that organizations can improve their capacity for solving-problems and managing future change.”(See: Organizational Development Theory) Organization Development deals with a range of features, including organizational climate, organizational culture (i.e., assumptions, values, norms/expectations, patterns of behavior) and organizational strategy. It seeks to strengthen and enhance the long-term “health” and performance of an organization, often by focusing on aligning organizations with their rapidly changing and complex environments through organizational learning, knowledge management and transformation of organizational norms and values.

What is an organization?

At the most abstract level, an organization is a collectivity of people, a social entity, which seeks to achieve specific aims and goals, and typically is characterized by a structure of designated roles, established rules, a system or structure of authority, a process for decision-making, and a division of labor among organizational members. Organizations are composed of a discreet “membership,” that is, a limited population of incumbents or personnel. Organizations exist in, affect, and are affected by, the larger social and economic environment. Formal, large-scale organizations may take the form of businesses, schools, the military, churches, prisons, foundations, and non-profits.

The Relationship Between Organizations and Evaluation

In seeking to achieve goals, organizations often design and implement discreet initiatives, policies, and programs. They mobilize resources to achieve specific ends. Non-profit organizations, for example, implement programs that mobilize resources in the form of activities, services, and products that are intended to improve the lives of program participants/recipients. “A program is a collection of resources in an organization and is geared to accomplish a certain goal or set of goals. Programs are one major aspect of the non-profit’s structure. The typical non-profit organizational structure is built around programs, that is, the non-profit provides certain major services, each of which is usually formalized into a program.”(See: Overview of Non-Profit Program Planning). In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, services, and materials, to positively affect the lives of those they serve.

Although evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations often collect, synthesize, and report information that can be used to improve the broader operation and health of the organization that hosts the program. Program evaluation thus can contribute to organization development, the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency and to enable the organization to achieve its strategic goals.

Brad Rose Consulting Aids Organizational Development

Brad Rose Consulting works at the intersection of evaluation and organization development. While our projects often begin with a focus on discrete initiatives and programs, the questions that drive our evaluation research provide insights into the effectiveness of the organizations that host, design, and fund those programs. Findings from our evaluations often have important implications for the development and sustainability of the entire host organization. This is especially true in the case of small-to-medium sized nonprofit organizations and educational organizations, whose core programs often comprise the bulk of the organization’s structure and raison d’être. Information from our evaluations can be used to clarify the organization’s goals and objectives, to identify key organizational challenges and help to develop ways to address these, and to strengthen the overall effectiveness of the organization’s efforts. Additionally, Brad Rose Consulting’s evaluations offer an ideal opportunity for an organization to reflect on its practices and purposes, to rethink ways to achieve the organization’s mission, and to identify new data-based strategies for enhancing the organization’s long-term viability and well-being. To learn more about our program development practice visit our Program development & Funding page.

Resources

See our previous post: Helpful Resources: Program Evaluation Supports Strategic Planning

Organization Development

Definition of an Organization 

Overview of Nonprofit Program Planning by Carter McNamara

July 2, 2015
02 Jul 2015

Theory of Change – A Map for Achieving Goals

What is a Theory of Change?

A theory of change is a method for planning, monitoring, and evaluating initiatives in the non-profit, philanthropic, and government sectors.  A theory of change articulates and graphically illustrates the assumptions that inform a change initiative, the prospective set of changes the initiative hopes to make, and the logical and chronological order in which causes and anticipated outcomes will occur. Theories of change ask that program planners, supporters, staff, and in some cases, participants, outline the causal pathway between an initiative’s actions and its ultimate goals.  Although a theory of change is like a logic model, a theory of change differs from a logic model in so far as it is a graphic representation of the steps necessary to address a given problem or issue.  A logic model is typically more focused on a specific program’s deployment of resources to produce specified outcomes.

Typically, theories of change include an explanation of how and why anticipated changes will occur, rather than simply mapping the relationship among inputs, outputs, and outcomes.  Consequently, theories of change are explanatory, while logic models are descriptive.  Clark and Anderson argue that logic models “usually start with a program and illustrate its components” while “theories of change (work) best when starting with a goal, before deciding which programmatic approaches are needed.” (For more information on the differences between a logic model and a theory of change see, Clark and Anderson’s “Theories of Change and Logic Models: Telling Them Apart”).

Patricia Rogers writes, “A ‘theory of change’ explains how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a program, a policy, a strategy or an organization” (see UNICEF—Theory of Change).

Why Develop a Theory of Change?

Everyone who is involved in a change process has an implicit idea of how changes come about.  Collaboratively developing a theory of change can help stakeholders to build a shared understanding of the causal steps (and the underlying implicit assumptions) that are prerequisite to achieving a desired goal. A theory of change can illustrate the causal logic in a way that says “if we do ‘x’, then ‘y’ is likely to occur.”   Essentially, theories of change are a diagram of how things should work, the resources needed to achieve a goal or set of goals, and the causal relationships between various factors that lead to the achievement of a goal.

Reasons for developing a theory of change also include:

  • assisting stakeholders to articulate how and why a change initiative will work
  • graphically representing the implicit and often tacit causal mechanisms at work in a service delivery system or programmatic initiative
  • enhancing an understanding of how information can be used to measure or indicate changes (outcomes and impacts)
  • clarifying for stakeholders why things may be working and not working
  • building a consensus about how an initiative should work, so that it reaches its goals

 

Resources:

What is a theory of change?
http://www.theoryofchange.org/what-is-theory-of-change/

Annie E. Casey Foundation: Theory of Change: A Practical Tool For Action, Results, and Learning
http://www.aecf.org/m/resourcedoc/aecf-theoryofchange-2004.pdf

Video: “How to Develop a Theory of Change”
Using a Theory of Change  in Program Design and Planning
http://www.theoryofchange.nl/topic/using-toc-programme-design-and-planning

Video: “DIY—Theory of Change
Theory of Change—Wikipedia
https://en.wikipedia.org/wiki/Theory_of_change

Unicef—Theory of Change
http://www.unicef-irc.org/publications/pdf/brief_2_theoryofchange_eng.pdf

A Community Builder’s Approach to a Theory of Change
http://dmeforpeace.org/sites/default/files/Community%20Builder%27s%20Approach%20to%20Theory%20of%20Change.pdf

Differences between a Logic Model and a Theory of Change
http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf

Theory of Change Portal (Netherlands)
http://www.theoryofchange.nl/

Harvard Family Research Institute—Theory of change
http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluation-methodology/an-introduction-to-theory-of-change

Theory of Change in the Philanthropic Sector
http://www.dochas.ie/Shared/Files/4/Theory_of_Change_Tool_Manual.pdf

March 3, 2015
03 Mar 2015

What is a needs assessment?

How do programs know what they should be doing— which target populations require services, the types of services programs should provide, the amounts of services, which kinds of services will be most effective, etc.?   Needs assessments are the best way to determine the needs of individuals, communities, and other populations.  A needs assessment is a systematic process for identifying and determining such needs.  Like program evaluations, needs assessments draw on a range of social science methods—from surveys and observations, to focus groups and individual interviews.

Needs assessments assume a clear definition of “a need”.  As James Altschuld and Ryan Watkins point out  in New Directions in Evaluation, No. 144, Winter, 2014  “A need, in the simplest sense, is a measurable gap between two conditions: what currently is, and what should be….This requires ascertaining what the circumstances are at a point in time, what is desired in the future, and a comparison of the two.”  Needs assessments don’t just exclusively focus on what is and should be, but also on gathering and synthesizing data about how to narrow the gap between the existing state and the desired state. Needs assessments also prioritize needs so that users of the assessment can address specified needs in a reasonable order, and devote appropriate resources to meeting identified needs.

By gathering data from a range of stakeholders, needs assessments are able to determine the best means to achieve the desired results. To be effective, however, needs assessments must not simply focus on deficits in individuals and communities, but must also explore existing strengths, capacities, and assets.  Too narrow a focus on “what’s missing” can blind researchers and program designers to the existing assets on which effective programming can be built.  Effective needs assessments therefore, ask questions about: 1) on-going needs, 2) current strengths/assets/ capacities, 3) and desired states.

Needs assessments may differ in their design, but regardless of design, most needs assessment follow these phases:

  1. Explore and gather data about the current condition/state of affairs (including existing assets)
  2. Explore/identify desired or optimal condition/state of affairs
  3. Analyze data to understand the difference or “gap” between the current condition and desired condition.
  4. Prioritize identified needs and “gaps.”
  5. With needs (and assets) in mind, design program to address (diminish or eliminate) the gap between existing needs and desired state.

When conducted in a timely and thoughtful way, needs assessments can be of substantial utility in helping programs to effectively deliver services to those who most need them. To find out more about our assessment methods visit our Impact & Assessment reporting page.

Resources:

“Needs Assessment: Trends and a View Toward the Future,” New Directions in Evaluation, No. 144, Winter, 2014 James W. Altschuld and Ryan Watkins (eds.)

Definition of Needs Assessment
https://en.wikipedia.org/wiki/Needs_assessment

Comprehensive Needs Assessment
http://www2.ed.gov/admins/lead/account/compneedsassessment.pdf

Needs Assessment Surveys
http://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/conducting-needs-assessment-surveys/main

Methods for Conducting an Educational Needs Assessment
http://www.cals.uidaho.edu/edcomm/pdf/BUL/BUL0870.pdf

January 19, 2015
19 Jan 2015

Program Evaluation and Organization Development

Brad Rose Consulting, Program Evaluation and Organization DevelopmentWhat are Nonprofit Organizations?

Organizations are social entities that have a collective purpose and that interact with larger environments (the economy, society, other organizations, etc.) Nonprofit organizations are a sub-type of organization that use “surplus revenues” (i.e. revenues beyond those used for operation and sustenance) to achieve desirable social ends, rather than to produce profits or dividends. In essence, nonprofits use their financial resources (provided by individual donations, foundation and government grants, etc.) to ameliorate the lives and conditions of community members.  In the US, there are a range of nonprofit organizations, including hospitals, charities, educational organizations, social welfare organizations, foundations, community organizations, etc. According to the National Center for Charitable Statistics, there are approximately 1.5 million nonprofits in the US.

Organizations vs. Programs

Many nonprofit organizations mobilize resources in the form of organizedprograms that provide activities and products that are designed to improve the lives of program participants. Carter MacNamara writes that, “A program is a collection of resources in an organization and is geared to accomplish a certain goal or set of goals. Programs are one major aspect of the non-profit’s structure. The typical non-profit organizational structure is built around programs, that is, the non-profit provides certain major services, each of which is usually formalized into a program.” (http://literacy.kent.edu/Oasis/grants/overviewprogplan.html) In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, and materials, to positively affect the lives of those they serve.

Program Evaluation Meets Organization Development

In order to assess the effectiveness and efficiency of programs, nonprofits will often conduct program evaluations. Program evaluations are customarily guided by a set of evaluation research questions, e.g., What are the effects of the program on participants? What challenges did participants encounter while in the program?  Did the program make a difference in the lives of those it was intended to serve? Did the program cause the observed changed in program participants? etc.  (for more examples of evaluation questions, see our previous posts “Questions Before Methods”  and “Approaching an Evaluation: 10 Issues to Consider” Although program evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations often collect, synthesize, and report information that can useful in improving the broader operation and health of the organization that hosts the program. Program evaluation thus can contribute to organization development, “the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency and/or to enable the organization to achieve its strategic goals.” (wikipedia)  

In fact, findings from program evaluations often have important implications for the development and sustainability of the entire host organization.  This is especially true in the case of small-to-medium sized nonprofit organizations, whose core programs often comprise the bulk of the organization’s structure and raison d’être. Consequently, information from program evaluations—especially formative evaluations which focus on strengthening program effectiveness—can be used to clarify the organization’s goals and objectives, to identify key organizational challenges and ways to address these, and to strengthen the overall effectiveness of the organization’s efforts.  Additionally, program evaluations can offer an ideal opportunity for an organization to reflect on its practices and purposes, to rethink ways to achieve the organization’s mission, and to identify new data-based strategies for enhancing the organization’s long-term viability and well-being.  Ultimately, program evaluation can, and in many cases should, be an integral component of organization development. If you want to learn more about our methods for program development and evaluation visit our Program development & Funding page.

Resources:

About Nonprofit Organizations:
https://en.wikipedia.org/wiki/Nonprofit_organization

Types of Non Profit Organizations in the US:
http://blogs.findlaw.com/free_enterprise/2009/08/5-types-of-501c3-non-profit-organizations.html

Overview of Nonprofit Program Planning by Carter McNamara:
http://literacy.kent.edu/Oasis/grants/overviewprogplan.html

Basic Guide to Nonprofit Program Design and Marketing:
http://managementhelp.org/programmanagement/nonprofit-programs.htm

What is Organization Development?

https://en.wikipedia.org/wiki/Organization_development

http://www.odportal.com/OD/whatisod.htm

http://www.toolpack.com/a/organizational-development.html

January 8, 2014
08 Jan 2014

Evaluation Serving Community Foundations and Donors

The Importance of Demonstrating to Community Foundation Funders Programs’ Impacts

As donors seek to strengthen and improve the lives of their communities, they are increasingly interested in understanding the effects of their social investments. Consequently, community foundations are now frequently asked to show that the programs and organizations they support are having the effects that donors would like to see. Program evaluations of community foundation-supported programs are an extremely useful way to document the effectiveness of programs. Evaluations can provide critical information about the effects of programs, show donors that their expectations are being met, and provide critical information with which to ensure that critical community needs are being effectively addressed. A recent article in the New Yorker noted, “Measuring the impact of giving is more important to donors aged 21-40 than it was to previous generations.” (New Yorker December, 2013)

Brad Rose Consulting, Inc. has evaluated a wide-range of philanthropic programs, ranging from human services and arts education programs, to youth development and domestic violence prevention programs. We work with foundations to ensure both effective and sensitive program evaluations that show the effects of donor- supported programming. Our evaluations are especially helpful to programs supported by designated funds. While our experience includes working with national-level foundations, our approach to tailored evaluation initiatives is especially geared to helping community-level foundations who are increasingly asked to demonstrate the effectiveness of the programs their donor funds support.

Program Evaluation to Support Planning

Although our evaluations gather data to show the effects of existing programs, our evaluation approach can be very helpful as funders and donors plan for future initiatives. By showing both the achievements and challenges of current programs, our evaluations provide to program planners and funders helpful insight into a variety of program area “best practices”. We also work with foundation grantees to help them develop their own capacities for evaluating the effects of their efforts. By educating grant recipients about the need for, and fundamental elements of, program evaluation, we prepare grantees to better plan and execute programs that demonstrate results.

If you are a community foundation staff member, we would welcome and opportunity to discuss with you the ways that Brad Rose Consulting, Inc. can help you and your grantees to strengthen the programs you support so that they better serve the goals and expectations of your donors. Similarly, if you are an organization that receives foundation support, we would welcome a chance to discuss with you how our program evaluations can help you strengthen your programmatic efforts and help to secure future foundation support. Please feel free to contact me, click here for contact information. Or to learn more about our work with community foundations visit our Community service page.

August 21, 2013
21 Aug 2013

Program Evaluation: Strengthening Schools, Not Just Holding Them Accountable

Most organizations conduct evaluations because they want to determine if they are making a difference in the lives of the people that they serve. Determining program effectiveness, showing the specific effects of programming, and using data to strengthen programs, are all important and laudable reasons for carrying out an evaluation.

In recent years, however “accountability” has become a driving force for many organizations (schools, government agencies, and non-profits) to conduct evaluations.  “Accountability” has become a watchword—especially in the educational and non-profit sectors.  While accountability has its legitimate purposes (i.e., demonstrating to stakeholders that an organization or program is responsible, ethical, committed to achieving its goals, etc.) too often the desire to demonstrate accountability, especially legal compliance,  overshadows the use of evaluations to enhance program effectiveness and strengthen program outcomes.

Brad Rose Consulting, Inc. is committed to evaluations that provide an evidence- based account of a program’s effects.  Equally importantly, however, we are committed to designing and conducting evaluations that help to strengthen a program (and its host organization) so that it can better achieve its desired outcomes.  We work with organizations to objectively find out what’s working and what needs to be strengthened.

When, for example, we work with educators (superintendents, principals, teachers and school staff) and school systems to evaluate their educational programs, we design evaluations that BOTH show program effectiveness AND provide data-based insights that help strengthen future outcomes. We understand that educators need BOTH to know if students are learning AND how to enhance future student achievement.  Such evaluations require not merely collecting static and tiresome data, but implementing evaluations that richly show how educational initiatives can be made more effective.  Often such evaluations transcend the collection merely of student test scores and look at the multiple factors that influence student achievement. We deliberately work to make our evaluations constructive opportunities for strengthening instruction and enhancing student achievement. To learn more about our work in education visit our Higher education & K-12 page.

More information about accountability:

What is “accountability”?

“Accountability” in Education Week

“What Does Accountability Mean in Education?”

“Accountability in Higher Education,” by the Center for Studies in Higher Education

“Why Teacher Evaluation Shouldn’t Rest on Student Test Scores”

“A Better Way to Assess Students and Evaluate Schools”

June 4, 2013
04 Jun 2013

Social Entrepreneurs: How Evaluation Can Make A Difference

In recent years, “social entrepreneur” has become a prominent term in the not-for-profit, foundation, and NGO worlds.  But what exactly is a “social entrepreneur?”  While social entrepreneurs share many of the characteristics ascribed to for-profit entrepreneurs, Roger L. Martin & Sally Osberg observe in their article in the Stanford Innovation Review  Social Entrepreneurship: The Case for Definition that “the social entrepreneur aims for value in the form of large-scale, transformational benefit that accrues either to a significant segment of society or to society at large.” They also note that “social entrepreneurship….is as vital to the progress of societies as is entrepreneurship to the progress of economies, and it merits more rigorous, serious attention than it has attracted so far.”

PBS (http://www.pbs.org/opb/thenewheroes/whatis/  ) notes “A social entrepreneur identifies and solves social problems on a large scale. Just as business entrepreneurs create and transform whole industries, social entrepreneurs act as the change agents for society, seizing opportunities others miss in order to improve systems, invent and disseminate new approaches and advance sustainable solutions that create social value.  Unlike traditional business entrepreneurs, social entrepreneurs primarily seek to generate “social value” rather than profits. And unlike the majority of non-profit organizations, their work is targeted not only towards immediate, small-scale effects, but sweeping, long-term change.”

The Ashoka Foundation  similarly stresses the large-scale effects that social entrepreneurs seek to make. “Social entrepreneurs are individuals with innovative solutions to society’s most pressing social problems. They are ambitious and persistent, tackling major social issues and offering new ideas for wide-scale change. Rather than leaving societal needs to the government or business sectors, social entrepreneurs find what is not working and solve the problem by changing the system, spreading the solution, and persuading entire societies to take new leaps.”  (See  https://www.ashoka.org/social_entrepreneur )

While the above sources highlight the often ambitious, indeed global, goals of social entrepreneurs, most social entrepreneurship, in fact, involves developing and sustaining specific organizations and, in turn, operating discrete programs.  These programmatic efforts can benefit from program evaluations that gather information to show that they are making the differences that their founders hope to make.  Evaluations of social entrepreneur-sponsored initiatives not only provide critical evidence of impact, but equally importantly, provide objective information gathered directly from program recipients and others program stakeholders about the ways that such efforts might be strengthened.  (See our previous post, “Listening to Those Who Matter Most: Beneficiaries.”)  Evaluations of social entrepreneur-sponsored initiatives are especially important because program participants, service recipients, and other beneficiaries are seldom in a position to directly provide feedback to the innovators who are responsible for the programming. Transformative initiatives–no less than smaller scale programs—can substantially benefit from program evaluations. To learn more about our work with Non-Profits visit our Non-Profits page.

See also “Advancing Evaluation Practices in Philanthropy” at The Stanford Innovation Review http://www.ssireview.org/supplement/advancing_evaluation_practices_in_philanthropy

Copyright © 2020 - Brad Rose Consulting