Talking Back to Foundations

Philanthropies play an important role in contemporary society. They are, by their very nature, focused on supporting the public good and human welfare. While philanthropies, each year, channel vast sums to the achievement of laudable goals, their social and economic power has raised questions about their unalloyed reputation for ‘doing good’. See, for example, our previous blogpost, “Philanthrocapitalism?”

In an attempt to provide a voice to those who have worked with foundations, a new website, GrantAdvisor, offers grantees of foundations a safe way to anonymously give feedback about the grantmaking/receiving process. GrantAdvisor also offers foundations an opportunity to learn about grantees’ experience working with foundation staff.

GrantAdvisor effectively serves as a kind “Yelp” to those in the non-profit community. Here for example is a link to grantees experience working with the Wal-Mart Foundation.

You can view GrantAdvisor here to learn more about how the site works and to review the reviews of a number of important national and local foundations.

Resources:

“A Place Where You Can Speak Your Mind to That Foundation,” Amy Costello. Non Profit Quarterly

“Benchmarking Foundation Evaluation Practices,” Center for Effective Philanthropy

“Helping Community Foundations Strengthen Grantees’ Effectiveness”

Power in Organizations

Most of us spend a good portion of our lives in organizations or indirectly relating to organizations (businesses, non-profits, civic and legal organizations, religious organizations, military and criminal justice organizations, etc.). One might say that in the modern world, we “live in” an environment composed largely of organizations. (See our previous blogposts “Organization Development: What Is It & How Can Evaluation Help?”  and “What’s the Difference? 10 Things You Should Know About Organizations vs. Programs”)

Organizations contain, utilize, and deploy various kinds of social power. Such power is the capacity of individuals and groups to affect, control, or influence outcomes (i.e. changes). Power doesn’t exist in isolation, but in relationships to other individuals and/or groups. If we want to accomplish goals at work—whether these goals are about producing widgets, or making the world a better place—we need to draw on and negotiate various kinds of formal and informal power. Sometimes it may be useful to think of various resources as sources of power. Tangible resources include, money, machinery, physical infrastructure, etc. Less tangible, but no less important, resources may include, authority, social status/prestige, social networks, individuals’ intelligence, professional experience, even social attractiveness and charisma. Both tangible and intangible resources are used as sources of power with which organizations achieve objectives and goals.

In her article, “Types of Powers in Organizations,” Diana Dahl summarizes 7 types of power in organizations. These include:

1. Coercive Power— a person or group is able to punish others for not following orders has coercive power.
2. Connection Power— connection power is gained by knowing and being listened to by influential people. Increasing connections and mastering political networking lead to a greater potential for connection power.
3. Reward Power— the ability to give rewards to other employees. Rewards are not always monetary, such as improved work hours and words of praise.
4. Legitimate Power (also known as legitimate authority)— when employees believe a person can give orders based on his position within the organization, such as when a manager orders staff members to complete a task and they comply because the orders came from their superior.
5. Referent Power— people who are liked, respected, and are viewed by other employees as worth emulating. Supervisors who lead by example, treat employees with respect, seek their collaboration and gain the trust of their employees possess referent power.
6. Informational Power— access to valued information. This power can be quickly fleeting because once the needed information is shared, the person’s power is gone.
7. Expert Power— the greater a person’s knowledge or specialized skill set, the greater her potential for expert power.

People in organizations must get things accomplished. Having a clear idea of what constitutes organizational power and the kinds of resources that need to mobilized to reach goals, may help us to better navigate what are often complicated and contentious organizations.

Resources

Power in Organizations: Structures, Processes and Outcomes, by Richard Hall and Pamela Tolbert Pearson/Prentice Hall, 2005, 9th edition

“Types of Powers in Organizations,” by Diana Dahl ; Updated September 26, 2017

Links to and summaries of “Theories of Organizational Power” at bankofinfo.com/theories-of-organizational-power

“Power and Politics in Organizational Life,” Abraham Zaleznik, Harvard Business Review

“Power in Organizations: A Way of Thinking About What You’ve Got, and How to Use It,” Roelf Woldring

Summary of tactics to gain organizational power 

Are There Any Questions?

Asking questions is a critical aspect of learning. We’ve previously written about the importance of questions in our blog post “Evaluation Research Interviews: Just Like Good Conversations.” In a recent article, “The Surprising Power of Questions,” which appears in the Harvard Business Review, May-June, 2018, authors Alison Wood Brooks and Leslie K. John offer suggestions for asking better questions.

As Brooks and John report, we often don’t ask enough questions during our conversations. Too often we talk rather than listen. Brooks and John, however, note that recent research shows that by asking good questions and genuinely listening to the answers, we are more likely to achieve both genuine information exchange and effective self-presentation. “Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding.”

Although asking more questions in our conversations is important, the authors show that asking follow-up questions is critical. Follow-up questions “…signal to your conversation partner that you are listening, care, and want to know more. People interacting with a partner who asks lots of follow-up questions tend to feel respected and heard.”

Another critical component of a question-asking is to be sure that we ask open-ended questions, not simply categorial (yes/no) questions. “Open-ended questions …can be particularly useful in uncovering information or learning something new. Indeed, they are wellsprings of innovation—which is often the result of finding the hidden, unexpected answer that no one has thought of before.”

Asking effective questions depends, of course, on the purpose and context of conversations. That said, it is vital to ask questions in an appropriate sequence. Counterintuitively, asking tougher questions first, and leaving easier questions until later “…can make your conversational partner more willing to open up.” On the other hand, asking tough questions too early in the conversation, can seem intrusive and sometimes offensive. If the ultimate goal of the conversation is to build a strong relationship with your interlocutor, especially with someone who you don’t know, or don’t know well, it may be better opening with less sensitive questions and escalate slowly. Tone and attitude are also important: “People are more forthcoming when you ask questions in a casual way, rather than in a buttoned-up, official tone.”

While question-asking is a necessary component of learning, the authors remind us that “The wellspring of all questions is wonder and curiosity and a capacity for delight. We pose and respond to queries in the belief that the magic of a conversation will produce a whole that is greater than the sum of its parts. Sustained personal engagement and motivation—in our lives as well as our work—require that we are always mindful of the transformative joy of asking and answering questions.”

Resources:

“The Surprising Power of Questions,” Alison Wood Brooks and Leslie K. John. Harvard Business Review, May–June 2018 (pp.60–67)

“Using Qualitative Interviews in Program Evaluations”

The Tyranny of Metrics

In a recent article “Against Metrics: How Measuring Performance by Numbers Backfires,” Jerry Z Muller argues that companies, educational institutions, government agencies, and philanthropies are now in the grip of what he calls “metric fixation,” “…the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardized data (metrics).”

In this brief and important article, Muller critiques the growing phenomenon of paying employees for performance. He points out that such schemes often lead to a narrowing measure of what is desirable for the organization, leads members of an organization to “game the system”, often undermines organizations ability to think more broadly about their purposes, and most importantly, impedes innovation.

Looking at the unintended outcomes of metric fixation, he writes:

“When reward is tied to measured performance, metric fixation invites just this sort of gaming. But metric fixation also leads to a variety of more subtle unintended negative consequences. These include goal displacement, which comes in many varieties: when performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a pay rise or raising the stock price at the time that stock options are vested), people focus on satisfying those measures – often at the expense of other, more important organizational goals that are not measured. The best-known example is ‘teaching to the test’, a widespread phenomenon that has distorted primary and secondary education in the United States since the adoption of the No Child Left Behind Act of 2001.”

Pay for performance schemes, however, are not alone in eliciting a narrowing of goals, and a tendency to game the system. Metric fixation (or what I term the “tyranny of measurement”) can be a risk for a range of non-profit organizations and educational institutions who often feel that demands for accountability can be addressed by merely counting the number of participants who receive services, or the number of students who score well on reading tests. While it is important to have clear goals, and to be able to indicate if these goals are met, organizations, in their rush to address demands from funders and other stakeholders for accountability, must be careful not to reduce their goals—indeed their organizations’ vision— to only a few countable variables. “What can and does get measured is not always worth measuring, may not be what we really want to know, and may draw effort away from the things we care about” (Muller). As Albert Einstein observed, “Not everything that counts can be counted, and not everything that can be counted, counts.”

Resources:

“Against Metrics: How Measuring Performance by Numbers Backfires,” Aeon, April 24, 2018

The Tyranny of Metrics, Jerry Z. Muller, Princeton University Press, 2018

Transparent as a Jellyfish? Why Privacy is Important

Two Cheers for Bureaucracy in Education

Evaluation in Complex and Evolving Environments

Program evaluations seldom occur in stable, scientifically controlled environments. Often programs are implemented in complex and rapidly evolving settings that make traditional evaluation research approaches—which depend upon the stability of the “treatment” and the designation of predetermined outcomes—difficult to utilize.

Michael Quinn Patton, one of the originators of Developmental Evaluation, says that “Developmental evaluation processes include asking evaluative questions and gathering information to provide feedback and support developmental decision-making and course corrections along the emergent path. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation, and intentional change. The evaluator’s primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and to facilitate data-based assessments and decision-making in the unfolding and developmental processes of innovation.”

In their paper, “A Practitioners Guide to Developmental Evaluation,” Dozios and her colleagues note, “Developmental Evaluation differs from traditional forms of evaluation in several key ways:”

• The primary focus is on adaptive learning rather than accountability to an external authority.
• The purpose is to provide real-time feedback and generate learnings to inform development.
• The evaluator is embedded in the initiative as a member of the team.
• The DE role extends well beyond data collection and analysis; the evaluator actively intervenes to shape the course of development, helping to inform decision-making and facilitate learning.
• The evaluation is designed to capture system dynamics and surface innovative strategies and ideas.
• The approach is flexible, with new measures and monitoring mechanisms evolving as understanding of the situation deepens and the initiative’s goals emerge

Development evaluation is especially useful for social innovators who often find themselves inventing the program as it is implemented, and who often don’t have a stable and unchanging set of anticipated outcomes. Following Patton, Dozois, Langlois, and Blanchet-Cohen observe that Developmental Evaluation is especially well suited to situations that are:

• Highly emergent and volatile (e.g., the environment is always changing)
• Difficult to plan or predict because the variables are interdependent and non-linear
• Socially complex— requiring collaboration among stakeholders from different organizations, systems, and/or sectors
• Innovative, requiring real-time learning and development

Developmental Evaluation, however, is increasingly appropriate for use in the non-profit world, especially where the stability of programs’ key components including the program’s core treatment and eventual, often evolving, outcomes, are not as certain or firm as program designers might wish.

Brad Rose Consulting is experienced in working with program’s whose environments are volatile and whose iterative program designs are necessarily flexible. We are adept at collecting data that can inform the on-going evolution of a program, and have 20+ years of providing meaningful data to program designers and implementers that help them to adjust to rapidly changing and highly variable environments.

Resources:

A Practitioner’s Guide to Developmental Evaluation, Elizabeth Dozois, Marc Langlois, and Natasha Blanchet-Cohen

Michael Quinn Patton on Developmental Evaluation

Developmental Evaluation

The Case for Developmental Evaluation

The Importance of Understanding Stakeholders

Every program evaluation is conducted in a context in which there are parties (persons, organizations, etc.) who have an interest, or a “stake,” in the operation and success of the program. In the corporate world, a “stakeholder” is any member of the “groups without whose support the organization would cease to exist.” (see Corporate Stakeholder). More recently, the idea of “a stakeholder” has been broadened to include “any group or individual who is affected by, or who can affect the achievement of, an organization’s objectives.” (The Stakeholder Theory Summary.)
Indeed, in the not-for-profit world, stakeholders may include an array of persons and organizations including funders, community members, program participants, family members, volunteers, staff, government agencies, and the broader public.

Stakeholders in non-profits usually fall into one of three categories of legal statuses:

1. Constitutional stakeholders such as board members or trustees of the non-profit organization
2. Contractual stakeholders, including paid staff, or any business, group or individual that has a formal relationship with the organization.
3. Third-party stakeholders including all the people and groups that may be affected by what the organization does. That includes businesses, the local government, and the citizens who live in the community. (See What is a Stakeholder of a Non-profit.)

Nonprofit stakeholders may range from those who support an organization, to those who oppose an organization. Stakeholder can include advocates, supporters, critics, competitors, and opponents. In its analysis of stakeholders in policy change efforts, the World Bank uses the categories of “promoter,” “defender,” “latents”, and “apathetics.” (See What is a Stakeholder Analysis.)

Conducting a stakeholder analysis is very useful for both evaluation and strategic planning efforts. Identifying various stakeholders’ interests in an organization’s mission and programming can help non-profit leaders and staff to be sure that their efforts and initiatives are achieving desired goals. They can also be useful in ensuring that the needs of those served are being directly met. For both evaluation and strategic planning purposes, a stakeholder analysis is an important process for achieving a shared understanding of each stakeholder’s specific interest in, and relevance to, the work of the non-profit or educational organization.

Brad Rose Consulting has developed a unique approach to stakeholder analysis, one that can be extremely useful as organizations examine the purposes, goals, specific activities, and desired outcomes of their work. We often work with organizations to implement stakeholder analyses. These analyses are helpful both in identifying where an organization is, at any given point in time, and for identifying where it wants to go in the future. You can see our basic stakeholder analysis form here.

Resources

Corporate Stakeholder

Stakeholder Theory

Summary of Stakeholder Theory

Stakeholders of a Typical Non-Profit Organization

What is a Stakeholder of a Non-profit

A Brief Guide to Evaluation (Part 2): Identifying Evaluation Stakeholders

When you and your colleagues consider conducting an evaluation, it is critical that you be clear about who the evaluation is for. Your organization is likely to have a number of stakeholders who are interested in learning about whether the programmatic work is achieving its desired results (i.e. outcomes). Some programs are funded by a single funder, others may have multiple funders. Funders are, of course, a very important audience for evaluation findings. They will want to know whether their investment is producing results. However, you will want to consider additional stakeholders as well (see the list below). When you are considering an evaluation—whether it will be conducted by an internal or external evaluator—you will benefit from considering who will be the audience for evaluation findings?

• Who are the major stakeholders or interested parties for evaluation results?
• Who wants/needs to know?
• Who will benefit from knowing about the effectiveness of your program’s efforts?
• Make a list of key stakeholders and what they might want to know.

• Current funders
• Potential funders
• Community stakeholders
• Advocacy organizations
• Government agencies
• Colleagues/Internal organizational constituencies
• Peer organizations
• Program managers and program staff
• The “field”/general public
• Political allies
• Voters
• Competitors

Evidence-Based Studies

“Evidence-based” – What is it?
“Evidenced-based” has become a common adjectival term for identifying and endorsing the effectiveness of various programs and practices in fields ranging from medicine to education, from psychology to nursing, from criminal justice to psychology. The motivation for marshalling objective evidence in order to guide practices and policies in these diverse fields has been the result of the growing recognition that professional practices—whether they be doctoring or teaching, social work, or nursing—need to be based on something more sound than custom/tradition, practitioners’ habit, professional culture, received wisdom, and hearsay.

What does “evidenced-based” mean?
While definitions of “evidence-based” vary, the most common characteristics of evidence-based research include: objective, empirical research that is valid and replicable, whose findings are based on a strong theoretical foundation, and include high quality data and data collection procedures. The most common definition of Evidence-Based Practice (EBP) is drawn from Dr. David Sackett’s, original (1996) definition of “evidence-based” practices in medicine, i.e., “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research” (Sackett D, 1996). This definition was subsequently amended to, “a systematic approach to clinical problem solving which allows the integration of the best available research evidence with clinical expertise and patient values” (Sackett DL, Strauss SE, Richardson WS, et al. Evidence-based medicine: how to practice and teach EBM. London: Churchill-Livingstone,2000). (See, “Definition of Evidenced-based Medicine”).

An evidenced-based program, whether it be in youth development or education, is comprised of a set of coordinated services/activities that demonstrate effectiveness, as such effectiveness has been established by sound research, preferably, scientifically based research. (See, “Introduction to Evidence-Based Practice“).

In education, evidence-based practices are those practices that are based on sound research that shows that desired outcomes follow from the employment of such practices.  “Evidence-based education is a paradigm by which education stakeholders use empirical evidence to make informed decisions about education interventions (policies, practices, and programs). ‘Evidence-based’ decision making is emphasized over ‘opinion-based’ decision making.” Additionally, “the concept behind evidence-based approaches is that education interventions should be evaluated to prove whether they work, and the results should be fed back to influence practice. Research is connected to day-to-day practice, and individualistic and personal approaches give way to testing and scientific rigor.” (See, “What is Evidence-Based Education?“).

Of course there are different kinds of evidence that can be used to show that practices, programs, and policies are effective.  In a subsequent blog post I will discuss the range of evidence-based studies—from individual case studies and quasi experimental designs, to randomized controlled trials (RCT).  The quality of the evidence as well as the quality of the study in which such evidence appears is a critical factor in deciding whether the practice or program is not just “evidence-based,” but in fact, effective. To learn more about our data collection and measurement click here.

Resources

Evidenced-based practice

What are Evidence-Based Interventions (EBI)?

Scientific Research and Evidence-Based Practice

Evidence-based medicine, Florida State College of Medicine

Evidenced-based studies in education

U.S Department of Education, “What Works” Clearinghouse

Defining Evidence-Based Programs

Child and Family Services

Linking Research with Practice in Youth Development – What Works and How Do We Know

Scientifically Based Research vs. Evidence-Based Practices and Instruction

How to Evaluate Evidence-Based or Research-Based Interventions

Issues in Defining and Applying Evidence-Based Practices Criteria for Treatment of Criminal-Justice Involved Clients

Can Randomized Trials Answer the Question of What Works?

Top Evaluation Resources from Brad Rose Consulting, 2015

With the end of the year fast approaching, this is good time to review the array of online resources that we’ve shared in recent months with non-profit leaders, foundation staff, educators, and fellow program evaluators. The links below represent the most popular blog posts of 2015. If you haven’t yet had a chance to look at one or more of these, please feel free to do so. You’re also welcome to share these links with colleagues and to keep these links on hand for future reference.

Approaching an Evaluation: 10 Issues to Consider

Program Evaluation Essentials for Non Evaluators

Theory of Change: A Map for Achieving Goals

Program Evaluation vs. Social Research

Understanding Different Types of Program Evaluation

4 Advantages of an External Evaluator

A Sample Evaluation Report

Integrity and Objectivity; Critical Components of Program Evaluation

Using A Logic Model

Questions Before Methods

There are also a number of additional program evaluation resources on the site.

We are looking forward to offering more thoughts and additional evaluation resources in 2016!