The Pandemic and Nonprofits

A Lifetime of Learning

Pablo Picasso once said, “It takes a long time to become young.” The same may be said about education and the process of becoming educated. While we often associate formal education with youth and early adulthood, the fact is that education is an increasingly recognized lifelong endeavor that occurs far beyond the confines of early adulthood and traditional educational institutions.

In a recent article “Lifetime Learner” by John Hagel III, John Seely Brown, Roy Mathew, Maggie Wooll & Wendy Tsu, The Atlantic the authors discuss the emergence of a rich and ever-expanding “ecosystem” of organizations and institutions that have arisen to serve the unmet educational needs and expectations of learners who are not enrolled in formal, traditional educational institutions (e.g. community colleges, colleges, and universities). “This ecosystem of semi-structured, unorthodox learning providers is emerging at “the edges” of the current postsecondary world, with innovations that challenge the structure and even the existence of traditional education institutions.”

Hagel III, et al. argue that economic forces together with emerging technologies are enabling learners to do an “end run” around traditional educational providers and to gain access to knowledge and information in new venues. The growing availability of, and access to, MOOCs (Massive Online Open Courses), Youtube, Open Educational Resources, and other online learning platforms enable more and more learners to advance their learning and career goals outside the purview of traditional post-secondary institutions.

While the availability of alternative, lifelong educational resources is helping some non-traditional students to advance their educational goals, it is also having an effect on traditional post-secondary institutions. Hagel III, Seely Brown, Wooll and Tsu, argue that, “The educational institutions that succeed and remain relevant in the future …will likely be those that foster a learning environment that reflects the networked ecosystem and (that will become) meaningful and relevant to the lifelong learner. This means providing learning opportunities that match the learner’s current development and stage of life.”  The authors site as examples, community colleges that are now experimenting with “stackable” credentials that provide short-term skills and employment value, while enabling students to return over time and assemble a coherent curriculum that helps them progress toward career and personal goals” and “some universities (that) have started to look at the examples coming from both the edges of education and areas such as gaming and media to imagine and conduct experiments in what a future learning environment could look like.”

The authors say that in the future colleges and universities will benefit from considering such things as:

1. Providing the facilities and locations for a variety of learning experiences, many of which will depend external sources for content
2. Aggregating knowledge resources and connecting these resources with appropriate learners rather than acting as sole “vendors” of knowledge
3. Acting as a lifelong “agents” for learners by helping learners to navigate a world of exponential change and an abundance of information

While these goals are ambitious, they highlight the remarkably changing terrain in continuing education. Educational “consumers” are increasingly likely to seek inexpensive and more accessible pathways to knowledge. As the authors point out, individuals’ lifelong learning needs are likely to continue to increase, so correspondingly, the pressures on traditional post-secondary education are likely to grow. Whether learners’ needs are more effectively addressed by re-orienting traditional post-secondary institutions or by the patchwork “ecosystem” of semi-structured, unorthodox learning-providers who inhabit what the authors of “Lifetime Learner” term “the edges” of the postsecondary world, is difficult to predict.

Resources:

Lifelong learning, Wikipedia

“Lifetime Learner” by John Hagel III, John Seely Brown, Roy Mathew, Maggie Wooll & Wendy Tsu, The Atlantic

“The Third Education Revolution: Schools are moving toward a model of continuous, lifelong learning in order to meet the needs of today’s economy” by Jeffrey Selingo, The Atlantic, Mar 22, 2018

Are There Any Questions?

Asking questions is a critical aspect of learning. We’ve previously written about the importance of questions in our blog post “Evaluation Research Interviews: Just Like Good Conversations.” In a recent article, “The Surprising Power of Questions,” which appears in the Harvard Business Review, May-June, 2018, authors Alison Wood Brooks and Leslie K. John offer suggestions for asking better questions.

As Brooks and John report, we often don’t ask enough questions during our conversations. Too often we talk rather than listen. Brooks and John, however, note that recent research shows that by asking good questions and genuinely listening to the answers, we are more likely to achieve both genuine information exchange and effective self-presentation. “Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding.”

Although asking more questions in our conversations is important, the authors show that asking follow-up questions is critical. Follow-up questions “…signal to your conversation partner that you are listening, care, and want to know more. People interacting with a partner who asks lots of follow-up questions tend to feel respected and heard.”

Another critical component of a question-asking is to be sure that we ask open-ended questions, not simply categorial (yes/no) questions. “Open-ended questions …can be particularly useful in uncovering information or learning something new. Indeed, they are wellsprings of innovation—which is often the result of finding the hidden, unexpected answer that no one has thought of before.”

Asking effective questions depends, of course, on the purpose and context of conversations. That said, it is vital to ask questions in an appropriate sequence. Counterintuitively, asking tougher questions first, and leaving easier questions until later “…can make your conversational partner more willing to open up.” On the other hand, asking tough questions too early in the conversation, can seem intrusive and sometimes offensive. If the ultimate goal of the conversation is to build a strong relationship with your interlocutor, especially with someone who you don’t know, or don’t know well, it may be better opening with less sensitive questions and escalate slowly. Tone and attitude are also important: “People are more forthcoming when you ask questions in a casual way, rather than in a buttoned-up, official tone.”

While question-asking is a necessary component of learning, the authors remind us that “The wellspring of all questions is wonder and curiosity and a capacity for delight. We pose and respond to queries in the belief that the magic of a conversation will produce a whole that is greater than the sum of its parts. Sustained personal engagement and motivation—in our lives as well as our work—require that we are always mindful of the transformative joy of asking and answering questions.”

Resources:

“The Surprising Power of Questions,” Alison Wood Brooks and Leslie K. John. Harvard Business Review, May–June 2018 (pp.60–67)

“Using Qualitative Interviews in Program Evaluations”

Evaluation Reporting:  True Stories, Well Told

In a recent, themed issue devoted to the topic of validity in program evaluation, the journal New Directions in Evaluation (No. 142, Summer, 2014) revisited and commented on Ernest House’s influential 1980 book, Evaluating with Validity. House then argued that validity in evaluation must not just be limited to classic scientific conceptions of the valid (i.e. empirically describing things as they are), but that it must also include an expanded dimension of argumentative validity, in which an evaluation “must be true, coherent, and just.”  Paying particular attention to social context, House argued that “there is more to validity than getting the facts right.” He wrote that “…the validity of an evaluation depends upon whether the evaluation is true, credible, and normatively correct.” House ultimately argued that evaluations must make compelling and persuasive arguments about what is true (about a program) and thereby bring “truth, beauty, and justice” to the evaluation enterprise.

In the same issue of New Directions in Evaluation, in her essay, “How ‘Beauty’ Can Bring Truth and Justice to Life,”  E. Jane Davidson argues that the process of creating a clear, compelling, and coherent evaluative story ( i.e., a “beautiful” narrative account) is the key to unlocking “validity (truth) and fairness (justice).” To briefly summarize, Davidson argues that a coherent evaluation story weaves together quantitate evidence, qualitative evidence, and clear evaluative reasoning to produce an account of what happened and the value of what happened. She says that an effective evaluation—one that is truly accessible, assumption-unearthing, and values-explicit—enables evaluators to “arrive at robust conclusions about not just what has happened, but how good, valuable, and important it is.” (P.31)

House’s book and Davidson’s essay highlight how effective evaluations—ones that allow us to clearly see and understand what has happened in a program— rely on strong narrative accounts that tell a coherent  and revealing story. Evaluations are not just tables and data—although these are necessary parts of any evaluation narrative—they are true, compelling, and fact-based stories about what happened, why things happened the way they did, and what the value (and meaning) is of the things that happened.

Davidson writes:

“When I reflect on what has improved the quality of my own work in recent years, it has been a relentless push toward succinctness and crystal clarity while grappling with some quite complex, and difficult material. For me this means striving to produce simple direct and clear answers to evaluation questions and being utterly transparent in the reasoning I have used to get to those conclusions.” (p.39)

Davidson further observes that evaluation reports often are often plagued by confusing, long-winded, and academic jargon, that make them not only difficult to read, but obfuscate the often muddled and ill-reasoned thinking behind the evaluation process itself.  She argues that evaluation reporting must be clear, accessible, and simple—which does not mean that reports need to be simplistic, but that they must be coherent and comprehensible. I am reminded of a statement by the philosopher John Searle, ‘If you can’t say it clearly, you don’t understand it yourself’.

Reflecting on Davidson’s article, I realize that the best evaluation reports are the product of well thought-out and effectively conducted evaluation research, presented in a clear and cogent way. The findings from such research may be complex, but they need not be obscure or enigmatic. On the contrary, clear evaluation reports must be true stories, well told. To learn more about our evaluation reporting visit our Impact & Assessment reporting page.

Focus Groups

Pioneered by market researchers and mid-20th century sociologists, focus groups are a qualitative research method that involves small groups of people in guided discussions about their attitudes, beliefs, experiences, and opinions about a selected topic or issue.  Often used by marketers who obtain feedback from consumers about a product or service, focus groups have also become an effective and widely recognized social science research tool that enables researchers to explore participants’ views, and to reveal rich data that often remain under-reported by other kinds of data collection strategies (e.g., surveys, questionnaires, etc. ).

Organized around a set of guiding questions, focus groups typically are composed of 6-10 people and a moderator who poses open-ended questions that allow participants to address questions. Focus groups usually include people who are somewhat similar in characteristics or social roles.  Participants are selected for their knowledge, reflectiveness, and willingness to engage topics or questions.  Ideally—although not always possible—it is best to involve participants who don’t previously know one another.

Focus group conversations enable participants to offer observations, define issues, pose and refine questions, and create informative debate/discussions.  Focus group moderators must: be attentive, pose useful and creative questions, create a welcoming and non-judgmental atmosphere, be sensitive to non-verbal cues and the emotional tenor of participants.   Typically, focus group sessions are recorded or videoed so that researchers can later transcribe and analyze participants’ comments.  Often an assistant moderator will take notes during the focus group conversation.

Focus groups have advantages over other date collection methods.  They often employ group dynamics that help to reveal information that would not emerge from an individual interview or survey: they produce relatively quick, low cost data (they produce an ‘economy of scale’ as compared to individual interviews); allow the moderator to pose appropriate and responsive follow-up questions;  enable the moderator to observe non-verbal data; and often produce greater and richer data than a questionnaire or survey.

Focus groups also can have some disadvantages, especially if not conducted by an experienced and skilled moderator:  Depending upon their composition, focus groups are not necessarily representative of the general population; respondents may feel social pressure to endorse other group members’ opinions or refrain from voicing their own; group discussions require effective “steering” so that key questions are answered, and participants don’t stray from the questions/topic.

Focus groups are often used in program evaluations.  I have had extensive experience conducting focus groups with a wide-range of constituencies.  During my 20 years of experience as a program evaluator, I’ve moderated focus groups composed of:  homeless persons; disadvantaged youth; university pr ofessors and administrators; k-12 teachers; k-12 and university students, corporate managers; and hospital administrators.  In each of these groups I’ve found that it’s been beneficial to: have a non-judgmental attitude, be genuinely curious; exercise a gentle guidance; and respect the opinions, beliefs, and experiences of each focus group member.   A sense of humor can also be extremely helpful. (See our previous post: “Interpersonal Skills Enhance Program Evaluation,”  Also “Listening to Those Who Matter Most, the Beneficiaries” ) Or if you want to learn more about our qualitative approaches visit our Data collection & Outcome measurement page.

Resources:

About focus groups:

http://sociology.about.com/od/Research-Methods/a/Focus-Groups.htm

About focus groups:

http://www.cse.lehigh.edu/~glennb/mm/FocusGroups.htm

How focus groups work:

http://money.howstuffworks.com/business-communications/how-focus-groups-work1.htm

Focus group interviewing:

http://www.tc.umn.edu/~rkrueger/focus.html

‘Focus groups’ at Wikipedia

https://en.wikipedia.org/wiki/Focus_group

Understanding How Programs Work: Using Logic Models to “Map” Cause and Effect

A logic model is a schematic representation of the elements of a program and the program’s resulting effects.  A logic model (also known, as a “theory of change”) is a useful tool for understanding the way a program intends to produce the outcomes (i.e. changes) it hopes to produce.  Logic models typically consist of a flowchart schematic that shows the logical connection between a program’s “inputs” (i.e. invested resources), “outputs” (program activities and actions), “short-term outcomes” (changes), “medium-term outcomes” (changes), and “long range impacts” (changes).

When developing a logic model many evaluators and program staff rightly focus on inputs, outputs, and program outcomes (the core of the program).  However, it is critical to also include in the logic model the implicit assumptions that underlie the program’s operation, the needs that the program aspires to address, and the program’s environment, or context.  Assumptions, needs, and context are crucial factors in understanding how the program does what it intends to do.  Ultimately these are crucial to understanding the causal mechanisms that produce the intended changes of any program.

Without clearly understanding the causal mechanisms at work in a program, program staff may work ineffectively, placing emphasis on the wrong or inefective activities—and ultimately fail to correctly address the challenges the program intends to address.   Similarly, without a clear understanding of the causal mechanisms that enable the program to achieve its outcomes, the program evaluation may not measure the proper outcomes or fail to see the changes the program, in fact, brings about.

Brad Rose Consulting, Inc. works with clients to develop simple, yet robust, logic models that explicitly document the causal mechanisms that are at work in a program.  By discussing, and explicitly identifying the often implicit causal assumptions,  as well as highlighting the needs for the program and the social context of a program, we not only ensure that the evaluation is properly designed and executed, we also help program implementers to ensure that they are activating the causal processes/mechanisms that yield the changes that the program strives to achieve.

Other Resources:

Read Brad’s current whitepaper “Logic Modeling”

“Monitoring and Evaluation: Some Tools Methods and Approaches, The World Bank.”

“The Logic Model Development Guide,” W.K. Kellogg Foundation.

Logic Model Resources at the University of Wisconsin

A Bibliography for Program Logic Models/Logframe Analysis

Transforming “Data” Into Knowledge

In his recent article in the New York Times, “What Data Can’t Do” (February 18, 2013, visit here ), David Brooks discusses some of the limits of “data.”

Brooks writes that we now live in a world that is saturated with gargantuan data collection capabilities, and that today’s powerful computers are able to handle huge data sets which “can now make sense of mind-bogglingly complex situations.” Despite these analytical capacities, there are a number of things that data can’t do very well. Brooks remarks that data is unable to fully understand the social world; often fails to integrate and deal with the quality (vs. quantity) of social interactions; and struggles to make sense of the “context,” i.e., the real environments, in which human decisions and human interactions are inevitably embedded. (See our earlier blog post “Context Is Critical”.)

Brooks insightfully notes that data often “obscures values,” by which he means that data often conceals the implicit assumptions, perspectives, and theories on which they are based. “Data is never ‘raw,’ it’s always structured according to somebody’s predispositions and values.” Data is always a selection of information. What counts as data depends upon what kinds of information the researcher values and thinks is important.

Program evaluations necessarily depend on the collection and analysis of data because data constitutes important measures and indicators of a program’s operation and results. While evaluations require data, it is important to note that data alone while, necessary, is insufficient for telling the complete story about a program and its effects. To get at the truth of a program, it is necessary to 1) discuss both the benefits and limitations of what constitutes “the data”—to understand what counts as evidence; 2) to use multiple kinds of data—both quantitative and qualitative; and 3) to employ experience- based judgment when interpreting the meaning of data.

Brad Rose Consulting, Inc., addresses the limitations pointed out by David Brooks, by working with clients and program stakeholders to identify what counts as “data,” and by collecting and analyzing multiple forms of data. We typically use a multi-method evaluation strategy, one which relies on both quantitative and qualitative measures. Most importantly, we bring to each evaluation project our experience-based judgment when interpreting the meaning of data, because we know that to fully understand what a program achieves (or doesn’t achieve), evaluators need robust experience so that they can transform mere information into genuine, useable knowledge. To learn about our diverse evaluation methods visit our Data collection & Outcome measurement page.

Sample Evaluation Report – MMUF

The Mellon Mays Undergraduate Fellowship is an initiative to to reduce the serious under-representation of minorities in the faculties of higher education. MMUF commissioned Brad Rose Consulting to do an evaluation of their success over the past 10 years. Here are Brad Rose’s findings. The report is a fantastic example of what you can expect from a commissioned evaluation.

To learn more about our work in education visit our Higher education & K-12 page.