Program evaluation services that increase program value for grantors, grantees, and communities. x

Tag Archive for: developmental evaluation

April 21, 2020
21 Apr 2020

Developmental Evaluation in Tumultuous Times, Tumultuous Environments

The current health crisis is already having a powerful effect on non-profit organizations, many of whom had been economically challenged even before the onset of the COVID-19 pandemic. (See “A New Mission for Nonprofits During the Outbreak: Survival” by David Streitfeld, New York Times, March 27, 2020) Despite economic challenges, and as the immediate health crisis develops, non-profits will need information, including accurate and robust evaluation and monitoring information, even more than ever.

Under conditions of uncertainty and tumultuous social-environmental environments, and rapid adaptation, non-profits will benefit from information gathered by flexible and adaptable evaluation approaches like Developmental Evaluation. “Developmental evaluation (DE) is especially appropriate for…organizations in dynamic and complex environments where participants, conditions, interventions, and context are turbulent, pathways for achieving desired outcomes are uncertain, and conflicts about what to do are high. DE supports reality-testing, innovation, and adaptation in complex dynamic systems where relationships among critical elements are nonlinear and emergent. Evaluation use in such environments focuses on continuous and ongoing adaptation, intensive reflective practice, and rapid, real-time feedback.”
As Michael Quinnn Patton has recently pointed out, “All evaluators must now become developmental evaluators, capable of adapting to complex dynamics systems, preparing for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for emergence of the unexpected. This is the current context around the world in general and this is the world in which evaluation will exist for the foreseeable future.”

Developmental Evaluation, the kind of evaluation approach Brad Rose Consulting has employed for many years, is extremely well-suited to serve the evaluation and information needs of non-profits, educational institutions, and foundations. For more information about our approach, please see our previous article “Developmental Evaluation: Evaluating Programs in the Real World’s Complex and Unpredictable Environment” and “Evaluation in Complex and Evolving Environments” .

Resources:

January 3, 2018
03 Jan 2018

Evaluation in Complex and Evolving Environments

Program evaluations seldom occur in stable, scientifically controlled environments. Often programs are implemented in complex and rapidly evolving settings that make traditional evaluation research approaches—which depend upon the stability of the “treatment” and the designation of predetermined outcomes—difficult to utilize.

Michael Quinn Patton, one of the originators of Developmental Evaluation, says that “Developmental evaluation processes include asking evaluative questions and gathering information to provide feedback and support developmental decision-making and course corrections along the emergent path. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation, and intentional change. The evaluator’s primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and to facilitate data-based assessments and decision-making in the unfolding and developmental processes of innovation.”

In their paper, “A Practitioners Guide to Developmental Evaluation,” Dozios and her colleagues note, “Developmental Evaluation differs from traditional forms of evaluation in several key ways:”

  • The primary focus is on adaptive learning rather than accountability to an external authority.
  • The purpose is to provide real-time feedback and generate learnings to inform development.
  • The evaluator is embedded in the initiative as a member of the team.
  • The DE role extends well beyond data collection and analysis; the evaluator actively intervenes to shape the course of development, helping to inform decision-making and facilitate learning.
  • The evaluation is designed to capture system dynamics and surface innovative strategies and ideas.
  • The approach is flexible, with new measures and monitoring mechanisms evolving as understanding of the situation deepens and the initiative’s goals emerge

Development evaluation is especially useful for social innovators who often find themselves inventing the program as it is implemented, and who often don’t have a stable and unchanging set of anticipated outcomes. Following Patton, Dozois, Langlois, and Blanchet-Cohen observe that Developmental Evaluation is especially well suited to situations that are:

  • Highly emergent and volatile (e.g., the environment is always changing)
  • Difficult to plan or predict because the variables are interdependent and non-linear
  • Socially complex— requiring collaboration among stakeholders from different organizations, systems, and/or sectors
  • Innovative, requiring real-time learning and development

Developmental Evaluation, however, is increasingly appropriate for use in the non-profit world, especially where the stability of programs’ key components including the program’s core treatment and eventual, often evolving, outcomes, are not as certain or firm as program designers might wish.

Brad Rose Consulting is experienced in working with program’s whose environments are volatile and whose iterative program designs are necessarily flexible. We are adept at collecting data that can inform the on-going evolution of a program, and have 20+ years of providing meaningful data to program designers and implementers that help them to adjust to rapidly changing and highly variable environments.

Resources:

A Practitioner’s Guide to Developmental Evaluation, Elizabeth Dozois, Marc Langlois, and Natasha Blanchet-Cohen

Michael Quinn Patton on Developmental Evaluation

Developmental Evaluation

The Case for Developmental Evaluation

March 10, 2014
10 Mar 2014

Developmental Evaluation: Evaluating Programs in the Real World’s Complex and Unpredictable Environment

Programs are seldom implemented under pristine laboratory conditions. Instead, they occur in the real world, in real time. They unfold in complex environments, with ever-changing circumstances and unforeseeable developments.  Consequently, program evaluations need to be adaptive, aware of the reality of programs’ often tumultuous contexts, and capable of suppleness and flexibility. This is especially true for evaluations that seek to assess the impact of innovative initiatives whose goals are often not standardized and pre-determined, but are evolving and emergent.

Over the last 20 years, Developmental Evaluation has emerged as an important evaluation approach for meeting the evaluation needs of innovative initiatives. As Michael Quinn Patton, a noted theorist and  practitioner of Developmental Evaluation has noted,

“Developmental evaluation (DE) is especially appropriate for innovative initiatives or organizations in dynamic and complex environments where participants, conditions, interventions, and context are turbulent, pathways for achieving desired outcomes are uncertain, and conflicts about what to do are high. DE supports reality-testing, innovation, and adaptation in complex dynamic systems where relationships among critical elements are nonlinear and emergent. Evaluation use in such environments focuses on continuous and ongoing adaptation, intensive reflective practice, and rapid, real-time feedback.”  (http://comm.eval.org/viewdocument/?DocumentKey=95f16941-7e8a-4785-907a-42615d919d7a )

Developmental Evaluation Serves Innovative Programs

While Developmental Evaluation is appropriate for many programs and organizations, it is especially useful for programs that aspire to continuous learning, that value adaptation, and that seek innovative means to address emerging (vs. “known”) issues. Such programs are typically found in the social philanthropic and non-profit sectors. Evaluators who practice Developmental Evaluation transcend the typical role of a traditional evaluator—they don’t just design formative or summative evaluations.  Developmental evaluators work closely with decision makers, program designers and staff to ask key questions about program design and logic, to collect data—sometimes in rapid time frames—to inform real-time program implementation and refinement, and to ensure that programs consistently employ the principles of learning and continuous improvement.  Patton observed in his book Utilization Focused Evaluation (3rd Edition):

“Developmental Evaluation refers to evaluation processes undertaken for the purpose of supporting program, project, staff and/or organizational development, including asking evaluative questions and applying evaluation logic for development purposes. The evaluator is part of a team whose members collaborate to conceptualize, design, and test new approaches in a long-term, on-going process of continuous improvement, adaptation and intentional change. The evaluator’s primary function is to elucidate them discussions with evaluative questions, data and logic, and to facilitate data-based decision-making…”

Brad Rose Consulting, Inc. utilizes the principles and insights of Developmental Evaluation. Our 20+ years of experience working with social entrepreneurs and innovative non-profit organizations has taught us that even seemingly “standard” program designs can benefit from a nuanced, responsive, context-sensitive, evaluation approach, one that draws on the practices of Developmental Evaluation. Additionally, innovative programs whose outcomes are not fully predictable nor exclusively pre-determined, will find that Developmental Evaluation provides the iterative feedback necessary to strengthen the program and to achieve enhanced outcomes.  Because Developmental Evaluation is essentially consultative, integrative, and built on a constructive and supportive relationship between the evaluator and the organization’s staff, it offers programs designers, managers, and implementers superior insights into the complex, often rapidly changing conditions in which genuine innovations occur. To learn more about our program development methods visit our Program development & Funding page.

Resources:
A Developmental Evaluation Primer, at J.W. McConnell Family Foundation
http://volunteer.ca/node/1787

Video Michael Quinn Patton on Developmental Evaluation

Link to Michael Quinn Patton, Developmental Evaluation
https://www.abp.org/abpwebsite/r3p/pre-read/Patton.DevelopmentalEval.pdf

A conversation with Michal Quinn Patton
http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/family-support/a-conversation-with-michael-quinn-patton

“Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.” Guilford Press New York. 2011

Copyright © 2020 - Brad Rose Consulting