Program evaluation services that increase program value for grantors, grantees, and communities. x

Tag Archive for: goals

You are here: Home » goals
June 12, 2014
12 Jun 2014

Questions Before Methods

Like other research initiatives, each program evaluation should begin with a question, or series of questions that the evaluation seeks to answer.  (See my previous bog post “Approaching and Evaluation: Ten Issues to Consider.”).  Evaluation questions are what guide the evaluation, give it direction, and express its purpose.   Identifying guiding questions is essential to the success of any evaluation research effort.

Of course, evaluation stakeholders can have various interests, and therefore, can have various kinds of questions about a program.  For example, funders often want to know if the program worked, if it was a good investment, and if the desired changes/outcomes were achieved (i.e., outcome/summative questions).  During the program’s implementation, program managers and implementers, may want to know what’s working and what’s not working, so they can refine the program to ensure that it is more likely to produce the desired outcomes (i.e., formative questions).  Program managers and implementers may also want to know which part of the program has been implemented, and if recipients of services are indeed, being reached and receiving program services (i.e., process questions).  Participants, community stakeholders, funders, and program implementers may all want to know if the program makes the intended difference in the lives of those who the program is designed to serve (i.e., outcome/summative questions).

While program evaluations seldom serve only one purpose, or ask only one type of question, it may be useful to examine the kinds of questions that pertain to the different types of evaluations.  Establishing clarity about the questions an evaluation will answer will maximize the evaluation’s effectiveness, and ensure that stakeholders find evaluation results useful and meaningful.

Types of Evaluation Questions

Although the list below is not exhaustive, it is illustrative of the kinds of questions that each type of evaluation seeks to answer.

▪ Process Evaluation Questions

  • Were the services, products, and resources of the program, in fact, produced and delivered to stakeholders and users?
  • Did the program’s services, products, and resources reach their intended audiences and users?
  • Were services, products, and resources made available to intended audiences and users in a timely manner?
  • What kinds of challenges did the program encounter in developing, disseminating, and providing its services, products, and resources?
  • What steps were taken by the program to address these challenges?

▪ Formative Evaluation Questions

  • How do program stakeholders rate the quality, relevance, and utility, of the program’s activities, products and services?
  • How can the activities, products, and services of the program be refined and strengthened during project implementation, so that they better meet the needs of participants  and stakeholders?
  • What suggestions do participants and stakeholders have for improving the quality, relevance, and utility, of the program?
  • Which elements of the program do participants find most beneficial, and which least beneficial?

▪ Outcome/Summative Evaluation Questions

  • What effect(s) did the program have on its participants and stakeholders (e.g., changes in knowledge, attitudes, behavior, skills and practices)?
  • Did the activities, actions, and services (i.e., outputs) of the program provide high quality services and resources to stakeholders?
  • Did the activities, actions, and services of the program raise the awareness and provide new and useful knowledge to participants?
  • What is the ultimate worth, merit, and value of the program?
  • Should the program be continued or curtailed?

The process of identifying which questions program sponsors want the evaluation to answer thus becomes a means for identifying the kinds of methods that an evaluation will use.  If, ultimately, we want to know if a program is causing a specific outcome, then the best method (the “gold standard”) is to design a randomized control experiment (RCT).  Often, however, we are interested not just in knowing if a program causes a particular outcome, but why it does so and how it does so.  In that case, it will be essential to use a mixed methods approach that draws not just on quantitative outcome data that compare the outcomes of treatment and control groups, but also to use qualitative methods (e.g., interviews, focus groups, direct observation of program functioning, document review,  etc.) that can help elucidate why what happens, happens, and what program participants experience.

Robust and useful program evaluations begin with the questions that stakeholders want answered, and then identifies the best methods to gather data to answer these questions. To learn more about our evaluation methods visit our Data collection & Outcome measurement page.

in Data Collection, Evaluation, Models/Tools, Research /by /#permalink
July 18, 2012
18 Jul 2012

Approaching An Evaluation – Ten Issues to Consider

Before beginning an evaluation, it may be helpful to consider the following questions:

1. Why is the evaluation being conducted? What is/are the purpose(s) of the evaluation?

Common reasons for conducting an evaluation are to:

  • monitor progress of program implementation and provide formative feedback to designers and program managers (i.e., a formative evaluation seeks to discover what is happening and why, for the purpose of program improvement and refinement.)
  • measure final outcomes or effects produced by the program (i.e., a summative evaluation);
  • provide evidence of a program’s achievements to current or future funders;
  • convince skeptics or opponents of the value of the program;
  • elucidate important lessons and contribute to public knowledge;
  • tell a meaningful and important story;
  • provide information on program efficiency;
  • neutrally and impartially document the changes produced in clients or systems;
  • fulfill contractual obligations;
  • advocate for the expansion or reduction of a program with current and/or additional funders.

Evaluations may simultaneously serve many purposes. For the purpose of clarity and to ensure that evaluation findings meet the client’s and stakeholders’ needs, the client and evaluator may want to identify and rank the top two or three reasons for conducting the evaluation. Clarifying the purpose(s) of the evaluation early in the process will maximize the usefulness of the
evaluation.

2. What is the “it” that is being evaluated? (A program, initiative, organization, network, set of processes or relationships, services, activities?) There are many things that may be evaluated in any given program or intervention. It may be best to start with a few (2-4) key questions and concerns (See #4, below ). Also, for purposes of clarity, it may be useful to discuss what isn’t being evaluated.

3. What are the intended outcomes that the program or intervention intends to produce? What is the program meant to achieve? What changes or differences does the program hope to produce, and in whom? What will be different as the result of the program or intervention? Please note that changes can occur in individuals, organizations, communities, and other social environments. While evaluations often look for changes in persons, changes need not be restricted to alterations in individuals’ behavior, attitudes, or knowledge, but can extend to larger units of analysis, like changes in organizations, networks of organizations, and communities. For collective groups or institutions, changes may occur in: policies, positions, vision/mission, collective actions, communication, overall effectiveness, public perception, etc. For individuals: changes may occur in behaviors, attitudes, skills, ideas, competencies, etc.

4. Every evaluation should have some basic questions that it seeks to answer. What are the key questions to be answered by the evaluation? What do clients want to be able to say (report) about the program to key stakeholders? By collaboratively defining key questions, the evaluator and the client will sharpen the focus the evaluation and maximize the clarity and usefulness of the evaluation findings.

5. Who is the evaluation for? Who are the major stakeholders or interested parties for evaluation results? Who wants to know? Who are the various “end users” of the evaluation findings?

6. How will evaluation findings be used? (To improve the program; to make judgements about the economic or social value of the program—it’s costs and benefits–; to document and publicize efforts; to expand or curtail the program?)

7. What information will stakeholders find useful and how will they use the evaluation findings?

8. What will be the “product” of the evaluation? What form will findings take? How are findings to be disseminated? (Written report, periodic briefings, analytic memos, a briefing paper, public presentation, etc.?)

9. What are the potential sources of information/data? (Interviews, program documents, surveys, quantitative/statistical data, comparison with other/similar programs, field observations, testimony of experts?) What are the most accessible and cost effective sources of information for the client?

10. What is the optimal design for the evaluation? Which Methods will yield the most valid, accurate, and persuasive evaluation conclusions? If interested in indicating a cause and effect relationship, is it possible (and desirable) to expend the resources necessary to conduct an experimental (i.e., “control group study”), or quasi-experimental design study? Experimental designs can be resource intensive and therefore, more costly. If resources are not substantial, the client and the evaluator will want to discuss other kinds of evaluation designs that will provide stakeholders with the most substantial, valid, and persuasive evaluative information.

To learn more about our evaluation methods visit our Data collection & Outcome measurement page.

in Data Collection, Evaluation /by /#permalink

Keywords

accountability actionable knowledge ai anecdotes Applicant Tracking Systems artificial intelligence assessment ATS authority automation Average basics beneficiaries bias bias in research big data Blended learning brandeis bureaucrazy Burnout busyness causation change charity charter schools cognitive bias collaboration College Admissions Scandal community foundations connections control group cost effectiveness COVID-19 data data collection data sources desired results developmental evaluation Digital technology in the classroom disadvantaged youth education educator emotional intelligence evaluation evaluation design evaluation findings Evaluation in complex environments evaluation questions evaluation research evaluation site visit evaluation vs. research evaluator skills evidence examples experience experimental design expertise external evaluator Flexwork focus groups formative evaluation formative evaluations Foundations funding game the system gaming the system gather data goals grantees Groups Groupthink helpful resources higher education Hustle culture Impact Evaluation In-groups integrity interpersonal interpersonal skills interview protocol Interviews intuition Learning learning outcomes Learning to Learn lifelong learning logic model logic modeling manufacturing max weber Mean measure measures members Meritocracy methodology methods Metric metrics models needs assessment non-profit programs non-profits Nonprofits Normal objectivity objectivity in the Social Sciences observations OD On-line learning open-ended questions organizational culture organizational development organization development organizations Out-groups outcome evaluation outcome evaluations outcomes Pandemic Philanthrocapitalism Philanthropies philanthropy planning Politics post-test power power organization pre-test privacy problem solving process process evaluation process evaluations program program evaluation program funder programming programs progress Pros and Cons of technology in the classroom Public Education public school privatization public schools public school system purpose of education qualitative evaluation qualitative measures qualitative methods quality quantitative evaluation quantitative measure quantitative measures quantitative methods quantity quasi-experimental design questions randomized control trials rationalization RCTs reports research resources respondents robo-graders robots Roles school choice Schooling Schools site visit Social Credit System social entrepreneurs social innovation social research social robotics stakeholder analysis stakeholders standardized testing starting an evaluation Statistics strategic assessment strategic planning Summative Evaluation Surveillance survey data surveys Systems theory Systems thinking Systems thinking and evaluation teachers teams Telecommuting The cult of busyness theories Thought partner tips tools treatment group types of evaluation variables workflow Working from home work teams world without work

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon

Sign up for our Email Newsletter

Copyright © 2020 - Brad Rose Consulting