“Context matters,” Debra Rog, past President of the American Evaluation Association reported in her address to the 2012 meeting of ASA, and recently wrote in her insightful article, “When Background Becomes Foreground: Toward Context-Sensitive Evaluation Practice (New Directions for Evaluation No.135, Fall, 2012)
Indeed, as Rog correctly points out, program evaluators (and program sponsors) often ignore at their own peril, program context. Too often evaluations begin with attention focused on methodology, then later—often when it’s too late–discover that the results, meanings, and uses, of evaluation findings have been hampered by insufficiently considering the context in which an evaluation is conducted.
Five Aspects of Context
Rog says there are 5 aspects of context that directly and indirectly affect the selection, design, and ultimate success of an evaluation:
1. the nature of the problem that a program or intervention seeks to address
2. the nature of the intervention—how the structure, complexity, and dynamics of the program (including program life cycle) affect the selection and implementation of the evaluation approach
3. the broader environment (or setting) in which the program is situated and operates (for example, the availability of affordable housing may profoundly affect the outcomes and successes of a program intended to assist homeless persons to accesses housing.)
4. the parameters of the evaluation, including the evaluation budget, allotted time for implementation of evaluation activities, and the availability of data
5. the decision-making context for the evaluation—who are the decision-makers that will use evaluation findings, which types of decisions do they need to make, and whcih standards of rigor and levels of confidence do decisions-makers require
Context-Sensitive Evaluations – What to Look For
Rog underscores the importance of conducting “context-sensitive” evaluations—evaluations that first consider the various aspects of the context in which programs operate and in which program evaluation activities will occur. She makes a plea to evaluators and evaluation sponsors to refrain from engaging in a “method’s first approach” which too often fetishizes methodologies at the cost of conducting appropriate evaluations that can be of maximum value and use to all program stakeholders.
In our experience, the most effective context-sensitive evaluations address:
- Who program stakeholders are (funders, program managers, program participants, community members, etc.)
- What stakeholders need to learn about a program’s operation and outcomes
- The social and economic context of the program’s operation
- Key questions to guide the evaluation research
- Research methodologies that provide robust and cost-effective findings
- A logic model that clearly specifies the program activities and results
- A wide range of evaluation research methodologies and data collection strategies to ensure that program results are systematically and rigorously measured.
- Clear, accessible reports so that all stakeholders benefit from evaluation findings
- Detailed recommendations for how sponsors and program managers can strengthen further efforts
When it comes to program evaluation, not only does context matter, it is on the critical path for getting the best results. To find out about how we consider context in our evaluation methods visit our Data collection & Outcome measurement page.
