Gathering evaluative information about a program or initiative often relies upon evaluators physically visiting the program’s location in order to observe program operations, to collect evidence of the program’s implementation and outcomes, and to interview staff and program participants. The empirical and observational nature of site visits offer evaluators a unique lens through which to “see” what the program actually is, and how it attempts to achieve the desired outcomes it hopes to achieve.

In their influential article, “Evaluative Site Visits: A Methodological Review,” American Journal of Evaluation, Vol. 24, No. 3, 2003, pp. 341–352, Lawrence, Keiser, and Levoie note that, “An evaluative site visit occurs when persons with specific expertise and preparation go to a site for a limited period of time and gather information about an evaluation object either through their own experience or through the reported experiences of others in order to prepare testimony addressing the purpose of the site visit.” Unlike case studies, which are of longer duration and often of greater depth, and which seek to describe in detail the instance or phenomena under study, site visits are of limited time duration, and are focused on gathering data that ultimately will inform judgement about a program’s worth/value. Site visits typically involve the use of a number of qualitative methods (e.g., individual and focus group interviews, observations, document review, etc. For more information on the kinds of data that site visits permit, see our previous blog post “Just the Facts: Data Collection.”

Michael Quinn Patton summarizes the essential elements of an evaluation site visit:

  1. Competence– Ensure that site‐visit team members have skills and experience in qualitative observation and interviewing. Availability and subject matter expertise does not suffice.
  2. Knowledge– For an evaluative site visit, ensure at least one team member, preferably the team leader, has evaluation knowledge and credentials.
  3. Preparation– Site visitors should know something about the site being visited based on background materials, briefings, and/or prior experience.
  4. Site participation– People at sites should be engaged in planning and preparation for the site visit to minimize disruption to program activities and services.
  5. Do no harm– Site‐visit stakes can be high, with risks for people and programs. Good intentions, naiveté, and general cluelessness are not excuses. Be alert to what can go wrong and commit as a team to do no harm.
  6. Credible fieldwork– People at the site should be involved and informed, but they should not control the information collection in ways that undermine, significantly limit, or corrupt the inquiry. The evaluators should determine the activities observed and people interviewed, and arrange confidential interviews to enhance data quality.
  7. Neutrality– An evaluator conducting fieldwork should not have a preformed position on the intervention or the intervention model.
  8. Debriefing and feedback– Before departing from the field, key people at the site should be debriefed on highlights of findings and a timeline of when (or if) they will receive an oral or written report of findings.
  9. Site review– Those at the site should have an opportunity to respond in a timely way to site visitors’ reports, to correct errors and provide an alternative perspective on findings and judgments. Triangulation and a balance of perspectives should be the rule.
  10. Follow-up– The agency commissioning the site visit should do some minimal follow‐up to assess the quality of the site visit from the perspective of the locals on site.

Lawrence, Keiser, and Levoie argue that evaluative site visits are not merely a venue in which a range of predominately qualitative methodologies are used, but a specific kind of methodology, which is distinguished by its use of observation. “We believe site visit methodology is based on ontological beliefs about the nature of reality and epistemological beliefs about whether and how valid knowledge can be achieved. Ontologically, in order to conduct site visits the evaluator must assume that there is a reality that can be seen or sensed and described. Epistemologically, site visits are based in the belief that site visitors are legitimate, sensing instruments and that they can obtain valid information through first-hand encounters with the object being evaluated.”

Accordingly, site visits are where evaluators can get “the feel” of what a program is and does. As a result, site visits are a critical means through which evaluators gather and interpret data with which to make judgements about the value and effects of a program.

Resources:

“Evaluative Site Visits: A Methodological Review,” Frances Lawrenz, Nanette Keiser, and Bethann Lavoie, American Journal of Evaluation, Vol. 24, No. 3, 2003, pp. 341–352.

See Michael Quinn Patton quoted in Editors’ Note, Randi K. Nelson and Denise L. Roselan, New Directions in Evaluation, December, 2017

“Using Qualitative Interviews in Program Evaluations”

Conducting and Using Evaluative Site Visits: New Directions for Evaluation, Number 156, February 2018

“Developmental Evaluation: Evaluating Programs in the Real World’s Complex and Unpredictable Environment”

Recommended Posts