Anecdote as Evidence

When discussing with clients potential sources of data about a program’s operations and effects, it has often been said to me, “But we just have anecdotal evidence.”  It’s as if anecdotal data don’t count.   Too often anecdotes are dismissed as unscientific and valueless—as if they are just stories.  In point of fact, anecdotes (qualitative accounts, “word-based” data) can be a valuable source of information and offer powerful insights about how a program works and the effects it produces.  When carefully collected and systematically analyzed, especially when combined with other sources of quantitative data, anecdotes can be a powerful “window” on a program.

In a recent blog post, (see link below) the evaluator Michael Quinn Patton reflects on the value and utility of anecdotal information.  Patton shows that, when collected in sufficient quantity, compared (or “triangulated’) with other kinds of data, and systematically and sensibly analyzed, anecdotes can provide important information about the character and meaning of a given phenomenon.  Furthermore, anecdotes are often the starting place for hypotheses and experiments that ultimately produce quantitative evidence of phenomena.  William Trochim underscores the importance of word-based, qualitative data (of which anecdotes are a specific type) when he points out. “All quantitative data is based on qualitative judgment.  Numbers in and of themselves can’t be interpreted without understanding the assumptions which underlie them…”(David Foster Wallace made a similar point from an entirely different vantage point, in Consider the Lobster: “You can’t escape language.  Language is everything and everywhere.  It’s what lets us have anything to do with one another. p. 70.)

Trochim goes on to say,

“All numerical information involves numerous judgments about what the number means. The bottom line here is that quantitative and qualitative data are, at some level, virtually inseparable.  Neither exists in a vacuum or can be considered totally devoid of the other. To ask which is “better” or more “valid” or has greater “verisimilitude” or whatever ignores the intimate connection between them. To do good research we need to use both the qualitative and the quantitative.”

Patton reminds us of the importance of anecdotes when he quotes N.G. Carr, author of  The Shallows: What the Internet is doing to our brains. W. W. Norton(2010). New York:

“We live anecdotally, proceeding from birth to death through a series of incidents, but scientists can be quick to dismiss the value of anecdotes. “Anecdotal” has become something of a curse word, at least when applied to research and other explorations of the real.. . . . The empirical, if it’s to provide anything like a full picture, needs to make room for both the statistical and the anecdotal.

The danger in scorning the anecdotal is that science gets too far removed from the actual experience of life, that it loses sight of the fact that mathematical averages and other such measures are always abstractions.”

I believe that it is important to use multiple kinds of information to understand what programs do, and what their outcomes are.  Quantitative data is essential for understanding abstract trends and at getting at the “larger picture.”  That said, it is nearly impossible to make sense of quantitative data without using language to reveal assumption, implications, explanations, and meaning of quantitative data.  Anecdotal data, as one kind of qualitative data, is critical to effective program evaluation research. To learn more about how we utilize both quantitative and qualitative data visit our Data collection & Outcome measurement page.

Resources:

Michael Quinn Patton, “Anecdote as Epithet – Rumination #1 from Qualitative Research and Evaluation Methods”
http://betterevaluation.org/blog/anecdote_as_epithet

William Trochim, “The Qualitative Debate”
http://www.socialresearchmethods.net/kb/qualdeb.php

“The Qualitative- Quantitative Debate”
http://writing.colostate.edu/guides/page.cfm?pageid=1383

Video about Qualitative, Quantitative, and Mixed-Methods Research
http://sites.macewan.ca/inspire/2014/10/09/qualitative-versus-quantitative-the-validity-debate/

A Table Summarizing Qualitative versus Quantitative Research: Key Points in a Classic Debate
http://wilderdom.com/research/QualitativeVersusQuantitativeResearch.html

Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research
http://www.webpages.uidaho.edu/css506/506%20Readings/sale%20mixed-methods.pdf