Program evaluation services that increase program value for grantors, grantees, and communities. x

Archive for month: July, 2020

July 28, 2020
28 Jul 2020

Is On-line Learning Effective?

The COVID-19 Pandemic has affected many aspects of life, not least of which is education. In April, 2020, The World Economic Forum estimated that school closures had affected 1.2 Billion children. While, worldwide, not all children have begun to participate in on-line learning, many have, and much of traditional classroom-based education has been compelled to move to on-line, computer-mediated education.

How effective is on-line learning? The results are mixed. On-line learning depends of course, on accessibility. For those that have access, research indicates that on-line learning can be as effective, and in some cases, a more efficient, mode of instruction than tradition classroom learning, including enhanced retention, speed of learning, and advantages of self-paced vs. other-paced instruction. “For those who do have access to the right technology…Some research shows that on average, students retain 25-60% more material when learning online compared to only 8-10% in a classroom. This is mostly due to the students being able to learn faster online; e-learning requires 40-60% less time to learn than in a traditional classroom setting because students can learn at their own pace, going back and re-reading, skipping, or accelerating through concepts as they choose. (See “5 Reasons Why Online Learning is More Effective”)

Effectiveness however, may vary by the age of students. Younger students often thrive in a more immersive, face-to-face environment, and benefit from learning a range of social and emotional skills that are often more difficult to convey in a “narrow-cast” learning venue. Classroom structure can itself be an important social-educational factor. The effectiveness of on-line learning also may vary depending on whether instruction is exclusively on-line, or if it is “blended” (i.e., includes both on-line and face-to-face instruction). Some research has shown that blended instruction results in better student outcomes than solely on-line learning. (See “The Effectiveness of Online Learning: Beyond No Significant Difference and Future Horizons,” Tuan Nguyen, Leadership, Policy, and Organization Peabody College, Vanderbilt University)

While there appear to be benefits to on-line learning for some students, Susanna Loeb, writing in a recent issue of Education Week reminds us “Just like in brick-and-mortar classrooms, online courses need a strong curriculum and strong pedagogical practices. Teachers need to understand what students know and what they don’t know, as well as how to help them learn new material. What is different in the online setting is that students may have more distractions and less oversight, which can reduce their motivation. The teacher will need to set norms for engagement—such as requiring students to regularly ask questions and respond to their peers—that are different than the norms in the in-person setting.” She further observes that some instruction (i.e. on-line) is better than no instruction, and that “especially (for) students with fewer resources at home, (these students) learn less when they are not in school. Right now, virtual courses are allowing students to access lessons and exercises and interact with teachers in ways that would have been impossible if an epidemic had closed schools even a decade or two earlier. So, we may be skeptical of online learning, but it is also time to embrace and improve it.”


“The COVID-19 pandemic has changed education forever. This is how” World Economici Forum, April 2020

“5 Reasons Why Online Learning is More Effective”

“The Effectiveness of Online Learning: Beyond No Significant Difference and Future Horizons,” Tuan Nguyen, Leadership, Policy, and Organization Peabody College, Vanderbilt Universy

“How Effective Is Online Learning? What the Research Does and Doesn’t Tell Us” By Susanna Loeb, Education Week, March 20, 2020

“Research on the Effectiveness of Online LearningA Compilation of Research on Online Learning”

July 14, 2020
14 Jul 2020

Big Data and Evaluation

In the latest issue of the American Journal of Evaluation (Vol 41, No.2, June 2020) Robert Picciotto argues that the time has arrived for evaluation to make use of Big Data. In “Evaluation and the Big Data Challenge,” Picciotto observes that evaluators, although not yet well versed in Big Data’s use, should nonetheless actively engage Big Data, which is data composed of large data sets that are too large and complex for traditional data processing tools and that imply just-in-time information for decision-making, continuous storage and processing, and the extensive use of algorithms. The scale and seeming availability of Big Data, coupled with exponentially growing computer power, Picciotto tells us, increasingly makes the use of such data more attractive for evaluators. Big Data makes it possible for evaluators to identify patterns and to gain insights that arise from large data sets, insights that can’t be secured though limited and costly access to traditional data. Additionally, Big Data, if handled correctly, may improve the quality of evaluation. It may even allow evaluators to more effectively wrestle with the persistently thorny problem of discerning causality in complex social systems.

While Big Data offers a range of new opportunities to evaluators, Big Data (and big tech that privately owns and deploys this data) is not without its challenges and drawbacks. Governments, corporations, and interest groups are increasingly reliant on Big Data to manipulate public opinion, shape consumer behavior though predatory advertising, and in some cases to manipulatively intervene in the civic and political lives of nations. (See our previous article, “Everybody Lies”) Picciotto acknowledges the often pernicious uses of private data, and points to the lamentably under-regulated use of Big Data to monitor and influence the behavior of citizens and consumers. He also notes that the algorithms now used to analyze the volumes of data are neither objective nor universally accurate. Despite these substantial challenges, Picciotto—somewhat sanguinely I think—believes that evaluators and greater governmental regulation of big tech may ameliorate some of the more egregious dimensions and uses of Big Data. “Big Data has let lose a host of social threats: oppressive surveillance, loss of privacy, reduced autonomy, digital addiction, spread of disinformation, social polarization, and so on.” Whether evaluators and the public are capable of taming an enterprise that is now overarchingly global, under-regulated, ethically questionable, and resistant to national constraints, will need to be seen.


“Evaluation and the Big Data Challenge,” Robert Picciotto, American Journal of Evaluation pp.166-181. (Vol 41, No.2, June 2020)

“Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are,” (Seth Stephens-Davidowitz, Dey St., 2017)

“What Is Big Data?” Lisa Arthur, Forbes, Aug 15, 2013

“What is Big Data?” Bernard Marr

“Ranking, Rating, and Measuring Everything”

The Metric Society: On the Quantification of the Social, (Polity, 2019)

Copyright © 2020 - Brad Rose Consulting