A web tool to interpret the impact of programs on student social and behavioral outcomes

By Marta Pellegrini, University of Cagliari, Italy

A recent study described the use of an online application to show the results of a meta-analysis of psychological and educational programs in K–12 on student social and behavioral outcomes. The researchers highlighted the huge variability of the effects based on filtering the types of programs, contexts, and students, and proposed a way to better interpret the meta-analysis results and make use of them in future research.

Data come from a meta-analysis of 323 experiments conducted in the U.S. in K–12 on student antisocial behaviors, self-regulation, social and interpersonal adjustment, and personal adjustment. Several characteristics were extracted to describe the study setting, such as the region of study, the type of program, its duration, grade level, ethnicity, and socio-economic status--all characteristics that may play a role in explaining the variability of the program effect.

ES Contextualizer is a web tool that makes it possible for users to select characteristics of a context to produce targeted estimates of the impact of programs on student outcomes. ES Contextualizer can be used by researchers to plan a new study or to facilitate effect size interpretation. The authors described several examples of how to use the web tool. In one of them, a researcher conducted an experiment on the effects of an intervention on middle school students’ social problem-solving skills based on teacher reports and obtained an average effect size of +0.15. Using ES Contextualizer, this estimate can be compared to similar interventions conducted in similar contexts and grade levels. If the effect size is compared to the correct benchmark of +0.05 for middle school students from the ES Contextualizer tool, the researchers can conclude that what they found is a meaningful impact. If the grade level is not selected, the tool shows a benchmark of +0.26 instead, possibly leading the researcher to draw a different conclusion.

The authors recommended using the web tool’s filters to generate more nuanced effect size interpretations and more accurate estimates for use in power analysis.