RSS Updates
Twitter
Facebook
|
|||||||||||
<< Back to Ask the Expert Home
Ask the Expert Your Question!
Question 3: The challenge we face is how to measure whether a particular transition strategy is having a positive impact on student success – such as academic achievement or graduation. The “n” (number) of students involved in some transition programs are typically quite small, and some approaches are so individualized that they vary from student to student. Both situations make measurement of impact very challenging. Do you have any recommendations for our group about how to measure the impact of a transition strategy, program, or approach? |
Thoughtful, empirical study of dropout prevention strategies is necessary to inform action, be accountable to stakeholders, and effectively advocate for investment in expansion of this work. Without it, we remain in the cave recreating the wheel. With it, we progress toward our ideals. It is not easy in the complex, messy world of education reform and social change, however, for all the reasons you point out and more. Let me offer three suggestions:
First, value and engage in each phase of learning and study: There is an understandable push to measure impact—we want to know if an idea works, after all, and evidence is increasingly a prerequisite for further investment in implementation, and rightly so in my view. But there is great value in initial exploration, description, and development of a strategy, or clusters of strategies, prior to major investment in the kind of study required to establish impact. I strongly recommend that your transitions group engage in qualitative study (via observation, interview, feedback surveys) to link the strategies to extant theory and practice, and to document and describe the strategies, their core elements or practices, and how they are being creatively adapted to different contexts (or different types of transitions) and improved via feedback through your group. This disciplined exploration is essential to understanding what is meant by a specific strategy (e.g. advisory, mentor, teacher team, social/study skills curriculum, grief counseling, attendance outreach, project-based learning, etc.) and what it takes to implement the strategy consistently and well.
An initial development phase can and should be accompanied by suggestive evidence that it holds promise of impact using benchmarks and multiple measures (see below), but needn’t be held to the standards of a full-blown impact study (with counterfactuals, power requirements, etc). Indeed, strong case descriptions that contain thoughtful specification of core elements and conditions lay important groundwork for further study. The Institute of Education Sciences (IES) now has a clear progression of study goals ranging from exploratory and developmental to quasi-experimental field testing to randomized control trials. See also Fixsen and Blase’s work on implementation research.
Second, use benchmarks and multiple measures: While everyone wants to know the impact of an intervention on test scores, dropout rates, and graduation rates, impact may also be measured more frequently and in the near term using other indicators. Attendance (average daily and # days missed), completed assignments/projects, grades and pass rates in core academic courses, grade promotion, behavior marks (referrals, suspensions), participation in extracurricular activities, surveys of students and adults assessing academic engagement and sense of belonging in school—all of these are indicators of being engaged in school and on track for graduation.
Third, use counterfactual where feasible. To determine whether a strategy has a positive impact, a study has to determine that students were significantly better off as a result of the intervention than they would have been without it. That means measuring outcomes not only for students who receive the treatment, but for a group of like comparison students as well. At a minimum, where data is available, compare improvements on outcomes of students experiencing the intervention to class, school, or district average change (note this requires pre-intervention baseline data and data for at least one and preferably multiple follow-up timepoints). Such comparisons can be compelling and persuasive and lay groundwork for wider implementation and study. Much better, if resources allow, is to create a comparison group either through random assignment or statistical matching techniques (multiple regression, propensity score matching). Sample size is an issue and I recommend that you team with a statistician familiar with social research who can guide you through the steps to determine the N you need given the type of test you want to run and assumptions about desired significance levels and power. MDRC, a social research firm based in NYC has conducted a number of rigorous studies on transitions and comprehensive school reform using counterfactuals and would be a good resource.
References
Disclaimer
It is important to note that the National High School Center does not endorse particular programs or practices.
