Evaluating a Data-Driven Redesign Process for Intelligent Tutoring Systems
Summary: arXiv:2603.29094v1 Announce Type: cross
Abstract
Past research has defined a general process for the data-driven redesign of educational technologies and has shown that in carefully-selected instances, this process can help make systems more effective. In the current work, we test the generality of the approach by applying it to four units of a middle-school mathematics intelligent tutoring system that were selected not based on suitability for redesign, as in previous work, but on topic. We tested whether the redesigned system was more effective than the original in a classroom study with 123 students. Although the learning gains did not differ between the conditions, students who used the Redesigned Tutor had more productive time-on-task, a larger number of skills practiced, and greater total knowledge mastery. The findings highlight the promise of data-driven redesign even when applied to instructional units not selected as likely to yield improvement, as evidence of the generality and wide applicability of the method.
Introduction
Intelligent tutoring systems (ITS) have become an integral part of educational technology, providing personalized learning experiences to students. The effectiveness of these systems can vary significantly based on their design and implementation. This article discusses a recent study that explores a data-driven redesign process aimed at enhancing the effectiveness of ITS in middle-school mathematics education.
Methodology
The study involved a systematic redesign of four instructional units within a middle-school mathematics ITS. Unlike previous research that focused on redesigning systems based on their prior performance, this study selected units based solely on the topics they covered. The redesign process included:
- Analysis of existing data from student interactions with the ITS.
- Identification of areas for improvement based on observed student behaviors.
- Implementation of changes aimed at enhancing engagement and skill mastery.
Results
The redesigned ITS was tested in a classroom setting involving 123 students. Key findings from the study included:
- No significant differences in overall learning gains between students using the original and redesigned systems.
- Students utilizing the Redesigned Tutor exhibited more productive time-on-task.
- Participants practiced a greater number of skills, leading to increased total knowledge mastery.
Discussion
While the redesigned system did not lead to statistically significant improvements in learning outcomes, the increased time-on-task and skill practice suggest that data-driven redesign has the potential to enhance student engagement. These findings underscore the importance of continual assessment and refinement of educational technologies, even when units are not initially deemed suitable for redesign.
Conclusion
The study presents compelling evidence for the generality and applicability of data-driven redesign processes in educational technology. It highlights the need for ongoing research to explore the nuances of redesigning ITS and the conditions under which they can be optimized for better student outcomes. Future work will continue to evaluate and refine these methods to ensure that intelligent tutoring systems can effectively support diverse learning needs in classrooms.