Steve Ritter of Carnegie Learning discusses a new program to integrate research into MATHia, Carnegie’s math tutoring software. Ulrich Boser interviewed Steve about the open-source software that will allow developers to better identify what works for students.
Can you tell us about the new A/B testing platform that you are developing?
Carnegie Learning has done many field tests of this type, but they have been difficult to implement. Keeping track of which students are assigned to get which features and making sure everything is compatible across software updates is difficult. There are several commercial systems designed to help software companies manage field testing, but none had the features that we felt that educational products needed.
Given our experience with this kind of testing, we saw the opportunity to design an open-source platform that can be used by all educational software developers.
Why is it important? What’s an example of the research that might be done on it?
Educational software is different from most other types of software in significant ways. One particular concern for software (like MATHia) that is used for school-based instruction is that students use the software as a class. Suppose we want to determine which of several fraction bar representations best helps students learn to divide fractions. If we randomly assign each student to a fraction bar representation, the teacher will need to support multiple methods, which can make it more difficult for her to support her students.
So one of our needs is to be able to assign conditions by group. In some cases, we might want a class, teacher, school or even district to have a consistent experience. In research design, this is called the “unit of assignment” issue.
The ability to try out improvements like this will lead to more effective approaches to instruction. We are working with learning science researchers to field-test basic principles that can apply across many instructional systems.”
How will the effort help students and teachers?
The ability to try out improvements like this will lead to more effective approaches to instruction. We are working with learning science researchers to field-test basic principles that can apply across many instructional systems, not just MATHia.
Ultimately, we want to build in supports for teachers to run their own experiments. We think it is important that teachers participate in the process of understanding what works for their students.
Where and when will it be released?
The first version of the system will be used within MATHia in the 2020/21 school year. After that, we will work with others to improve the platform for use with other software.