Q&A About A/B Testing In Education

users to one of two or more versions of a feature (“A” and “B”) and measure which one works better. Sellers on Amazon, for example, can try out different product descriptions or pricing and see how changes affect sales. Facebook and Google tweak their interfaces to determine whether proposed improvements do, in fact, lead to actual improved results.

Carnegie Learning has done many field tests of this type, but they have been difficult to implement. Keeping track of which students are assigned to get which features and making sure everything is compatible across software updates is difficult. There are several commercial systems designed to help software companies manage field testing, but none had the features that we felt that educational products needed.

Given our experience with this kind of testing, we saw the opportunity to design an open-source platform that can be used by all educational software developers.

Why is it important? What’s an example of the research that might be done on it?
Educational software is different from most other types of software in significant ways. One particular concern for software (like MATHia) that is used for school-based instruction is that students use the software as a class. Suppose we want to determine which of several fraction bar representations best helps students learn to divide fractions. If we randomly assign each student to a fraction bar representation, the teacher will need to support multiple methods, which can make it more difficult for her to support her students.

So one of our needs is to be able to assign conditions by group. In some cases, we might want a class, teacher, school or even district to have a consistent experience. In research design, this is called the “unit of assignment” issue.​

Another consideration is consistency for individual students. If a student is assigned to use fraction bar A, then we probably want that student to use that representation throughout the whole class. If would likely be disruptive to the student if they were given one representation when initially learning and a different one when studying for a test a month later. It’s important in education that, even when the test is over, the student experience stays the same.

How will the effort help students and teachers? 
The ability to try out improvements like this will lead to more effective approaches to instruction. We are working with learning science researchers to field-test basic principles that can apply across many instructional systems, not just MATHia.

Ultimately, we want to build in supports for teachers to run their own experiments. We think it is important that teachers participate in the process of understanding what works for their students.

Where and when will it be released? 
The first version of the system will be used within MATHia in the 2020/21 school year. After that, we will work with others to improve the platform for use with other software.