What are mathematics teachers’ considerations in grouping students, and how could automated formative assessment systems help them in doing it? In this study, nine teachers were asked to use data on students’ performance in a mathematics task, derived from an automated formative assessment system, to create pairs in which students could contribute to their peers. We called this grouping strategy “complementary.” The teachers were also asked to explain their considerations for each grouping. We found two main grouping strategies in addition to the complementary one: based on similar answers (“similarity”), and based on dissimilar answers, in which one student performed better than another and could teach the other (“hierarchy”). Findings show that despite the experimenter’s request to group students based on complementarity, teachers mostly grouped based on other considerations, at times even grouping students whose answers were complementary using hierarchical considerations. In some cases, different teachers formed the same groups of students based on different grouping strategies. The findings confirm the hypothesis that informed grouping may be challenging for teachers, and may benefit greatly from an automated pairing system.