The COVID-19 pandemic led to temporary closures of schools around the world, resulting in a change from face-to-face teaching to distance teaching, which had been practiced minimally until then. In this study, we investigated the effects of pandemic-related school closures on students’ chemistry-related competencies, at the transition from primary to secondary school. We also explored the extent to which at-home or in-school data collection influenced the results. We measured the competencies of 2,262 students from grades 5 to 9 in Germany. Data collection took place before, during, and after the pandemic-related school closures, based on test booklets completed by students. The results showed that the competencies of students in Chemistry, who were taught in school before the closures, were similar to those of students who were taught via distance learning. Thus, students’ competencies were similar before and after the school closures. The school closures led to differences not only in teaching, but also in the way the data in this study was collected. During school closures, students worked on their test booklets at home, and before and after school closures, the data were collected at school. This also enabled us to examine the effects of the different data collection designs on the test scores. We found differences between the results of the test booklets completed at home, and those completed at school, only for younger students. For students in higher grades, there were no differences.