Potentially engaging mathematics teaching practices in secondary classrooms: Investigating observation tool validity and score reliability

The purpose of this report is to present our process for establishing evidence of validity and reliability of an observation tool [PEMTP: Potentially Engaging Mathematics Teaching Practices] used to investigate mathematics teaching practices that have the potential to engage high school students. We first present our practices for establishing content-oriented evidence of validity for our observation tool. Next, we describe how we developed a framework to identify and specify potentially engaging mathematics teaching practices, including our efforts to pilot the tool to design descriptive levels for rubrics. We then report on a reliability investigation we conducted to examine the nature of coding disagreements. Our results indicate the value of reporting on validity processes for tools such as the PEMTP to illustrate the ways that the tool has been designed and revised, which can inform the use cases and potential future applications of the tool. These results also show how such validation efforts can lead to meaningful revisions–in the case of the PEMTP, the addition of two rubrics aligned with teaching mathematics for equity. Finally, the results of our reliability investigation indicate the value of using multiple raters and collaborative resolution meetings when investigating complex phenomena like classroom instruction.