Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching

Kitto, Kirsty and Williams, Cameron and Alderman, Lyn ORCID: https://orcid.org/0000-0001-6863-5633 (2019) Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching. Assessment and Evaluation in Higher Education, 44 (3). pp. 338-360. ISSN 0260-2938


Abstract

Student evaluations of teaching (SETs) have been used to evaluate higher education teaching performance for decades. Reporting SET results often involves the extraction of an average for some set of course metrics, which facilitates the comparison of teaching teams across different organisational units. Here, we draw attention to ongoing problems with the naive application of this approach. Firstly, a specific average value may arise from data that demonstrates very different patterns of student satisfaction. Furthermore, the use of distance measures (e.g. an average) for ordinal data can be contested, and finally, issues of multiplicity increasingly plague approaches using hypothesis testing. It is time to advance the methodology of the field. We demonstrate how multinomial distributions and hierarchical Bayesian methods can be used to contextualise the SET scores of a course to different organisational units and student cohorts, and then show how this approach can be used to extract sensible information about how a distribution is changing.


Statistics for USQ ePrint 43357
Statistics for this ePrint Item
Item Type: Article (Commonwealth Reporting Category C)
Refereed: Yes
Item Status: Live Archive
Faculty/School / Institute/Centre: No Faculty
Faculty/School / Institute/Centre: No Faculty
Date Deposited: 01 Dec 2021 23:20
Last Modified: 07 Dec 2021 05:00
Uncontrolled Keywords: contextualisation; distributions; evaluation; hierarchical Bayesian model; sensemaking; student evaluations of teaching (SET)
Fields of Research (2008): 13 Education > 1301 Education Systems > 130103 Higher Education
Fields of Research (2020): 39 EDUCATION > 3904 Specialist studies in education > 390402 Education assessment and evaluation
Socio-Economic Objectives (2008): C Society > 93 Education and Training > 9303 Curriculum > 930301 Assessment and Evaluation of Curriculum
Socio-Economic Objectives (2020): 16 EDUCATION AND TRAINING > 1603 Teaching and curriculum > 160302 Pedagogy
Identification Number or DOI: https://doi.org/10.1080/02602938.2018.1506909
URI: http://eprints.usq.edu.au/id/eprint/43357

Actions (login required)

View Item Archive Repository Staff Only