An experimental analysis of assessor specific bias in a programming assessment in multi-assessor scenarios utilizing an eye tracker

Debnath, Tanmoy and Paul, Manoranjan and Chakraborty, Subrata and Podder, Pallab Kanti and Gururajan, Raj and Hafeez-Baig, Abdul (2016) An experimental analysis of assessor specific bias in a programming assessment in multi-assessor scenarios utilizing an eye tracker. In: London International Conference on Education (LICE-2016), 14-17 Nov 2016, London, United Kingdom.

[img]
Preview
Text (Submitted Version)
An experimental analysis of assessor specific bias in a programming assessment.pdf

Download (351Kb) | Preview

Abstract

It has been experienced and reported by academic institutions around the globe that marking of most subject’s assessment scripts vary when different assessors are utilized for a given subject. To understand the difference, we capture and analysis pattern while they are marking the scripts. For this, a Java programming assessment from a real life university examination is marked by independent assessors. The assessors marked the scanned assessment scripts on a computer screen in front of an Eye tracker machine and their eye gaze data were recorded real time. Data indicate that different assessors marked the same answer script differently and their visual pattern are also varied although they were given the exact same instructions which demonstrates bias to a degree. For quality marking, several findings including the number of assessors needed are also presented in this manuscript.


Statistics for USQ ePrint 29726
Statistics for this ePrint Item
Item Type: Conference or Workshop Item (Commonwealth Reporting Category E) (Paper)
Refereed: Yes
Item Status: Live Archive
Additional Information: Copyright © LICE-2016. No evidence of copyright restrictions preventing deposit of Submitted version.
Faculty / Department / School: Current - Faculty of Business, Education, Law and Arts - School of Management and Enterprise
Date Deposited: 08 Aug 2017 06:04
Last Modified: 31 Oct 2017 01:56
Uncontrolled Keywords: marker bias, Eye tracking, assessment
Fields of Research : 13 Education > 1301 Education Systems > 130103 Higher Education
08 Information and Computing Sciences > 0806 Information Systems > 080699 Information Systems not elsewhere classified
Socio-Economic Objective: C Society > 93 Education and Training > 9305 Education and Training Systems > 930599 Education and Training Systems not elsewhere classified
URI: http://eprints.usq.edu.au/id/eprint/29726

Actions (login required)

View Item Archive Repository Staff Only