Can design choices for language-based editors be analysed with keystroke-level models?

Abstract

We have been concerned for some time with the lack of rigorous experimental evaluation of design options chosen for tools used by software engineers. In a series of studies using various evaluation techniques we built Keystroke-Level Models (KLM) and conducted an empirical usability study of a design issue (choice of editing paradigm for language-based editors) that has reached a 'subjective stalemate' in the research community. The KLM analysis enabled us to predict usage differences and while this was useful we also noticed several problems, in particular we were concerned about the estimated value and placement of the memory operator. By utilizing the same tasks in the usability study as in the KLM analysis, we were able to compare results from both evaluations and effectively validate the overall KLM estimates and the specific operator values involved.