Issues in the design of experiments for studying user interaction with software development tools

Abstract

In the past software engineers have been reluctant to conduct experiments with software and the users of that software. As this reluctance is overcome through, for example, incorporation of human-computer interaction as a topic in software engineering education, we believe that some of the main issues concerning experiments will become task design for an experiment, subject selection, experimental design, measurement, statistical analysis and results interpretation. We are not new in advocating the use of experiments nor in examining the issues associated with such experimentation but we do examine it in the context of one domain where the resistance to such ideas seems to have been strong - the development of software tools used by software engineers.