HCI area - Quantitative and Qualitative Modeling and Evaluation

Chapter 1

Gr

- MENU

Georgia Tech.

HCI area - Quantitative and Qualitative Modeling and Evaluation Introduction

Two activities go hand-in-hand in a majority of HCI research: modeling and evaluation. Modeling addresses what you know about the user, and often their surrounding social and physical environment. A variety of existing models, such as the Human-Model Processor, and modeling techniques, such as Contextual Inquiry, address differing domains and levels of specificity. Models may be used to predict performance, organize field data, and describe potential interactions with a computer interface. As you read, examine the various models and modeling techniques that provide the foundation for the research. When will these models be useful in other research settings? What do you need to know to complete a model? How can you gather that information?

One use of models is to inform the evaluation of an interface. These activities are linked as the specificity and domain of the models constrains the questions that can be addressed in an evaluation. You will notice that specific, quantitative models are used to inform specific, quantitative evaluations. Likewise, more general, qualitative models are often the basis for various qualitative studies. The feasibility of combining various evaluation techniques is influenced by the compatibility of the underlying models. If the models make conflicting assumptions about the user, perhaps even disagreeing on what can or cannot be known, then the validity of combining evaluation techniques is in question.

One of the distinguishing characteristics of the HCI area in Computer Science is the importance of evaluation of how any computer- assisted system impacts its intended user population. Evaluation in HCI (and other human-centered disciplines) is quite different from evaluation in other areas of Computer Science, mainly because it is sometimes hard to construct experiments or observations that give definitive quantitative answers regarding the merit of one system over another. Instead, evaluation in HCI consists of demonstrating a scientific approach to answer questions about a systems relative merit in its context of use. This approach can consist of a myriad of techniques. Sometimes, a very reliable quantitative result is derivable, as is the case in narrowly-focused human motor observations such as a Fitts' Law experiment or a Keystroke-Level Model analysis. Other times, when the impact on work practices is sought, it is nearly impossible to control all influences in a natural setting. A student of HCI should become familiar with the variety of evaluation techniques and develop a sense of suitability of these techniques.

One of the best ways to achieve the ability to critique evaluation approaches is to read examples of evaluation work in the literature. As you read, critique the research based on the repeatability of the experimentation (Could a competent researcher reproduce the findings following the procedures described by the authors?) and the strength of the analysis and conclusions (Did the authors do enough to convince you of their evaluation results?) This is a particularly good way to assess quantitative results, and although this criteria can be used in the assessment of qualitative research, another useful criteria is to ask about the depth of explanation of the particular phenomena being reported.

One way to organize the information that you gather is to fill in the simple, 2x2 matrix:

You should pay attention to the horizontal connections between modeling and evaluation techniques. Likewise, notice the connections and disconnections between quantitative qualitative techniques.