Automated Diagnosis of Usability Problems Using Statistical Computational Methods

The effects of poor usability range from mere inconvenience to disaster. Human factors specialists employ usability analysis to reduce the likelihood or impact of such failures. However, good usability analysis requires usability reports that are rarelycollected, rarely complete, and difficult to analyze.Aptima and the Center for Intelligent Information Retrieval (U. Mass. Amherst) have partnered to develop a usability analysis system that addresses these problems. The system will consist of (1) an interface to elicit useful usability reports in naturallanguage, (2) a text analysis engine that classifies these reports (or existing usability reports) using a validated taxonomy, and (3) an analysis interface for analyzing individual usability reports and trends in usability problems. We will train and testthe system using a very large corpus of publicly available usability reports categorized into an extension of the User Action Framework. Aptima and CIIR will deliver a report of results, a software prototype, and a corpus of manually categorized usabilityreports. The anticipated benefits of the proposed Usability Reports Diagnostic Tool are as follows. The proposed tool will help to collect and classify usability reports. During usability tests, the tool will facilitate the articulation of observedusability problems. The tool's automatic classifier will analyze the resulting usability reports using a text analysis algorithm and categorize them according to a classification schema (also to be developed under this project). The output will identifythe problem observed, classify it by type and by cause, and allow further refine by the user.The Usability Reports Diagnostic Tool will be useful for both usability professionals within software development organizations who must generate reports for internal use and customers, and human factors or other usability professionals in customerorganizations who are evaluating the usability of the products. Our solution will enable users (and observers such as usability engineers) to express usability problems in their own terms, while systematically categorizing those problems in meaningful wayfor designers.