Innovative Hands-Free Point-and-Click Computer Control Device Within a Moving Vehicle

Most computer programs require both keyboard data entry and Point-and-Click (PAC) operations with a mouse or trackball device to operate efficiently. In a moving environment such as an airplane, car, bus, or armored vehicle, PAC operations become moredifficult. Speech recognition can replace tedious data entry, and with domain specific grammar corrections it can ensure spoken data and dictation are recognized and entered properly. Adding a secondary device to complement speech input, which couldperform all the PAC operations efficiently and require no steady hand movements, would permit full use of a computer in moving vehicles. The use of a multiple-controller approach enables the interaction to be tailored to the task and environmentalconstraints, as well as user preferences. A natural language software-based speech recognizer in combination with eye-tracking, facial gesture, brain wave and neuromuscular sensor technologies are to be configured in a variety of ways to provide therequired functionality. Unique to this proposal is the development of an integrated set of human-computer control suite technologies using standard Application Program Interfaces (APIs) in the commercial Windows and Unix environment that are compatiblewith the Army Technical Architecture for efficient hands-free operation of computer systems. Some PAC needs could be met by using a combination of eye movement, facial muscle, and brain wave bio-potential combined technologies detected at the forehead, togenerate computer inputs. Both the frequency bands and electrical signal content levels derived from these sensors have shown progress in the selective mind and voluntary neuromuscular control of a computer display pointer and clicking on a displayedselection. Projective task analysis, driven by a decision support activity scenario, will be conducted for various combinations of interface technologies to derive an effective and feasible PAC conceptual design integrated with natural language.Quantitative performance and qualitative human interaction analysis will be documented and provided with the conceptual design.This research would be helpful for the military in controlling combat simulations and interacting with command and controlconsoles, it would also make a useful commercial gaming device and afford many handicapped individuals, who cannot efficiently use a mouse, trackball, or keyboard; the opportunity to access the next generation internet.