A flexible, web-based system for design, support, and delivery of adaptive tests

DESCRIPTION (provided by applicant): Awareness of the enormous advantages of computerized adaptive testing (CAT) in an item response theory framework (IRT) is growing rapidly in the behavioral research community. CAT has the capacity to significantly reduce respondent burden without losing measurement reliability. Although a small number of computerized adaptive tests (CATs) exist for behavioral measurement, they have been designed as single-use CATs which were developed to assess a fixed set of domains. The fact that currently available CATs cannot be easily altered by individuals other than the original developers to incorporate new domains severely limits the usability of the CATs beyond their initial purpose. Thus, despite the growing awareness of this state-of-the-art methodology, it is not currently feasible for the majority of behavioral researchers to implement CAT in their research designs. The construction and maintenance of CATs requires ongoing attention by individuals with extensive psychometric and computing experience. The few available software products for creating CATs are also extremely limited: they are PC-based so necessary software must be deployed on each computer that is going to be used for CAT administration; once software is installed on a PC that PC must be taken to the participants or the participants must be brought to the PC; subsequent software updates require internet connectivity or physical media; and these programs have extensive system requirements. The proposed project aims to overcome these barriers to CAT, making CAT an available tool for any behavioral researcher with access to the internet. [At the completion of Phase II] of the proposed project, a flexible web-based CAT system will be available so that any researcher can log-on to a web-site and create an adaptive test. [As a final goal,] the system will be flexible enough that users can, if desired, create and administer CATs remotely and without VPG assistance. However, researchers wishing to use this system will have the VPG staff available to them to help create and administer CATs, and to incorporate the resulting measures into their research design. The completion of Phase I will result in a prototype web-based CAT system. This [basic] prototype will be designed to receive input from a user through upload of files or direct user entry, incorporate the user input into a CAT algorithm implemented with IRT models, administer an adaptive test, and generate output for direct printing or download by the user. [The focus of Phase I is on the implementation of the core CAT algorithms and basic support input/output systems as dynamic web applications.]Phase II of the proposed project will solicit feedback from a wide array of potential users to expand the prototype, focusing on usability and continued flexibility, and will include pilot testing and refinement to ensure the web-based CAT system and support services are designed to fully meet the needs of behavioral research and assessment communities. PUBLIC HEALTH RELEVANCE: Computerized adaptive testing (CAT) has the capacity to significantly reduce respondent burden without any loss of reliability. However, CAT technology is currently accessible only to those with extensive psychometric knowledge thereby limiting the majority of behavioral researchers from implementing CAT in their own research. The proposed project aims to overcome these barriers to access by creating a flexible web-based CAT system that will allow behavioral researchers to incorporate this state-of-the-art methodology into their study designs.