AS THE DOD IS MOVING TOWARDS THE LARGE-SCALE USE OF ARTIFICIAL INTELLIGENCE SYSTEMS FOR SUPPORTING DECISION-MAKING AND INFORMATION ANALYSIS, IT BECOMES INCREASINGLY IMPORTANT THAT WE DEVELOP MECHANISMS FOR ASSESSING THE PERFORMANCE OF THESE SYSTEMS.

AS THE DOD IS MOVING TOWARDS THE LARGE-SCALE USE OF ARTIFICIAL INTELLIGENCE SYSTEMS FOR SUPPORTING DECISION-MAKING AND INFORMATION ANALYSIS, IT BECOMES INCREASINGLY IMPORTANT THAT WE DEVELOP MECHANISMS FOR ASSESSING THE PERFORMANCE OF THESE SYSTEMS. WE BELIEVE THAT ASKING "HOW INTELLIGENT IS THE SYSTEM?" IS NOT THE CORRECT PERFORMANCE QUESTION AND THAT MORE APPROPRIATE ARE "DOES THE SYSTEM HELP THE USER?" "IN WHAT WAY?" AND "HOW CAN THIS BE MEASURED?" OUR APPROACH IS BASED ON THE PREMISE THAT THE USERS, DEVELOPERS, MANAGERS AND FUNDERS OF THESE SYSTEMS HAVE "EXPERT" KNOWLEDGE ABOUT PERFORMANCE AND EVALUATION THAT NEEDS TO BE CAPTURED IF A COMPREHENSIVE EVALUATION IS TO BE PERFORMED. THE METHODOLOGY WE PROPOSE STRUCTURES THIS KNOWLEDGE BY USING A HIERARCHY OF EVALUATION CONTEXTS, USES A RULE-BASED LANGUAGE FOR EXPRESSING THE KNOWLEDGE, AND EMPLOYS THE MATHEMATICS OF FUZZY SET THEORY TO MODEL THE INHERENT VAGUENESS IN THE KNOWLEDGE. THE PROPOSAL CONTAINS AN OVERVIEW OF OUR EVALUATION METHODOLOGY AND ILLUSTRATES IT WITH A SIMPLE EXAMPLE. WE ALSO DISCUSS THE NEED TO DESIGN A SET OF TOOLS TO HELP OPERATIONALIZE THE METHODOLOGY, AND CONSIDER SOME PILOT EXPERIMENTS THAT SHOULD BE PERFORMED TO VALIDATE THE IDEAS.