EVALUATION METHODS AND METRICS FOR IMAGE PROCESSING AND UNDERSTANDING SYSTEMS

THE PROJECT PROPOSED WILL DEVELOP A SUITE OF METRICS FOR THE EVALUATION OF OVERALL PERFORMANCE OF IU SYSTEMS, AS WELL AS FOR THE ASSESSMENT AND DIAGNOSIS OF INDIVIDUAL SYSTEM COMPONENTS. THE METRICS WILL ADDRESS THE ISSUES OF THE GENERALITY AND STANDARDIZATION OF METRICS TO PERMIT CROSS-SYSTEM COMPARISONS. THE DIMENSIONS TO BE INVESTIGATED FOR METRICS DEVELOPMENT WILL INCLUDE: ACCURACY, COMPATIBILITY, CONFORMITY, COMPUTATION EFFICIENCY, COST/BENEFIT, EASE OF USE, FLEXIBILITY, ROBUSTNESS, SPEED, TRANSPORTABILITY, AND UTILITY. INDIVIDUAL METRICS WILL BE DEVELOPED FOR THE FOLLOWING SYSTEM COMPONENTS: SCENE REGISTRATION, IMAGE FEATURE EXTRACTION, SCENE OBJECT MODELLING, THE KNOWLEDGE INTERFACE, AND THE CONTROL TECTURE. THE PROPOSED METRICS WILL BE REVIEWED FOR CONFORMANCE TO TECHNICAL STANDARDS, SUCH AS PRECISION AND VALIDITY, AND FOR OVERALL UTILITY. DETAILED PROCEDURES WILL BE PROVIDED FOR EACH METRIC, TO ENSURE APPROPRIATE APPLICATION. IN ADDITION TO THE METRICS THEMSELVES, WE PROPOSE TO RECOMMEND A WEIGHTING SYSTEM, THAT WILL PERMIT END-USERS AND DEVELOPERS TO AGGREGATE MULTIPLE METRICS INTO A SINGLE MEASURE, REFLECTING THE IMPORTANCE THAT THEY ATTACH TO THE INDIVIDUAL DIMENSIONS AND COMPONENTS. FINALLY, WE WILL DEVELOP A PROPOSED ARCHITECTURE FOR A DATABASE SUPPORTING INFORMATION ACCESS TO METRICS AND PROCEDURES, AS WELL AS TO DATA FROM OTHER EVALUATIONS, AND EVENTUALLY TO BENCHMARK RESULTS. THERE ARE TWO MAJOR BENEFITS ANTICIPATED. THE FIRST IS THE DEVELOPMENT OF A STANDARD SET OF UNBIASED METRICS THAT WILL PERMIT CROSS-SYSTEM COMPARISONS. THIS WILL BE PARTICULARLY USEFUL FOR PROCUREMENT DECISIONS. THE SECOND IS THE DEVELOPMENT OF A SET OF STANDARD METRICS FOR THE EVALUATION OF SYSTEM COMPONENTS THAT WILL PERMIT BOTH PERFORMANCE ASSESSMENTS OF SEPARATE COMPONENTS, AS WELL AS DIAGNOSTIC EVALUATION OF THESE COMPONENTS, AND IDENTIFICATION OF FAILURE POINTS AND AREAS MOST LIKELY TO BENEFIT FROM IMPROVEMENT.