Extracting Valid Inferences from Data, Evidence, and Novel Tests (EVIDENT)

ABSTRACT: A recent study by the National Research Council found that sophisticated statistical techniques are finding their way into DoD operational test design, but that current DoD practices are still substantially behind the state of the art. The study concluded this has resulted in inefficient test designs, wasted resources, and less effective acquisition decision making. US Air Force test and evaluation (T & E) organizations need a toolkit that gives analysts access to sophisticated statistical techniques. To help T & E analysts design more efficient tests and draw statistically defensible inferences from test data that cannot be analyzed using basic statistical techniques, we propose to design a toolkit for Extracting Valid Inferences from Data, Evidence, and Novel Tests (EVIDENT). EVIDENT will help Air Force T & E personnel: (1) assess the feasibility and technical risk of comparing test data sets; (2) compare test results that are known to reflect bias; (3) analyze the results of tests conducted over time or in a sequence; and (4) design tests when it is impractical to randomly select test subjects or generate large samples to support the usual confidence levels. BENEFIT: We will pursue a two-part plan that transitions the technologies in the EVIDENT toolkit to Government and military customers, and that extends our existing AgentWorks commercial product to increase its appeal to a number of new markets and customers. The technology in the EVIDENT toolkit will be of value to any analyst who must design efficient tests or draw statistically defensible conclusions from data that cannot be analyzed using basic statistical techniques. Therefore, we expect the full-scope EVIDENT toolkit to offer immediate and tangible benefits for any DoD organization involved in T & E planning or analyzing test data. Our main approach to commercializing the technologies developed under this program is to incorporate them into our AgentWorks product, which will increase its appeal as a commercial product and enable us to use the tool to provide consulting services based on AgentWorks to customers in DoD, other Federal agencies, and commercial markets.