An Optimality Metrics Reporting Toolkit for SMART NAS

This SBIR project aims to develop a software module for the SMART NAS Test Bed (or another similar simulation environment) that allows an apples-to-apples comparison of system performance across scenarios and a comparison to a 'best possible' case. The module, named TOMO (Toolkit for Optimality Metrics Overlay), is a metrics toolkit for comparing SMART NAS simulation runs to the optimal decisions that should be made/should have been made relative to a selected metric to help compare the performance of the scenario being simulated to a best possible outcome, either in shadow-mode or in post-operations mode. The output from TOMO is not only a normalized metric, but the 4-D trajectories of all aircraft in an optimally-performing system. A key component to the success of any simulation environment is the quality of the metrics that it is able to report back to a user to allow informed decision-making. TOMO addresses the need to develop metrics that are comparable across scenarios by computing a 'baseline' for each scenario that represents the best that the system could perform given the operating constraints, weather, and demand. By normalizing metrics relative to this baseline, it allows for more direct comparisons across scenarios along multiple dimensions both in shadow mode and playback scenarios. When used in shadow mode, TOMO will identify the actions that should be taken to optimize for a given objective. In addition to computing the metrics, TOMO's output includes descriptive information on how the trajectories in the scenario being simulated differ from those in the optimal solution, and provides insight into how system performance may be improved.