Deep Generative Modeling of Infrared Datasets for Aided Target Recognition

Current Machine Learning and Deep Convolutional Neural Networks offer outstanding performance in carrying out Computer Vision tasks, including Aided Target Recognition (AiTR), which is of paramount importance for a number of DoD applications. However, the data sets used to train and validate such algorithms are prevalently available in the visible wavelengths domain, whereas the Army uses a number of sensor systems that operate in mid-wave Infrared or long-wave Infrared. Furthermore, of all the branches of the military, the Army faces some of the most formidable challenges, having to operate on the ground, often in very cluttered environments that makes development and validation of AiTR algorithms more problematic. A potential solution to the lack of labeled data in the IR is to use computer-generated and yet physically realistic single-frames and videos of the targets of interest. While there are tools to perform such tasks, the output is still too pristine to be representative of the images and realistic phenomenology of IR imaging sensors. Thus, there is a need for a generative suite of algorithms which can artificially generate realistic IR video that can be conditionally modified to specific localities and target type to aid in the training of advanced machine learning tools for AiTR. To this end Lynntech is developing a Synthetic InfraRed Video Suite (SIRVS) that is a cutting-edge fast deep-learning-based solution to this need that performs multi-frame IR video generative tasks jointly while also imposing seamless video continuity.    During the Phase I effort Lynntech developed the proof of concept for a Simulated-to-Real (Sim2Real) tool that can enhance the realism of IR Video produced by the current Night Vision and Electronic Sensors Directorate (NVESD) computer-generated IR imagery. Having met the technology objectives in Phase I and demonstrated the ability to enhance simulated IR imagery in realistic ways, Lynntech proposes to advance to the Phase II by developing a prototype software tool that can be applied to a number of evaluation scenarios relevant to NVESD by reproducing specific sensor-type characteristics.  The ultimate aim is to develop advanced generative tools that can extend the capability of current automated systems by fully leveraging the advantages of deep learning by the creation of synthetic IR video datasets through advanced data augmentation of existing IR data and modelling resources by either style transfer (i.e. changing time of day or the environment) or by coherently adding novel prescribed objects or targets to a video sequence.