Deblurring and Rolling Shutter Artifact Mitigation for fast Panning

Rolling shutters are a very common solution to achieve high-spatial resolution in CMOS imagers for a large variety of imaging and video applications. The tradeoff between speed of acquisition, sensitivity, and image quality makes this technology very common in a large variety of professional and consumer imaging devices. However, they produce certain typical image artifacts that present problems in image interpretation, under certain conditions. Some Navy fast-panning periscopes also employ this technology and traditional artifact/deblurring techniques are not well suited to restore a level of image quality that is required for fast and correct perception by human operators and automatic systems alike. This is due to the lack of accurate inertial measurements or to the lack of image textures typical of maritime environments to perform inter-frame registration and deconvolution. In order to mitigate these effects and restore the Navy’s personnel ability to perform accurate assessment and  measurements to perform Intelligence, Surveillance, and Reconnaissance (ISR) missions, Lynntech proposes combining traditional Computer Vision techniques based on motion compensation from sensors, with fast-and-accurate image deconvolution performed through modern AI techniques, such as a Conditional Generative Adversarial Network (CGAN) used to compensate for rolling-shutter image artifacts. Our novel Deep Neural Network architecture and training approach allow us to overcome the limitations of classical GANs in the ability to generalize and their being prone to their own artifact creation, through a training methodology that robustly combines simulated and real-world data and by the tuning of a perception function specifically for the domain at hand. Lynntech’s team is uniquely qualified to perform this project due to the many years of experience in Computer Vision and the unique expertise in image enhancement and adversarial examples that our company possess. Through a collaboration with a major defense contractor, we have already established a viable transition path and the ability to access training data from imaging systems very similar to those used by the U.S. Navy. In addition, our proven record of technology transition and the technological excellence of our solutions reduces risk for the future deployment of this vital technology for the Navy (e.g. PMS 435) and the DoD as a whole. The technology is expected to be low cost, leveraging existing systems and hardware, thus enhancing existing capabilities that can be made available to the warfighters in a short time span, since Lynntech expects the prototype in a relevant environment to be proven in Phase I and be ready for field tests at the end of the Phase II. The technology developed with this project will also be of immediate transferability to the commercial sector where many more consumer products and professional imaging systems can benefit from rolling-shutter effects mitigation.