MsObjDET: AI-based Multi-Spectral Passive Object Detection using Spatiotemporal Transformer and YOLO with Dynamic Tracking Integration

The complexity of multi-observer and multi-spectral passive object detection in maritime environments limits the Navyís ability to track, classify, and identify surface and aerial targets with high confidence. Traditional methods suffer from sensor misalignment, environmental interference, and suboptimal spectral fusion, leading to detection inaccuracies and delayed situational awareness. In this project, InfoBeyond proposes MsObjDET, an AI-enhanced passive detection system that fuses multi-spectral sensor data from shipborne EO/IR arrays, UAV-mounted platforms, fixed maritime stations, and radar systems to enhance real-time object detection and tracking. At first, MsObjDET includes a Multi-Spectral Spatiotemporal Transformer to deeply extract extracts spectral and spatial features across signals. ÝThe Transformer employs a Spectral Competitive Mechanism to dynamically evaluate the reliability and relevance of spectral features under operational noises. ÝMeanwhile, a Spatiotemporal Attention is designed to capture temporal dynamics and spatial correlations, enabling the alignment of multi-spectral data streams across time and space. ÝSecondly, the YOLO Detection Head is proposed to identify and localize targets with high precision, addressing challenges such as small object detection, multi-target interference, and dynamic background noise. Further, a Dynamic Tracking & Integration module is proposed to learn a unified and high-confidence tracking Common Operational Picture (COP), addressing challenges such as trajectory fragmentation, temporal inconsistencies, and multi-target interactions. Superior to existing system, MsObjDET corrects spectral inconsistencies, compensates for platform motion over time, and integrates confidence-aware detections to provide precise object classification and tracking across Aegis-integrated systems. Designed for seamless deployment within the Aegis Combat System (ACS), including AN/SPY-1 radar and passive radar arrays, MsObjDET enables persistent surveillance of high-threat regions while mitigating sensor limitations.Ý