SBIR Phase II: Low latency and ultra-high quality 360 video streaming platform for highly immersive VR experiences

The broader/commercial impact of this SBIR Phase II project is the first video streaming solution providing ultra-high-quality video with low latency for Virtual/Augmented/Extended Reality (VR/AR/XR) applications. Such applications are revolutionizing innumerable fields, with a projected market volume of $10.3 Billion in 2024 in the U.S. Yet, in reality the potential of VR/AR/XR remains untapped due to a remarkable limitation of existing video streaming solutions: high-quality videos entail huge latencies of 30-50 seconds, while low latency is only achieved with low quality video. However, both are important factors in VR/AR/XR, since high video quality is essential to generate a truly immersive experience for users, and low latency is essential to allow for live virtual interaction. The proposed technology promises to overcome this problem and unleash the full potential of VR/AR/XR, through advanced applications in education, business, healthcare, defense, leisure, and more. This proprietary technology is the result of years of R&D, making it difficult for competitors to match its superior performance. If successful, this technology will represent a key commercial milestone for the proposing company, with forecasted annual revenues of $74.5 million by year three of commercialization, through a business model based on pay-per-use fees. This Small Business Innovation Research Phase II project seeks to develop a video streaming solution that can manage 360° videos with ultra-high quality (>8K at a minimum of 60 fps) at low latencies (<1 second), enabling seamlessly switching between multiple cameras and audio tracks. The trade-off between video quality and streaming latency presents the main barrier for truly immersive and interactive Virtual/Augmented/Extended Reality (VR/AR/XR) experiences. Thus, this project’s research objectives are focused on: 1) extending the existing Android solution to iOS devices; 2) low latency composed experiences, supporting synchronized content from multiple cameras; 3) supporting multiple audio tracks within one stream; 4) platform validation, benchmarking various cloud service instances; 5) documenting the workflow and establishing a procedure for third party operators to deploy the end-to-end system; 6) determining Graphic Processing Units that meet the systems performance requirements; and 7) extending the platform to new generation XR devices. By the end of Phase II, the goal is to provide a platform that enables high-resolution 360° video streaming with latencies of less than two seconds, demonstrating sufficient maturity in each of its components, and to have clearly defined the hardware specifications that are required to provide optimal AR/VR/XR experiences. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.