VAST-CQA: Video Annotation and Statistics Toolkit for Crowdsourcing Quality Assessment

The Video-National Imagery Interpretability Rating Scale (VNIIRS) is a task-based scale that reflects observable semantic content in videos. Intelligence agencies such as the NGA use VNIIRS for applications such as efficient storage and retrieval, data browsing, and testing of automated video assessment systems. Existing methods for video quality assessment rely on expert analysts to view and tag videos. Unfortunately, this approach is time-consuming, especially considering the scale of data. Crowdsourcing is attractive as an efficient approach to quickly tag large amounts of data. However, the results obtained through crowdsourcing are usually of lower quality than those obtained by expert analysts, and are sometimes noisy, missing, or incomplete as VNIIRS ratings reflect subjective assessment of semantic content. \n\n During Phase I, we have demonstrated the novel VAST-CQA annotation toolkit that allows for rapid video annotation, multi-user capabilities, smart annotation assignment, and an evaluation of an automated VNIIRS assessment system. In Phase II, we propose to extend the user interface capabilities, enhance the smart selection to expand the active learning work begun in Phase I, and further streamline the processing workflow. IAI will build and deliver the VAST-CQA software that is ready for integration at NGA by the end of Phase II.