(4) A Novel Approach to Recognizing Video Events with Varying Rhythms

In this proposal, Applied Research LLC (ARLLC) and the Arizona State University (ASU) propose to investigate the problem of recognizing long and complex events with varying action rhythms, which has not been considered in the literature but is a practical challenge. Our work is inspired in part by how humans identify events with varying rhythms: quickly catching frames contributing most to a specific event. We propose a two-stage end-to-end framework, in which the first stage selects the most significant frames while the second stage recognizes the event using the selected frames. Our model needs only event-level labels in the training stage, and thus is more practical when the sub-activity labels are missing or difficult to obtain. The results of extensive experiments show that our model can achieve significant improvement in event recognition from long videos while maintaining high accuracy even if the test videos suffer from severe rhythm changes. This demonstrates the potential of our method for real-world video-based applications, where test and training videos can differ drastically in rhythms of sub-activities.