Auto-Transcription for Citizen Science

We propose to develop a system for transcribing semi-structured data, including tables, from handwritten or typed document images. Our objective is to design an end-to-end system that takes document images as inputs and extracts a digital, tabular output. The proposed approach employs a joint reasoning architecture where optical character recognition and structure recognition is combined in a single neural network to achieve high accuracy. In addition, our approach will allow users to rapidly fix any extraction mistakes in a “example-based” manner, so that working as a man-machine team the complete task can be efficiently accomplished, even in cases where the system does not initially produce a perfect result. Our aim is to empower citizen scientists to quickly and easily participate in the extraction process in an intuitive way.