Automatic scene understanding and generation of textual descriptions

We address an important yet challenging image processing problem: automatic scene understanding and generating textual description. Answers to the following two questions would be needed to provide a good semantic description of a scene: (i) What objects are present in the scene? (ii) Where are they located in the scene with respect to each other? The answer to what is defined by nouns in a language, and where is defined by prepositions. We present a novel semi-automated method to provide labels to the image regions, thus providing answers to what. This approach uses low-level visual features and clustering techniques to assist a human in quickly training a labeling system. Note that this is an offline system; no human intervention would be required at the time of actual operation of the system. In order to define where the objects are with respect to each other, we plan to create grammar rules for the particular prepositions of interest, which would establish context-based relationship between regions of the image. The region attributes and their relationships are then used to make higher level semantic interpretations about the scene followed by generating textual descriptions.