Finding and labeling the subject of a captioned depictive natural photograph
Rowe, Neil C.
MetadataShow full item record
We address the problem of finding the subject of a photographic image intended to illustrate some physical object or objects ("depictive") and taken by usual optical means without magnification ("natural"). This could help in developing digital image libraries since important image properties like subject size and color of a photograph are not usually mentioned in accompanying captions and can help rank the photograph retrievals for a user. We explore an approach that identifies "visual focus" of the image and "depicted concepts" in a caption and connects them. Visual focus is determined using eight domain-independent characteristics of regions in the segmented image, and caption depiction is identified by a set a rules applied to the parsed and interpreted caption. Visual-focus determination also does combinatorial optimization on sets of regions to find the set that best satisfies focus criteria. Experiments on 100 randomly selected image-caption pairs show significant improvement in precision of retrieval over simpler methods, and particularly emphasize the value of segmentation of the image.
This paper appeared in IEEE Transactions on Knowledge and Data Engineering, Vol. 14, No. 1 (January/February 2002), pp. 202-207.
Showing items related by title, author, creator and subject.
A caption-based natural-language interface handling descriptive captions for a multimedia database system Dulle, John David (Monterey, California. Naval Postgraduate School, 1990-06);This research examined the grammar structure of descriptive English captions on multimedia data. The research was composed of three phases. The first phase was to investigate the grammar structure of example descriptive ...
Exploiting Captions for Multimedia Data Mining / Chapter in Encyclopedia of Multimedia Technology and Networking Rowe, Neil C. (Monterey, California. Naval Postgraduate School, 2005);Captions are essential accompaniments to multimedia data objects as a way to facilitate their data mining. This article describes the kinds of possible captions and the task of recognizing them. It then discusses the forms ...
Rowe, Neil C. (Monterey, California. Naval Postgraduate School, 1998-07);We discuss the obstacles to inference of correspondances between objects within photographic images and their counterpart concepts in descriptive captions of those images. This is important for information retrieval of ...