Finding and labeling the subject of a captioned depictive natural photograph
Loading...
Authors
Rowe, Neil C.
Subjects
information retrieval
multimedia
caption
subject
photograph
image processing
segmentation
background
natural-language understanding
depiction
multimedia
caption
subject
photograph
image processing
segmentation
background
natural-language understanding
depiction
Advisors
Date of Issue
2002
Date
January / February 2002
Publisher
Monterey, California. Naval Postgraduate School
Language
Abstract
We address the problem of finding the subject of a photographic image intended to illustrate some physical
object or objects ("depictive") and taken by usual optical means without magnification ("natural"). This could
help in developing digital image libraries since important image properties like subject size and color of a
photograph are not usually mentioned in accompanying captions and can help rank the photograph retrievals
for a user. We explore an approach that identifies "visual focus" of the image and "depicted concepts" in a
caption and connects them. Visual focus is determined using eight domain-independent characteristics of
regions in the segmented image, and caption depiction is identified by a set a rules applied to the parsed and
interpreted caption. Visual-focus determination also does combinatorial optimization on sets of regions to
find the set that best satisfies focus criteria. Experiments on 100 randomly selected image-caption pairs show
significant improvement in precision of retrieval over simpler methods, and particularly emphasize the value
of segmentation of the image.
Type
Conference Paper
Description
This paper appeared in IEEE Transactions on Knowledge and Data Engineering, Vol. 14, No. 1 (January/February 2002), pp. 202-207.
Series/Report No
Department
Organization
Identifiers
NPS Report Number
Sponsors
supported by the U.S. Army Artificial Intelligence Center, and by the U. S. Naval Postgraduate School
Funder
funds provided by the Chief for Naval Operations
Format
Citation
Distribution Statement
Approved for public release; distribution is unlimited.