Retrieving Captioned Pictures Using Statistical Correlations and a Theory of Caption-Picture Co-reference

Abstract

The MARIE project is investigating new methods for efficient information retrieval of captioned multimedia from multimedia libraries. Captions are essential to understanding multimedia and to finding relevant examples quickly. Our approach, shared by (Srihari, 1994), is to analyze both the caption and the picture in advance, then match user English queries to these semantic networks in a way that maximizes retrieval speed. Our theory and algorithms are being tested on a particular example, 100,000 captions constituting the entire unclassified portion of the photographic library at NAWC-WD, the U.S. Navy test facility in China Lake, California, USA (Rowe and Guglielmo, 1993), plus some of the pictures. Our focus is thus on technical captions and technical pictures that required specialized domain-dependent knowledge to interpret. (Rowe and Guglielmo, 1993) shows experimental results that confirm that our approach to caption-only processing gets higher precision for similar recall than a keyphrasematching system for the same application.