Automatic Classification of Objects in Captioned Depictive Photographs for Retrieval

Abstract

We investigate the robust classification of objects within photographs in a large and varied picture library of natural photographs. We assume the photographs have captions describing and locating only imprecisely some of the objects present in the picture, as is common in libraries. Our approach does not match to shape templates nor do full image picture understanding, neither of which works well for natural photographs where appearance varies considerably with lighting and perspective. Instead, we strike a robust compromise by statistically characterizing photograph regions with 17 key domain-independent parameters covering shape, color, texture, and contrast. We explored two ways to use the parameters to classify picture regions, case-based reasoning and a neural network, both of which require training. We found the neural network outperformed case-based reasoning, especially when we included caption information and a separate neuron inferring likelihood that a region was the "visual focus" of the picture. Then 25- category shape classification succeeded 48.1% of the time on a set of pictures randomly selected from a large picture library currently in use. Our work represents good progress on the difficult problem of retrieval by content from large real-world picture libraries.