What friends are for: collaborative intelligence analysis and search

Abstract

Intelligence analysts face a glut of information and limited time to identify which information is relevant. Also, they are unaware of other analysts with similar intelligence problems, preventing collaboration and often causing intelligence failure. To identify relevant information, analysts use adopted commercial search engines designed for internet-sized databases containing hyperlinked web-pages that are not effective on intelligence databases consisting of non-hyperlinked documents. This thesis outlines a model to fundamentally increase search effectiveness and collaboration by using a social network of like-minded users based on user biographies and search behavior. After entering a query, the likelihood of returning a relevant document is increased by leveraging data from other, similar users. The model goes beyond standard search engine design by presenting similar analysts for collaboration and presenting relevant documents without queries. Our framework is mathematically grounded in a Markov random field information retrieval model and recent developments in recommender systems. We build and test a prototype system on datasets from the National Institute of Standards & Technology. The test results combine with computational sensitivity analyses to show significant improvements over existing search models. The improvements are shown to be robust to high levels of human error and low similarity between users.