Improving automated lexical and discourse analysis of online chat dialog
Loading...
Authors
Forsyth, Eric N.
Subjects
Advisors
Martell, Craig H.
Date of Issue
2007-09
Date
Publisher
Monterey, California. Naval Postgraduate School
Language
Abstract
One of the goals of natural language processing (NLP) systems is determining the meaning of what is being transmitted. Although much work has been accomplished in traditional written and spoken language domains, little has been performed in the newer computer-mediated communication domain enabled by the Internet, to include text-based chat. This is due in part to the fact that there are no annotated chat corpora available to the broader research community. The purpose of our research is to build a chat corpus, initially tagged with lexical and discourse information. Such a corpus could be used to develop stochastic NLP applications that perform tasks such as conversation thread topic detection, author profiling, entity identification, and social network analysis. During the course of our research, we preserved 477,835 chat posts and associated user profiles in an XML format for future investigation. We privacy-masked 10,567 of those posts and part-of-speech tagged a total of 45,068 tokens. Using the Penn Treebank and annotated chat data, we achieved part-ofspeech tagging accuracy of 90.8%. We also annotated each of the privacy-masked corpus's 10,567 posts with a chat dialog act. Using a neural network with 23 input features, we achieved 83.2% dialog act classification accuracy.
Type
Thesis
Description
Series/Report No
Department
Organization
Naval Postgraduate School
Identifiers
NPS Report Number
Sponsors
Funder
Format
xiv, 111 p.;
Citation
Distribution Statement
Approved for public release; distribution is unlimited.