Persuasion detection in conversation

Abstract

In this thesis, we present a system for annotating persuasion in conversation based on a social-psychological model. We augmented the social model developed by James Cialdini with some of our own categories for annotators to label. The conversations consisted of 37 hostage negotiation transcripts from private and public sources, with all personal information removed from the private source transcripts. We evaluated the level of agreement between annotators using Cohen's Kappa measurement. Our initial results showed only fair to moderate agreement, with an average kappa score around 0.41 for transcripts of significant length (over 200 utterances). Based on these results, annotators revised the annotation model and eliminated some categories of persuasion while adding "other" as a catchall for any persuasive utterance not covered by Cialdini. The revised model showed a significant increase in agreement with an average kappa score of 0.78 for transcripts of significant length (over 200 utterances). Based on this revised model, annotators adjudicated a final persuasion corpus for the 37 transcripts that will be used in future works on persuasion detection.