CIS SIG-NLP Praveen Chandar Ravichandran
This is a past event
Speaker: Praveen Chandar Ravichandran
Title: Annotation Strategies for Clustering Answer Snippets
Abstract: Intelligence analysts are often faced with scenarios where they are confronted by informal textual sources of a social nature, and would like to study relationships among people involved in the discussion, points of view expressed regarding a specific event, and their relative weight or frequency. Typically, a system built to aid analysts facing such scenarios would return a list of relevant answer snippets for a given natural language question. Often, the questions are open ended in nature i.e. address various aspects of the topic. Therefore, it is a good idea to cluster (group) the returned answer snippets based on some criteria.
In this work, we take the initial steps into investigating how human subjects cluster a list of answer snippets for a given question. The clustering task is hard even for humans. We investigate the influence of annotation strategies in obtaining consistent annotations for the clustering task. We study the effect of using different interfaces by comparing a simple labeling interface with a pairwise clustering interface. Also, it is not clear if the human annotators prefer to generate the cluster labels themselves or if they prefer pre-defined cluster labels. We conducted a user study with two annotators and present the details of the study along with the results.
This is a joint work with Hema Raghavan, Ramesh Nallapati, Vittorio Castelli and Radu Florian (IBM T. J. Watson Research Center)
Monday, September 30, 2013 at 1:25pm
to 2:15pm
Smith Hall, Room 102A
Smith Hall, University of Delaware, Newark, DE 19716, USA