Theresa Wilson, presenting joint work with Janyce Wiebe and Rebecca Hwa.
This will be a practice talk for AAAI 2004.
Abstract: There has been a recent swell of interest in the automatic identification and extraction of opinions and emotions in text.
In this paper, we present the first experimental results classifying the strength of opinions and other types of subjectivity and classifying the subjectivity of deeply nested clauses. We use a wide range of features, including new syntactic features developed for opinion recognition. In 10-fold cross-validation experiments using support vector regression, we achieve improvements in mean-squared error over baseline ranging from 57\% to 64\%.
I have created an internal page which will host presentations and related materials for our weekly meetings. The page address is:
http://nlp.cs.pitt.edu/presentations/
Kappa is the primary statistic used in NLP research to evaluate agreement among raters. However, there are many problems with the kappa statistic. In this talk I will discuss kappa and how to account for problems not addressed by kappa with different statistics. I will also describe how to calculate a generalizability coefficient that measures the reliability or reproducability of a reference standard created from human raters. I will use data from a current study we are evaluating to help understand how all the agreement statistics can help answer the question, "How good is my reference standard?"
Rebecca Hwa
This is not so much a talk but a round-table discussion that I'd like to host. With the conference season fast approaching, it might be good for us to get together and trade ideas on giving presentations.
Diane Litman and Kate Forbes-Riley
We compare the learning gains from tutoring with spoken versus typed dialogue. In one experiment, the tutor was a human. In the other experiment, the tutor was a tutoring system. The main results of our study are that changing the modality from text to speech caused large differences in the learning gains, time and superficial dialogue characteristics of human tutoring, but for computer tutoring, it made less difference. (This is material that will be presented at the Intelligent Tutoring Systems Conference).
We now have a repository for NLP related conference papers.
Please note that due to copy right restrictions, this page is only accessible within the relevant subdomains of pitt.edu (cs, isp, lrdc, etc.).
If you're not able to access the page from your domain (inside Pitt), please contact Behrang.
See the "Teaching Computers to Teach Like Humans" article in the June 7, 2004 Pitt Chronicle!
http://www.discover.pitt.edu/media/pcc/comps_like_humans.html
Also visit Pitt's website (www.pitt.edu) to see a press release from June 3 2004 about our research.
The text of the release is also below.
FOR IMMEDIATE RELEASE
June 3, 2004
Contact: Patricia Lomando White
412-624-9101
laer@pitt.edu
Pitt Researchers Developing Computers That Teach Like Humans
Natural language recognition key to improved tutoring by machines
PITTSBURGH—While new federal education rules emphasizing testing and standards have fueled a tutoring boom, relatively few pupils enjoy access to effective but costly one-on-one teaching. In an effort to spread the intellectual wealth, scientists at the University of Pittsburgh’s Learning Research and Development Center (LRDC) are working to bring individual instruction to all students.
With $2.5 million from the National Science Foundation (NSF), principal investigator (PI) Kurt VanLehn, a Pitt computer science professor and LRDC senior scientist, is working to build less expensive computer tutors as good as their more expensive human counterparts. Looking specifically at the best ways to teach and learn physics, VanLehn and his colleagues are probing both tutor and student behavior.
“The computer tutors available in stores today just tell you if your answer is right or wrong,” VanLehn said. “With a human tutor, though, students can do much more,” including discussing their reading with the tutor and getting help solving longer, more complex problems.
A major difference between human and computer tutors has been that only human tutors understand unconstrained natural language—the conversational, open-ended give-and-take that can often flummox the smartest software.
Today, commercial educational technology involves two response formats: multiple choice and mathematical formulas. If all goes as planned, a tutoring program should be on the market in five to 10 years that can handle open-ended questions and analyze the students’ text or speech responses.
The LRDC team’s basic approach to improving computer tutoring is to simply study and learn from interactions between humans and computer tutors. As more effective dialogue strategies are identified, they will be incorporated into a natural language-based tutoring system.
LRDC’s new tutoring venture builds on a recently completed five-year, $5 million NSF-funded Center for Interdisciplinary Research on Constructive Learning Environments, led by VanLehn. The center developed several prototypes of natural language tutoring systems both at LRDC and at Carnegie Mellon University. The center also developed tools for building more such tutors.
Capitalizing on LRDC’s ability to attract and link researchers from a wide variety of disciplines, the computer tutor study includes researchers specializing in the cognitive psychology of human tutoring, the technology of natural language processing, and the design of effective tutoring systems.
The Co-PIs are Diane J. Litman, a Pitt computer science professor and LRDC research scientist; Michelene Chi, a Pitt psychology professor and LRDC senior scientist; Pamela W. Jordan, a LRDC research associate; and Carolyn P. Rose, a research scientist at Carnegie Mellon.
The group’s grant is administered under NSF’s Information Technology Research program, which supports innovative multidisciplinary research that extends the frontiers of information technology, leads to new and unanticipated technologies, creates revolutionary applications, or provides alternative approaches to complete important activities.
###
6/3/04/tmw
Co-training for Predicting Emotions with Spoken Dialogue Data Beatriz Maeireizo, Diane Litman, Rebecca Hwa
Jan Wiebe will describe a new project entitled Opinions in Question Answering. The project is part of the ARDA AQUAINT Question Answering program, and is joint with Claire Cardie at Cornell and Ellen Riloff at Utah. The goals of the project are to extract detailed information about opinions from text and then create summary representations of the opinions expressed about a topic in one or many documents.