Sparse Machine Learning Methods for Understanding Large Text Corpora

Download: .pdf

Authors: Laurent El Ghaoui, Guan-Cheng Li, Viet-An Duong, Vu Pham, Ashok Srivastava, Kanishka Bhaduri.

Status: Accepted for publication in Proc. Conference on Intelligent Data Understanding, 2011.

Abstract: Sparse machine learning has recently emerged as powerful tool to obtain models of high-dimensional data with high degree of interpretability, at low computational cost. This paper posits that these methods can be extremely useful for understanding large collections of text documents, without requiring user expertise in machine learning. Our approach relies on three main ingredients: (a) multi-document text summarization and (b) comparative summarization of two corpora, both using sparse regression or classification; (c) sparse principal components and sparse graphical models for unsupervised analysis and visualization of large text corpora. We validate our approach using a corpus of Aviation Safety Reporting System (ASRS) reports and demonstrate that the methods can reveal causal and contributing factors in runway incursions. Furthermore, we show that the methods automatically discover four main tasks that pilots perform during flight, which can aid in further understanding the causal and contributing factors to runway incursions and other drivers for aviation safety incidents.

Bibtex reference:

@inproceedings{ELDPSB:11,
  Author = {L. {El Ghaoui} and G.-C. Li and  V.-A. Duong and V. Pham and A. Srivastava and K. Bhaduri},
  Title = {Sparse Machine Learning Methods for Understanding Large Text Corpora},
  Booktitle={Proc.\ Conference on Intelligent Data Understanding},
  Year = {2011},
	Month = oct,
	note = {Accepted for publication, July 2011}
}