Regression analysis for massive datasets

Tsai Hung Fan, Dennis K.J. Lin, Kuang Fu Cheng

Research output: Contribution to journalArticlepeer-review

20 Scopus citations


In the past decades, we have witnessed a revolution in information technology. Routine collection of systematically generated data is now commonplace. Databases with hundreds of fields (variables), and billions of records (observations) are not unusual. This presents a difficulty for classical data analysis methods, mainly due to the limitation of computer memory and computational costs (in time, for example). In this paper, we propose an intelligent regression analysis methodology which is suitable for modeling massive datasets. The basic idea here is to split the entire dataset into several blocks, applying the classical regression techniques for data in each block, and finally combining these regression results via weighted averages. Theoretical justification of the goodness of the proposed method is given, and empirical performance based on extensive simulation study is discussed.

Original languageEnglish
Pages (from-to)554-562
Number of pages9
JournalData and Knowledge Engineering
Issue number3
StatePublished - Jun 2007


  • Best linear unbiased estimator
  • Minimum variance
  • Optimal weight


Dive into the research topics of 'Regression analysis for massive datasets'. Together they form a unique fingerprint.

Cite this