LaSA: A locality-aware scheduling algorithm for Hadoop-MapReduce resource assignment

Tseng Yi Chen, Hsin Wen Wei, Ming Feng Wei, Ying Jie Chen, Tsan Sheng Hsu, Wei Kuan Shih

研究成果: 書貢獻/報告類型會議論文篇章同行評審

27 引文 斯高帕斯(Scopus)

摘要

Cloud computing has become more popular for a decade; it has been under continuous development with advances in architecture, software, and network. Hadoop-MapReduce is a common software framework processing parallelizable problem across big datasets using a distributed cluster of processors or stand-alone computers. Cloud Hadoop-MapReduce can scale incrementally in the number of processing nodes. Hence, the Hadoop-MapReduce is designed to provide a processing platform with powerful computation. Network traffic is always a most important bottleneck in data-intensive computing and network latency decreases significant performance in data parallel systems. Network bottleneck is caused by network bandwidth and the network speed is much slower than disk data access. So that, good data locality can reduces network traffic and increases performance in data-intensive HPC systems. However, Hadoop's scheduler has a defect of data locality in resource assignment. In this paper, we present a locality-aware scheduling algorithm (LaSA) for Hadoop-MapReduce scheduler. Firstly, we propose a mathematical model of weight of data interference in Hadoop scheduler. Secondly, we present the LaSA algorithm to use weight of data interference to provide data locality-aware resource assignment in Hadoop scheduler. Finally, we build an experimental environment with 3 cluster and 35 VMs to verify the LaSA's performance.

原文???core.languages.en_GB???
主出版物標題Proceedings of the 2013 International Conference on Collaboration Technologies and Systems, CTS 2013
頁面342-346
頁數5
DOIs
出版狀態已出版 - 2013
事件2013 International Conference on Collaboration Technologies and Systems, CTS 2013 - San Diego, CA, United States
持續時間: 20 5月 201324 5月 2013

出版系列

名字Proceedings of the 2013 International Conference on Collaboration Technologies and Systems, CTS 2013

???event.eventtypes.event.conference???

???event.eventtypes.event.conference???2013 International Conference on Collaboration Technologies and Systems, CTS 2013
國家/地區United States
城市San Diego, CA
期間20/05/1324/05/13

指紋

深入研究「LaSA: A locality-aware scheduling algorithm for Hadoop-MapReduce resource assignment」主題。共同形成了獨特的指紋。

引用此