Load Rebalancing for Heterogeneous Distributed System

Abstract

     Distributed file systems are basic building block for cloud environment based on the Map Reduce programming model. In distributed file systems nodes simultaneously serve storage and computing function. When the file upload, it is partitioned in to number of chunks and allocate to distinct nodes so that map reduce can be performed in parallel among the nodes. In cloud atmosphere disaster can arise, nodes can be added, deleted, upgraded and replaced in the system. In accumulation to this files can be dynamically created, deleted and appended. It might result in load disproportion problem in distributed file system, means file chunk are not distributed uniformly between the nodes. To overcome this problematic a distributed load rebalancing algorithm is proposed. The aim is to assign file chunk as uniformly as possible among the existing node in the Hadoop cluster. The distributed load rebalancing algorithm is incorporated in Hadoop Distributed File System (HDFS) by using cluster environment. The proposed Load rebalancing algorithm decrease the movement cost and reallocate the file chunk uniformly between the nodes. This process is repeated until all node balance.