You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Tomasz Chmielewski <ma...@wpkg.org> on 2011/10/31 14:49:37 UTC

rebalancing data on disks?

Hi,

I have a HDFS cluster consisting of several hosts.

On each node, I add a new disk when the current capacity is close to full.

Right now, every server has more or less such distribution of data:

/dev/sdf              493G  468G   51M 100% /data1
/dev/sdg              493G  468G   51M 100% /data2
/dev/sdh              493G  103G  365G  22% /data3
/dev/sdi              493G  100G  368G  22% /data4


So, /dev/sdf and /dev/sdg almost 100% full, and there is lots of free 
space on /dev/sdh and /dev/sdi.

Disks which are 100% full don't make monitoring very happy.

Is it possible to rebalance data on the disks on one HDFS server (or, 
more servers)?


"hadoop balancer" will want to rebalance data between the servers, but 
not between the disks on one server.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: rebalancing data on disks?

Posted by Harsh J <ha...@cloudera.com>.
Which is why you need your HDFS shutdown before this is done.

See http://search-hadoop.com/m/fSof91EYbe9 and http://wiki.apache.org/hadoop/FAQ#On_an_individual_data_node.2C_how_do_you_balance_the_blocks_on_the_disk.3F

On 31-Oct-2011, at 8:39 PM, Tomasz Chmielewski wrote:

> On 31.10.2011 15:37, Jeff Bean wrote:
>> IIUC you can move the data block files from one disk to another until they are evenly distributed. Give it a shot.
> 
> Just like that? Using "mv" command?
> 
> I.e. move /data2/hdfs/data/current/subdir44/* to /data3/hdfs/data/current/subdir44/
> 
> ?
> 
> 
> It's not atomic, so at some point, /data3/ would contain a file which is not a full copy of the same file on /data2/; later, both /data2 and /data3 would contain identical file.
> 
> 
> -- 
> Tomasz Chmielewski
> http://wpkg.org
> 


Re: rebalancing data on disks?

Posted by Tomasz Chmielewski <ma...@wpkg.org>.
On 31.10.2011 15:37, Jeff Bean wrote:
> IIUC you can move the data block files from one disk to another until they are evenly distributed. Give it a shot.

Just like that? Using "mv" command?

I.e. move /data2/hdfs/data/current/subdir44/* to 
/data3/hdfs/data/current/subdir44/

?


It's not atomic, so at some point, /data3/ would contain a file which is 
not a full copy of the same file on /data2/; later, both /data2 and 
/data3 would contain identical file.


-- 
Tomasz Chmielewski
http://wpkg.org


Re: rebalancing data on disks?

Posted by Jeff Bean <jw...@cloudera.com>.
IIUC you can move the data block files from one disk to another until they are evenly distributed. Give it a shot.

Sent from my iPad

On Oct 31, 2011, at 6:49, Tomasz Chmielewski <ma...@wpkg.org> wrote:

> Hi,
> 
> I have a HDFS cluster consisting of several hosts.
> 
> On each node, I add a new disk when the current capacity is close to full.
> 
> Right now, every server has more or less such distribution of data:
> 
> /dev/sdf              493G  468G   51M 100% /data1
> /dev/sdg              493G  468G   51M 100% /data2
> /dev/sdh              493G  103G  365G  22% /data3
> /dev/sdi              493G  100G  368G  22% /data4
> 
> 
> So, /dev/sdf and /dev/sdg almost 100% full, and there is lots of free space on /dev/sdh and /dev/sdi.
> 
> Disks which are 100% full don't make monitoring very happy.
> 
> Is it possible to rebalance data on the disks on one HDFS server (or, more servers)?
> 
> 
> "hadoop balancer" will want to rebalance data between the servers, but not between the disks on one server.
> 
> 
> -- 
> Tomasz Chmielewski
> http://wpkg.org