You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Billy Pearson <sa...@pearsonwholesale.com> on 2009/01/20 07:28:27 UTC
hadoop balanceing data
Why do we not use the Remaining % in place of use Used % when we are
selecting datanode for new data and when running the balancer.
form what I can tell we are using the use % used and we do not factor in non
DFS Used at all.
I see a datanode with only a 60GB hard drive fill up completely 100% before
the other servers that have 130+GB hard drives get half full.
Seams like Trying to keep the same % free on the drives in the cluster would
be more optimal in production.
I know this still may not be perfect but would be nice if we tried.
Billy
Re: What happens in HDFS DataNode recovery?
Posted by Aaron Kimball <aa...@cloudera.com>.
Also, see the balancer tool that comes with Hadoop. This background process
should be run periodically (Every week or so?) to make sure that data's
evenly distributed.
http://hadoop.apache.org/core/docs/r0.19.0/hdfs_user_guide.html#Rebalancer
- Aaron
On Sat, Jan 24, 2009 at 7:40 PM, jason hadoop <ja...@gmail.com>wrote:
> The blocks will be invalidated on the returned to service datanode.
> If you want to save your namenode and network a lot of work, wipe the hdfs
> block storage directory before returning the Datanode to service.
> dfs.data.dir will be the directory, most likley the value is
> ${hadoop.tmp.dir}/dfs/data
>
> Jason - Ex Attributor
>
> On Sat, Jan 24, 2009 at 6:19 PM, C G <pa...@yahoo.com> wrote:
>
> > Hi All:
> >
> > I elected to take a node out of one of our grids for service. Naturally
> > HDFS recognized the loss of the DataNode and did the right stuff, fixing
> > replication issues and ultimately delivering a clean file system.
> >
> > So now the node I removed is ready to go back in service. When I return
> it
> > to service a bunch of files will suddenly have a replication of 4 instead
> of
> > 3. My questions:
> >
> > 1. Will HDFS delete a copy of the data to bring replication back to 3?
> > 2. If (1) above is yes, will it remove the copy by deleting from other
> > nodes, or will it remove files from the returned node, or both?
> >
> > The motivation for asking the questions are that I have a file system
> which
> > is extremely unbalanced - we recently doubled the size of the grid when a
> > few dozen terabytes already stored on the existing nodes. I am wondering
> if
> > an easy way to restore some sense of balance is to cycle through the old
> > nodes, removing each one from service for several hours and then return
> it
> > to service.
> >
> > Thoughts?
> >
> > Thanks in Advance,
> > C G
> >
> >
> >
> >
> >
> >
> >
>
Re: What happens in HDFS DataNode recovery?
Posted by jason hadoop <ja...@gmail.com>.
The blocks will be invalidated on the returned to service datanode.
If you want to save your namenode and network a lot of work, wipe the hdfs
block storage directory before returning the Datanode to service.
dfs.data.dir will be the directory, most likley the value is
${hadoop.tmp.dir}/dfs/data
Jason - Ex Attributor
On Sat, Jan 24, 2009 at 6:19 PM, C G <pa...@yahoo.com> wrote:
> Hi All:
>
> I elected to take a node out of one of our grids for service. Naturally
> HDFS recognized the loss of the DataNode and did the right stuff, fixing
> replication issues and ultimately delivering a clean file system.
>
> So now the node I removed is ready to go back in service. When I return it
> to service a bunch of files will suddenly have a replication of 4 instead of
> 3. My questions:
>
> 1. Will HDFS delete a copy of the data to bring replication back to 3?
> 2. If (1) above is yes, will it remove the copy by deleting from other
> nodes, or will it remove files from the returned node, or both?
>
> The motivation for asking the questions are that I have a file system which
> is extremely unbalanced - we recently doubled the size of the grid when a
> few dozen terabytes already stored on the existing nodes. I am wondering if
> an easy way to restore some sense of balance is to cycle through the old
> nodes, removing each one from service for several hours and then return it
> to service.
>
> Thoughts?
>
> Thanks in Advance,
> C G
>
>
>
>
>
>
>
What happens in HDFS DataNode recovery?
Posted by C G <pa...@yahoo.com>.
Hi All:
I elected to take a node out of one of our grids for service. Naturally HDFS recognized the loss of the DataNode and did the right stuff, fixing replication issues and ultimately delivering a clean file system.
So now the node I removed is ready to go back in service. When I return it to service a bunch of files will suddenly have a replication of 4 instead of 3. My questions:
1. Will HDFS delete a copy of the data to bring replication back to 3?
2. If (1) above is yes, will it remove the copy by deleting from other nodes, or will it remove files from the returned node, or both?
The motivation for asking the questions are that I have a file system which is extremely unbalanced - we recently doubled the size of the grid when a few dozen terabytes already stored on the existing nodes. I am wondering if an easy way to restore some sense of balance is to cycle through the old nodes, removing each one from service for several hours and then return it to service.
Thoughts?
Thanks in Advance,
C G
Re: hadoop balanceing data
Posted by Billy Pearson <sa...@pearsonwholesale.com>.
did not thank about that good points
I found a way to keep it from happening
I set dfs.datanode.du.reserved in the config file
"Hairong Kuang" <ha...@yahoo-inc.com> wrote in
message news:C59F9164.ED09%hairong@yahoo-inc.com...
> %Remaining is much more fluctuate than %dfs used. This is because dfs
> shares
> the disks with mapred and mapred tasks may use a lot of disks temporally.
> So
> trying to keep the same %free is impossible most of the time.
>
> Hairong
>
>
> On 1/19/09 10:28 PM, "Billy Pearson"
> <sa...@pearsonwholesale.com> wrote:
>
>> Why do we not use the Remaining % in place of use Used % when we are
>> selecting datanode for new data and when running the balancer.
>> form what I can tell we are using the use % used and we do not factor in
>> non
>> DFS Used at all.
>> I see a datanode with only a 60GB hard drive fill up completely 100%
>> before
>> the other servers that have 130+GB hard drives get half full.
>> Seams like Trying to keep the same % free on the drives in the cluster
>> would
>> be more optimal in production.
>> I know this still may not be perfect but would be nice if we tried.
>>
>> Billy
>>
>>
>
>
Re: hadoop balanceing data
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
%Remaining is much more fluctuate than %dfs used. This is because dfs shares
the disks with mapred and mapred tasks may use a lot of disks temporally. So
trying to keep the same %free is impossible most of the time.
Hairong
On 1/19/09 10:28 PM, "Billy Pearson" <sa...@pearsonwholesale.com> wrote:
> Why do we not use the Remaining % in place of use Used % when we are
> selecting datanode for new data and when running the balancer.
> form what I can tell we are using the use % used and we do not factor in non
> DFS Used at all.
> I see a datanode with only a 60GB hard drive fill up completely 100% before
> the other servers that have 130+GB hard drives get half full.
> Seams like Trying to keep the same % free on the drives in the cluster would
> be more optimal in production.
> I know this still may not be perfect but would be nice if we tried.
>
> Billy
>
>