You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jeffrey Buell <jb...@vmware.com> on 2011/02/25 23:52:32 UTC
balancing and replication in HDFS
I'm a newbie to hadoop and HDFS. I'm seeing odd behavior in HDFS that I hope somebody can clear up for me. I'm running hadoop version 0.20.1+169.127 from the cloudera distro on 4 identical nodes, each with 4 cpus and 100GB disk space. Replication is set to 2.
I run:
hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar teragen 50000000 tera_in5
This produces the expected 10GB of data on disk (5GB * 2 copies). But the data is spread very unevenly across the nodes, ranging from 1.7 to 3.2 GB on each node. Then I sort the data:
hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar terasort tera_in5 tera_out5
It finishes successfully, and HDFS recognizes the right amount of data:
$ hadoop fs -du /user/hadoop/
Found 2 items
5000023410 hdfs://namd-1/user/hadoop/tera_in5
5000170993 hdfs://namd-1/user/hadoop/tera_out5
However all the new data is on one node (apparently randomly chosen), and the total disk usage is only 15GB, which means that the output data is not replicated. For nearly all the elapsed time of the sort, the other 3 nodes are idle. Some of the output data is in dfs/data/current, but a lot is in one of 64 new subdirs (dfs/data/current/subdir0 through subdir63).
Why is all this happening? Am I missing some tunables that make HDFS do the right balance and replication?
Thanks,
Jeff
Re: balancing and replication in HDFS
Posted by Todd Lipcon <to...@cloudera.com>.
When you run terasort, pass -Dmapred.reduce.tasks=4 and see how that
goes for you. See this old thread for info:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3Ccbbf4b570906300617ma4505f5o2aa1b9fb87b31868@mail.gmail.com%3E
-Todd
On Fri, Feb 25, 2011 at 4:45 PM, Jeffrey Buell <jb...@vmware.com> wrote:
>
> Hi Todd,
>
> Thanks for the quick response.
>
> I added <final>true</final> to dfs.replication, but I still get just one output copy. Can hadoop apps overwrite the replication level even with this parameter?
>
> I tried increasing mapred.tasktracker.reduce.tasks.maximum from 1 to 4, but that didn't make any difference: output is still all on one node. I thought that parameter controls the number of reduce tasks per node so 1 should be sufficient, is that correct logic? The other reduce parameters are at their defaults. Which parameters should I be trying?
>
> There's no way the sort can run efficiently if the gen is unbalanced. At 5 GB, there're plenty of blocks to make the distribution even. How does HDFS decide to spread the storage across the nodes? Note the sizes of the 4 chunks (1.7,...,3.2 GB) seem to be repeatable, but they land on different nodes each time teragen is run.
>
> Jeff
>
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Friday, February 25, 2011 3:13 PM
> > To: hdfs-user@hadoop.apache.org
> > Subject: Re: balancing and replication in HDFS
> >
> > Hi Jeff,
> > The output of terasort has replication level 1 by default. This is so
> > it goes faster with the default settings and makes for more impressive
> > benchmark results :)
> > The reason you see it all on one machine is probably that you're
> > running with one reducer. Try configuring your terasort to use more
> > reduce tasks and you should see the load (and space usage) even out.
> > -Todd
> >
> > On Fri, Feb 25, 2011 at 2:52 PM, Jeffrey Buell <jb...@vmware.com>
> > wrote:
> > >
> > > I'm a newbie to hadoop and HDFS. I'm seeing odd behavior in HDFS
> > that I hope somebody can clear up for me. I'm running hadoop version
> > 0.20.1+169.127 from the cloudera distro on 4 identical nodes, each with
> > 4 cpus and 100GB disk space. Replication is set to 2.
> > >
> > > I run:
> > >
> > > hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar teragen 50000000
> > tera_in5
> > >
> > > This produces the expected 10GB of data on disk (5GB * 2 copies).
> > But the data is spread very unevenly across the nodes, ranging from
> > 1.7 to 3.2 GB on each node. Then I sort the data:
> > >
> > > hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar terasort tera_in5
> > tera_out5
> > >
> > > It finishes successfully, and HDFS recognizes the right amount of
> > data:
> > >
> > > $ hadoop fs -du /user/hadoop/
> > > Found 2 items
> > > 5000023410 hdfs://namd-1/user/hadoop/tera_in5
> > > 5000170993 hdfs://namd-1/user/hadoop/tera_out5
> > >
> > > However all the new data is on one node (apparently randomly chosen),
> > and the total disk usage is only 15GB, which means that the output data
> > is not replicated. For nearly all the elapsed time of the sort, the
> > other 3 nodes are idle. Some of the output data is in
> > dfs/data/current, but a lot is in one of 64 new subdirs
> > (dfs/data/current/subdir0 through subdir63).
> > >
> > > Why is all this happening? Am I missing some tunables that make HDFS
> > do the right balance and replication?
> > >
> > > Thanks,
> > >
> > > Jeff
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
RE: balancing and replication in HDFS
Posted by Jeffrey Buell <jb...@vmware.com>.
Todd,
Much better. I also had to adjust the number of map tasks for the gen step. Storage was spread across 3 nodes instead of 4, but a bit more playing with these parameters should do the trick.
Thanks for your help.
Jeff
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Friday, February 25, 2011 8:09 PM
To: hdfs-user@hadoop.apache.org
Cc: Jeffrey Buell
Subject: Re: balancing and replication in HDFS
When you run terasort, pass -Dmapred.reduce.tasks=4 and see how that goes for you. See this old thread for info:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3Ccbbf4b570906300617ma4505f5o2aa1b9fb87b31868@mail.gmail.com%3E
-Todd
On Fri, Feb 25, 2011 at 4:45 PM, Jeffrey Buell <jb...@vmware.com>> wrote:
Hi Todd,
Thanks for the quick response.
I added <final>true</final> to dfs.replication, but I still get just one output copy. Can hadoop apps overwrite the replication level even with this parameter?
I tried increasing mapred.tasktracker.reduce.tasks.maximum from 1 to 4, but that didn't make any difference: output is still all on one node. I thought that parameter controls the number of reduce tasks per node so 1 should be sufficient, is that correct logic? The other reduce parameters are at their defaults. Which parameters should I be trying?
There's no way the sort can run efficiently if the gen is unbalanced. At 5 GB, there're plenty of blocks to make the distribution even. How does HDFS decide to spread the storage across the nodes? Note the sizes of the 4 chunks (1.7,...,3.2 GB) seem to be repeatable, but they land on different nodes each time teragen is run.
Jeff
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com<ma...@cloudera.com>]
> Sent: Friday, February 25, 2011 3:13 PM
> To: hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: Re: balancing and replication in HDFS
>
> Hi Jeff,
> The output of terasort has replication level 1 by default. This is so
> it goes faster with the default settings and makes for more impressive
> benchmark results :)
> The reason you see it all on one machine is probably that you're
> running with one reducer. Try configuring your terasort to use more
> reduce tasks and you should see the load (and space usage) even out.
> -Todd
>
> On Fri, Feb 25, 2011 at 2:52 PM, Jeffrey Buell <jb...@vmware.com>>
> wrote:
> >
> > I'm a newbie to hadoop and HDFS. I'm seeing odd behavior in HDFS
> that I hope somebody can clear up for me. I'm running hadoop version
> 0.20.1+169.127 from the cloudera distro on 4 identical nodes, each with
> 4 cpus and 100GB disk space. Replication is set to 2.
> >
> > I run:
> >
> > hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar teragen 50000000
> tera_in5
> >
> > This produces the expected 10GB of data on disk (5GB * 2 copies).
> But the data is spread very unevenly across the nodes, ranging from
> 1.7 to 3.2 GB on each node. Then I sort the data:
> >
> > hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar terasort tera_in5
> tera_out5
> >
> > It finishes successfully, and HDFS recognizes the right amount of
> data:
> >
> > $ hadoop fs -du /user/hadoop/
> > Found 2 items
> > 5000023410 hdfs://namd-1/user/hadoop/tera_in5
> > 5000170993 hdfs://namd-1/user/hadoop/tera_out5
> >
> > However all the new data is on one node (apparently randomly chosen),
> and the total disk usage is only 15GB, which means that the output data
> is not replicated. For nearly all the elapsed time of the sort, the
> other 3 nodes are idle. Some of the output data is in
> dfs/data/current, but a lot is in one of 64 new subdirs
> (dfs/data/current/subdir0 through subdir63).
> >
> > Why is all this happening? Am I missing some tunables that make HDFS
> do the right balance and replication?
> >
> > Thanks,
> >
> > Jeff
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
RE: balancing and replication in HDFS
Posted by Jeffrey Buell <jb...@vmware.com>.
Hi Todd,
Thanks for the quick response.
I added <final>true</final> to dfs.replication, but I still get just one output copy. Can hadoop apps overwrite the replication level even with this parameter?
I tried increasing mapred.tasktracker.reduce.tasks.maximum from 1 to 4, but that didn't make any difference: output is still all on one node. I thought that parameter controls the number of reduce tasks per node so 1 should be sufficient, is that correct logic? The other reduce parameters are at their defaults. Which parameters should I be trying?
There's no way the sort can run efficiently if the gen is unbalanced. At 5 GB, there're plenty of blocks to make the distribution even. How does HDFS decide to spread the storage across the nodes? Note the sizes of the 4 chunks (1.7,...,3.2 GB) seem to be repeatable, but they land on different nodes each time teragen is run.
Jeff
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Friday, February 25, 2011 3:13 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: balancing and replication in HDFS
>
> Hi Jeff,
> The output of terasort has replication level 1 by default. This is so
> it goes faster with the default settings and makes for more impressive
> benchmark results :)
> The reason you see it all on one machine is probably that you're
> running with one reducer. Try configuring your terasort to use more
> reduce tasks and you should see the load (and space usage) even out.
> -Todd
>
> On Fri, Feb 25, 2011 at 2:52 PM, Jeffrey Buell <jb...@vmware.com>
> wrote:
> >
> > I'm a newbie to hadoop and HDFS. I'm seeing odd behavior in HDFS
> that I hope somebody can clear up for me. I'm running hadoop version
> 0.20.1+169.127 from the cloudera distro on 4 identical nodes, each with
> 4 cpus and 100GB disk space. Replication is set to 2.
> >
> > I run:
> >
> > hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar teragen 50000000
> tera_in5
> >
> > This produces the expected 10GB of data on disk (5GB * 2 copies).
> But the data is spread very unevenly across the nodes, ranging from
> 1.7 to 3.2 GB on each node. Then I sort the data:
> >
> > hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar terasort tera_in5
> tera_out5
> >
> > It finishes successfully, and HDFS recognizes the right amount of
> data:
> >
> > $ hadoop fs -du /user/hadoop/
> > Found 2 items
> > 5000023410 hdfs://namd-1/user/hadoop/tera_in5
> > 5000170993 hdfs://namd-1/user/hadoop/tera_out5
> >
> > However all the new data is on one node (apparently randomly chosen),
> and the total disk usage is only 15GB, which means that the output data
> is not replicated. For nearly all the elapsed time of the sort, the
> other 3 nodes are idle. Some of the output data is in
> dfs/data/current, but a lot is in one of 64 new subdirs
> (dfs/data/current/subdir0 through subdir63).
> >
> > Why is all this happening? Am I missing some tunables that make HDFS
> do the right balance and replication?
> >
> > Thanks,
> >
> > Jeff
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: balancing and replication in HDFS
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jeff,
The output of terasort has replication level 1 by default. This is so
it goes faster with the default settings and makes for more impressive
benchmark results :)
The reason you see it all on one machine is probably that you're
running with one reducer. Try configuring your terasort to use more
reduce tasks and you should see the load (and space usage) even out.
-Todd
On Fri, Feb 25, 2011 at 2:52 PM, Jeffrey Buell <jb...@vmware.com> wrote:
>
> I'm a newbie to hadoop and HDFS. I'm seeing odd behavior in HDFS that I hope somebody can clear up for me. I'm running hadoop version 0.20.1+169.127 from the cloudera distro on 4 identical nodes, each with 4 cpus and 100GB disk space. Replication is set to 2.
>
> I run:
>
> hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar teragen 50000000 tera_in5
>
> This produces the expected 10GB of data on disk (5GB * 2 copies). But the data is spread very unevenly across the nodes, ranging from 1.7 to 3.2 GB on each node. Then I sort the data:
>
> hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar terasort tera_in5 tera_out5
>
> It finishes successfully, and HDFS recognizes the right amount of data:
>
> $ hadoop fs -du /user/hadoop/
> Found 2 items
> 5000023410 hdfs://namd-1/user/hadoop/tera_in5
> 5000170993 hdfs://namd-1/user/hadoop/tera_out5
>
> However all the new data is on one node (apparently randomly chosen), and the total disk usage is only 15GB, which means that the output data is not replicated. For nearly all the elapsed time of the sort, the other 3 nodes are idle. Some of the output data is in dfs/data/current, but a lot is in one of 64 new subdirs (dfs/data/current/subdir0 through subdir63).
>
> Why is all this happening? Am I missing some tunables that make HDFS do the right balance and replication?
>
> Thanks,
>
> Jeff
--
Todd Lipcon
Software Engineer, Cloudera