You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Seunghwa Kang <s....@gatech.edu> on 2009/07/18 00:57:28 UTC
Data-local map tasks lower than Launched map tasks even with full
replication
Hello,
I am running Hadoop on my 4 nodes system.
Initially, I pick the replication factor of 2, and nearly 100% of map
tasks run in local up to 3 nodes, but the ratio drops to 80% if I use
all 4 nodes.
As my nodes have quite high I/O bandwidth (24 disks per node), but only
limited network bandwidth (1 GigE), this really hampers the scalability.
Just for test purpose, I increase the replication factor to 4, and check
that input data actually has replication factor of 4 with 'hadoop fs
-stat %r%n' but find that the ratio is still around 80% for 4 nodes.
How this can happen, and which configuration parameter should I change
to fix this?
Thank you very much,
-seunghwa
Re: Data-local map tasks lower than Launched map tasks even with full
replication
Posted by Ted Dunning <te...@gmail.com>.
This is a very non-typical node for hadoop clusters. 8 cores is not that
uncommon, but normally nodes have only 2 disks. The rationale for a small
number of spindles per machine is that total disk bandwidth scales with
number of busses and controllers rather than number of spindles. The same
thing goes for aggregate network bandwidth. Thus the way to increase total
bandwidth is to increase the number of independent front-side busses (which
is done by increasing the number of machines).
For a typical cluster of 10-100 nodes with replication = 3, you should get
80-95% data local map tasks. Experiments with small clusters like this are
very sensitive to oddities such as what you noticed. Moreover, for decent
switch topology, you should get very high aggregate usable bandwidth in a
cluster like that.
If you want to see how scaling will work in terms of data locality, try
using a much larger number of virtual machines. That will give you a test
bed to see how many map tasks will be data-local. It won't be an effective
way to actually scale your work-load, just an interesting experiment so that
you can see a larger hadoop cluster in action.
On Fri, Jul 17, 2009 at 4:59 PM, Seunghwa Kang <s....@gatech.edu> wrote:
> My single node has 4 dual core cpus, and 24 hard disks, I need to get
> input data for 4 dual core cpus via 1 GigE network interface and seems
> like this hurts the scalability.
>
--
Ted Dunning, CTO
DeepDyve
Re: Data-local map tasks lower than Launched map tasks even with
full replication
Posted by Seunghwa Kang <s....@gatech.edu>.
Yes, it was 1 MB/s (I was foolish), and it shouldn't affect the result.
Also, log for 4th node says it wasn't properly working as a datanode,
and regardless of replication factor, data is replicated to only 3 of 4
nodes (I was foolish again :-().
My single node has 4 dual core cpus, and 24 hard disks, I need to get
input data for 4 dual core cpus via 1 GigE network interface and seems
like this hurts the scalability.
Thanks a lot for all the help!!!
On Fri, 2009-07-17 at 16:21 -0700, Todd Lipcon wrote:
> On Fri, Jul 17, 2009 at 4:16 PM, Seunghwa Kang <s....@gatech.edu>
> wrote:
> I checked with
>
> bin/hadoop fs -stat "%n %r" input/*
>
> part-00000 4
> part-00001 4
> part-00002 4
> part-00003 4
> part-00004 4
> part-00005 4
> part-00006 4
> part-00007 4
>
> and see replication factor is 4.
>
> Also, I set replication factor to 4 in hadoop-site.xml, run
> stop-all.sh
> and start-all.sh, re-load the data, and re-run the code but
> still
> getting the same result.
>
> I am searching for hadoop-default.xml and find
>
> <property>
> <name>dfs.balance.bandwidthPerSec</name>
> <value>1048576</value>
> <description>
> Specifies the maximum amount of bandwidth that each datanode
> can utilize for the balancing purpose in term of
> the number of bytes per second.
> </description>
> </property>
>
> 1048576 is 1 GB/s and seems like higher than 1 Gbit/s for my
> nodes. I am
> going to change this value and see what happens.
>
> That's in bytes per second, so that's 1MB/sec. If anything you may
> want to raise it on a small cluster. This also affects the dfs
> balancer, not rereplication of underreplicated blocks, so shouldn't
> matter.
>
> Check fsck and see what it says for a count of underreplicated blocks.
> Also, if you use the NN web UI to navigate to view one of these files,
> it should tell you where the blocks are hosted.
>
> Overall, I wouldn't worry about this on a small cluster unless you've
> seen on your monitoring graphs that your network is getting saturated.
> You'll probably be CPU bound before you're network bound unless you
> have VERY low locality and very fast CPUs.
>
> -Todd
>
>
>
>
>
>
> On Fri, 2009-07-17 at 16:07 -0700, Ted Dunning wrote:
> >
> > Does [hadoop fs -fsck /] show any under-replicated
> files/blocks? you
> > may not waited long enough after increasing the target
> replication
> > rate.
> >
> > Another thing to watch out for in a production node is the
> > distribution of node blocks. You should be careful to load
> data from
> > outside the cluster to ensure random placement of file
> blocks. That
> > is critical for getting good locality. This obviously
> doesn't apply
> > to your situation with 4 replicas on 4 nodes.
> >
> > Todd's comment about -setrep is also very important to note.
> >
> > On Fri, Jul 17, 2009 at 3:57 PM, Seunghwa Kang
> <s....@gatech.edu>
> > wrote:
> >
> > Just for test purpose, I increase the replication
> factor to 4,
> > and check
> > that input data actually has replication factor of 4
> with
> > 'hadoop fs
> > -stat %r%n' but find that the ratio is still around
> 80% for 4
> > nodes.
> >
> >
>
>
>
Re: Data-local map tasks lower than Launched map tasks even with full
replication
Posted by Todd Lipcon <to...@cloudera.com>.
On Fri, Jul 17, 2009 at 4:16 PM, Seunghwa Kang <s....@gatech.edu> wrote:
> I checked with
>
> bin/hadoop fs -stat "%n %r" input/*
>
> part-00000 4
> part-00001 4
> part-00002 4
> part-00003 4
> part-00004 4
> part-00005 4
> part-00006 4
> part-00007 4
>
> and see replication factor is 4.
>
> Also, I set replication factor to 4 in hadoop-site.xml, run stop-all.sh
> and start-all.sh, re-load the data, and re-run the code but still
> getting the same result.
>
> I am searching for hadoop-default.xml and find
>
> <property>
> <name>dfs.balance.bandwidthPerSec</name>
> <value>1048576</value>
> <description>
> Specifies the maximum amount of bandwidth that each datanode
> can utilize for the balancing purpose in term of
> the number of bytes per second.
> </description>
> </property>
>
> 1048576 is 1 GB/s and seems like higher than 1 Gbit/s for my nodes. I am
> going to change this value and see what happens.
That's in bytes per second, so that's 1MB/sec. If anything you may want to
raise it on a small cluster. This also affects the dfs balancer, not
rereplication of underreplicated blocks, so shouldn't matter.
Check fsck and see what it says for a count of underreplicated blocks. Also,
if you use the NN web UI to navigate to view one of these files, it should
tell you where the blocks are hosted.
Overall, I wouldn't worry about this on a small cluster unless you've seen
on your monitoring graphs that your network is getting saturated. You'll
probably be CPU bound before you're network bound unless you have VERY low
locality and very fast CPUs.
-Todd
>
> On Fri, 2009-07-17 at 16:07 -0700, Ted Dunning wrote:
> >
> > Does [hadoop fs -fsck /] show any under-replicated files/blocks? you
> > may not waited long enough after increasing the target replication
> > rate.
> >
> > Another thing to watch out for in a production node is the
> > distribution of node blocks. You should be careful to load data from
> > outside the cluster to ensure random placement of file blocks. That
> > is critical for getting good locality. This obviously doesn't apply
> > to your situation with 4 replicas on 4 nodes.
> >
> > Todd's comment about -setrep is also very important to note.
> >
> > On Fri, Jul 17, 2009 at 3:57 PM, Seunghwa Kang <s....@gatech.edu>
> > wrote:
> >
> > Just for test purpose, I increase the replication factor to 4,
> > and check
> > that input data actually has replication factor of 4 with
> > 'hadoop fs
> > -stat %r%n' but find that the ratio is still around 80% for 4
> > nodes.
> >
> >
>
>
Re: Data-local map tasks lower than Launched map tasks even with
full replication
Posted by Seunghwa Kang <s....@gatech.edu>.
I checked with
bin/hadoop fs -stat "%n %r" input/*
part-00000 4
part-00001 4
part-00002 4
part-00003 4
part-00004 4
part-00005 4
part-00006 4
part-00007 4
and see replication factor is 4.
Also, I set replication factor to 4 in hadoop-site.xml, run stop-all.sh
and start-all.sh, re-load the data, and re-run the code but still
getting the same result.
I am searching for hadoop-default.xml and find
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>1048576</value>
<description>
Specifies the maximum amount of bandwidth that each datanode
can utilize for the balancing purpose in term of
the number of bytes per second.
</description>
</property>
1048576 is 1 GB/s and seems like higher than 1 Gbit/s for my nodes. I am
going to change this value and see what happens.
Any other suggestions?
Thank you very much,
-seunghwa
On Fri, 2009-07-17 at 16:07 -0700, Ted Dunning wrote:
>
> Does [hadoop fs -fsck /] show any under-replicated files/blocks? you
> may not waited long enough after increasing the target replication
> rate.
>
> Another thing to watch out for in a production node is the
> distribution of node blocks. You should be careful to load data from
> outside the cluster to ensure random placement of file blocks. That
> is critical for getting good locality. This obviously doesn't apply
> to your situation with 4 replicas on 4 nodes.
>
> Todd's comment about -setrep is also very important to note.
>
> On Fri, Jul 17, 2009 at 3:57 PM, Seunghwa Kang <s....@gatech.edu>
> wrote:
>
> Just for test purpose, I increase the replication factor to 4,
> and check
> that input data actually has replication factor of 4 with
> 'hadoop fs
> -stat %r%n' but find that the ratio is still around 80% for 4
> nodes.
>
>
Re: Data-local map tasks lower than Launched map tasks even with full
replication
Posted by Ted Dunning <te...@gmail.com>.
Does [hadoop fs -fsck /] show any under-replicated files/blocks? you may
not waited long enough after increasing the target replication rate.
Another thing to watch out for in a production node is the distribution of
node blocks. You should be careful to load data from outside the cluster to
ensure random placement of file blocks. That is critical for getting good
locality. This obviously doesn't apply to your situation with 4 replicas on
4 nodes.
Todd's comment about -setrep is also very important to note.
On Fri, Jul 17, 2009 at 3:57 PM, Seunghwa Kang <s....@gatech.edu> wrote:
>
> Just for test purpose, I increase the replication factor to 4, and check
> that input data actually has replication factor of 4 with 'hadoop fs
> -stat %r%n' but find that the ratio is still around 80% for 4 nodes.
>
>
Re: Data-local map tasks lower than Launched map tasks even with full
replication
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Suenghwa,
It's important to note that changing the dfs.replication config variable
does not change the current files in HDFS. You have to use fs -setrep on
those files to change their replication count. The replication count is set
when the files were created and not modified thereafter unless you
explicitly do so.
-Todd
On Fri, Jul 17, 2009 at 4:01 PM, Seunghwa Kang <s....@gatech.edu> wrote:
> I found I forgot to mention my hadoop version.
>
> I am using 0.19.1.
>
> Thanks again,
>
> -seunghwa
>
> On Fri, 2009-07-17 at 18:57 -0400, Seunghwa Kang wrote:
> > Hello,
> >
> > I am running Hadoop on my 4 nodes system.
> >
> > Initially, I pick the replication factor of 2, and nearly 100% of map
> > tasks run in local up to 3 nodes, but the ratio drops to 80% if I use
> > all 4 nodes.
> >
> > As my nodes have quite high I/O bandwidth (24 disks per node), but
> > only
> > limited network bandwidth (1 GigE), this really hampers the
> > scalability.
> >
> > Just for test purpose, I increase the replication factor to 4, and
> > check
> > that input data actually has replication factor of 4 with 'hadoop fs
> > -stat %r%n' but find that the ratio is still around 80% for 4 nodes.
> >
>
>
Re: Data-local map tasks lower than Launched map tasks even with
full replication
Posted by Seunghwa Kang <s....@gatech.edu>.
I found I forgot to mention my hadoop version.
I am using 0.19.1.
Thanks again,
-seunghwa
On Fri, 2009-07-17 at 18:57 -0400, Seunghwa Kang wrote:
> Hello,
>
> I am running Hadoop on my 4 nodes system.
>
> Initially, I pick the replication factor of 2, and nearly 100% of map
> tasks run in local up to 3 nodes, but the ratio drops to 80% if I use
> all 4 nodes.
>
> As my nodes have quite high I/O bandwidth (24 disks per node), but
> only
> limited network bandwidth (1 GigE), this really hampers the
> scalability.
>
> Just for test purpose, I increase the replication factor to 4, and
> check
> that input data actually has replication factor of 4 with 'hadoop fs
> -stat %r%n' but find that the ratio is still around 80% for 4 nodes.
>