You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marc Sturlese <ma...@gmail.com> on 2012/08/24 22:09:29 UTC
doubt about reduce tasks and block writes
Hey there,
I have a doubt about reduce tasks and block writes. Do a reduce task always
first write to hdfs in the node where they it is placed? (and then these
blocks would be replicated to other nodes)
In case yes, if I have a cluster of 5 nodes, 4 of them run DN and TT and one
(node A) just run DN, when running MR jobs, map tasks would never read from
node A? This would be because maps have data locality and if the reduce
tasks write first to the node where they live, one replica of the block
would always be in a node that has a TT. Node A would just contain blocks
created from replication by the framework as no reduce task would write
there directly. Is this correct?
Thanks in advance
--
View this message in context: http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: doubt about reduce tasks and block writes
Posted by Raj Vishwanathan <ra...@yahoo.com>.
Harsh
I did leave an escape route open witth a bit about "corner cases" :-)
Anyway I agree that HDFS has no notion of block 0. I just meant that had the dfs.replication is 1, there will be,under normal circumstances :-), no blocks of output file will be written to node A.
Raj
----- Original Message -----
> From: Harsh J <ha...@cloudera.com>
> To: common-user@hadoop.apache.org; Raj Vishwanathan <ra...@yahoo.com>
> Cc:
> Sent: Saturday, August 25, 2012 4:02 AM
> Subject: Re: doubt about reduce tasks and block writes
>
> Raj's almost right. In times of high load or space fillup on a local
> DN, the NameNode may decide to instead pick a non-local DN for
> replica-writing. In this way, the Node A may get a "copy 0" of a
> replica from a task. This is per the default block placement policy.
>
> P.s. Note that HDFS hardly makes any differences between replicas,
> hence there is no hard-concept of a "copy 0" or "copy 1"
> block, at the
> NN level, it treats all DNs in pipeline equally and same for replicas.
>
> On Sat, Aug 25, 2012 at 4:14 AM, Raj Vishwanathan <ra...@yahoo.com>
> wrote:
>> But since node A has no TT running, it will not run map or reduce tasks.
> When the reducer node writes the output file, the fist block will be written on
> the local node and never on node A.
>>
>> So, to answer the question, Node A will contain copies of blocks of all
> output files. It wont contain the copy 0 of any output file.
>>
>>
>> I am reasonably sure about this , but there could be corner cases in case
> of node failure and such like! I need to look into the code.
>>
>>
>> Raj
>>> ________________________________
>>> From: Marc Sturlese <ma...@gmail.com>
>>> To: hadoop-user@lucene.apache.org
>>> Sent: Friday, August 24, 2012 1:09 PM
>>> Subject: doubt about reduce tasks and block writes
>>>
>>> Hey there,
>>> I have a doubt about reduce tasks and block writes. Do a reduce task
> always
>>> first write to hdfs in the node where they it is placed? (and then these
>>> blocks would be replicated to other nodes)
>>> In case yes, if I have a cluster of 5 nodes, 4 of them run DN and TT and
> one
>>> (node A) just run DN, when running MR jobs, map tasks would never read
> from
>>> node A? This would be because maps have data locality and if the reduce
>>> tasks write first to the node where they live, one replica of the block
>>> would always be in a node that has a TT. Node A would just contain
> blocks
>>> created from replication by the framework as no reduce task would write
>>> there directly. Is this correct?
>>> Thanks in advance
>>>
>>>
>>>
>>> --
>>> View this message in context:
> http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185.html
>>> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>>>
>>>
>>>
>
>
>
> --
> Harsh J
>
Re: doubt about reduce tasks and block writes
Posted by Harsh J <ha...@cloudera.com>.
Raj's almost right. In times of high load or space fillup on a local
DN, the NameNode may decide to instead pick a non-local DN for
replica-writing. In this way, the Node A may get a "copy 0" of a
replica from a task. This is per the default block placement policy.
P.s. Note that HDFS hardly makes any differences between replicas,
hence there is no hard-concept of a "copy 0" or "copy 1" block, at the
NN level, it treats all DNs in pipeline equally and same for replicas.
On Sat, Aug 25, 2012 at 4:14 AM, Raj Vishwanathan <ra...@yahoo.com> wrote:
> But since node A has no TT running, it will not run map or reduce tasks. When the reducer node writes the output file, the fist block will be written on the local node and never on node A.
>
> So, to answer the question, Node A will contain copies of blocks of all output files. It wont contain the copy 0 of any output file.
>
>
> I am reasonably sure about this , but there could be corner cases in case of node failure and such like! I need to look into the code.
>
>
> Raj
>>________________________________
>> From: Marc Sturlese <ma...@gmail.com>
>>To: hadoop-user@lucene.apache.org
>>Sent: Friday, August 24, 2012 1:09 PM
>>Subject: doubt about reduce tasks and block writes
>>
>>Hey there,
>>I have a doubt about reduce tasks and block writes. Do a reduce task always
>>first write to hdfs in the node where they it is placed? (and then these
>>blocks would be replicated to other nodes)
>>In case yes, if I have a cluster of 5 nodes, 4 of them run DN and TT and one
>>(node A) just run DN, when running MR jobs, map tasks would never read from
>>node A? This would be because maps have data locality and if the reduce
>>tasks write first to the node where they live, one replica of the block
>>would always be in a node that has a TT. Node A would just contain blocks
>>created from replication by the framework as no reduce task would write
>>there directly. Is this correct?
>>Thanks in advance
>>
>>
>>
>>--
>>View this message in context: http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185.html
>>Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>>
>>
>>
--
Harsh J
Re: doubt about reduce tasks and block writes
Posted by Marc Sturlese <ma...@gmail.com>.
Thanks, Raj you got exactly my point. I wanted to confirm this assumption as
I was guessing if a shared HDFS cluster with MR and Hbase like this would
make sense:
http://old.nabble.com/HBase-User-f34655.html
--
View this message in context: http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185p4003211.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: doubt about reduce tasks and block writes
Posted by Raj Vishwanathan <ra...@yahoo.com>.
But since node A has no TT running, it will not run map or reduce tasks. When the reducer node writes the output file, the fist block will be written on the local node and never on node A.
So, to answer the question, Node A will contain copies of blocks of all output files. It wont contain the copy 0 of any output file.
I am reasonably sure about this , but there could be corner cases in case of node failure and such like! I need to look into the code.
Raj
>________________________________
> From: Marc Sturlese <ma...@gmail.com>
>To: hadoop-user@lucene.apache.org
>Sent: Friday, August 24, 2012 1:09 PM
>Subject: doubt about reduce tasks and block writes
>
>Hey there,
>I have a doubt about reduce tasks and block writes. Do a reduce task always
>first write to hdfs in the node where they it is placed? (and then these
>blocks would be replicated to other nodes)
>In case yes, if I have a cluster of 5 nodes, 4 of them run DN and TT and one
>(node A) just run DN, when running MR jobs, map tasks would never read from
>node A? This would be because maps have data locality and if the reduce
>tasks write first to the node where they live, one replica of the block
>would always be in a node that has a TT. Node A would just contain blocks
>created from replication by the framework as no reduce task would write
>there directly. Is this correct?
>Thanks in advance
>
>
>
>--
>View this message in context: http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185.html
>Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
>
Re: doubt about reduce tasks and block writes
Posted by Minh Duc Nguyen <md...@gmail.com>.
Marc, see my inline comments.
On Fri, Aug 24, 2012 at 4:09 PM, Marc Sturlese <ma...@gmail.com>wrote:
> Hey there,
> I have a doubt about reduce tasks and block writes. Do a reduce task always
> first write to hdfs in the node where they it is placed? (and then these
> blocks would be replicated to other nodes)
>
Yes, if there is a DN running on that server (it's possible to be running
TT without a DN).
> In case yes, if I have a cluster of 5 nodes, 4 of them run DN and TT and
> one
> (node A) just run DN, when running MR jobs, map tasks would never read from
> node A? This would be because maps have data locality and if the reduce
> tasks write first to the node where they live, one replica of the block
> would always be in a node that has a TT. Node A would just contain blocks
> created from replication by the framework as no reduce task would write
> there directly. Is this correct?
>
I believe that it's possible that a map task would read from node A's DN.
Yes, the JobTracker tries to schedule map tasks on nodes where the data
would be local, but it can't always do so. If there's a node with a free
map slot, but that node doesn't have the data blocks locally, the
JobTracker will assign the map task to that free map slot. Some work done
(albeit slower than the ideal case because of the increased network IO) is
better than no work done.
> Thanks in advance
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
Re: doubt about reduce tasks and block writes
Posted by Bertrand Dechoux <de...@gmail.com>.
Assuming that node A only contains replica, there is no garante that its
data would never be read.
First, you might lose a replica. The copy inside the node A could be used
to create the missing replica again.
Second, data locality is on best effort. If all the map slots are occupied
except one on one node without a replica of the data then your node A is as
likely as any other to be chosen as a source.
Regards
Bertrand
On Fri, Aug 24, 2012 at 10:09 PM, Marc Sturlese <ma...@gmail.com>wrote:
> Hey there,
> I have a doubt about reduce tasks and block writes. Do a reduce task always
> first write to hdfs in the node where they it is placed? (and then these
> blocks would be replicated to other nodes)
> In case yes, if I have a cluster of 5 nodes, 4 of them run DN and TT and
> one
> (node A) just run DN, when running MR jobs, map tasks would never read from
> node A? This would be because maps have data locality and if the reduce
> tasks write first to the node where they live, one replica of the block
> would always be in a node that has a TT. Node A would just contain blocks
> created from replication by the framework as no reduce task would write
> there directly. Is this correct?
> Thanks in advance
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/doubt-about-reduce-tasks-and-block-writes-tp4003185.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
--
Bertrand Dechoux