You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Saptarshi Guha <sa...@gmail.com> on 2009/06/22 18:23:53 UTC
When is configure and close run
Hello,
In a mapreduce job, a given map JVM will run N map tasks. Are the
configure and close methods executed for every one of these N tasks?
Or is configure executed once when the JVM starts and the close method
executed once when all N have been completed?
I have the same question for the reduce task. Will it be run before
for every reduce task? And close is run when all the values for a
given key have been processed?
We can assume there isn't a combiner.
Regards
Saptarshi
Re: When is configure and close run
Posted by Saptarshi Guha <sa...@gmail.com>.
Thank you! Just to confirm. Consider a JVM (that is being reused), has
to reduce K1,{V11,V12,V13..} and K2,{V21,V22,V23,....}. Then the
configure and close methods are called once each for both K1,{V11,...}
and K2,{V2,....}?
Is my understanding correct?
Once again, there is no combiner, and it makes sense that it is not called.
Thank you
Saptarshi
On Mon, Jun 22, 2009 at 10:55 PM, jason hadoop<ja...@gmail.com> wrote:
> configure and close are run for each task, mapper and reducer. The configure
> and close are NOT run on the combiner class.
>
> On Mon, Jun 22, 2009 at 9:23 AM, Saptarshi Guha <sa...@gmail.com>
> wrote:
>>
>> Hello,
>> In a mapreduce job, a given map JVM will run N map tasks. Are the
>> configure and close methods executed for every one of these N tasks?
>> Or is configure executed once when the JVM starts and the close method
>> executed once when all N have been completed?
>>
>> I have the same question for the reduce task. Will it be run before
>> for every reduce task? And close is run when all the values for a
>> given key have been processed?
>>
>> We can assume there isn't a combiner.
>>
>> Regards
>> Saptarshi
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: When is configure and close run
Posted by jason hadoop <ja...@gmail.com>.
configure and close are run for each task, mapper and reducer. The configure
and close are NOT run on the combiner class.
On Mon, Jun 22, 2009 at 9:23 AM, Saptarshi Guha <sa...@gmail.com>wrote:
> Hello,
> In a mapreduce job, a given map JVM will run N map tasks. Are the
> configure and close methods executed for every one of these N tasks?
> Or is configure executed once when the JVM starts and the close method
> executed once when all N have been completed?
>
> I have the same question for the reduce task. Will it be run before
> for every reduce task? And close is run when all the values for a
> given key have been processed?
>
> We can assume there isn't a combiner.
>
> Regards
> Saptarshi
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals