org.apache.hadoop.mapreduce.Mapper setup method equivalent in streaming or pipes

[amit handa <am...@gmail.com>]

Hi,

I want to do some initial one time set up before each of my map task starts.
Is there some method which is similar to
org.apache.hadoop.mapreduce.Mapper -setup method, in hadoop pipes or
streaming ?

Another newbie question - Does hadoop keep the map tasks alive for a given
job e.g if i set max map tasks to 4 and  if i call setup method , can i
assume that it won't be called more than 4 times (once for each map task)?
I am assuming the task runs normally (no task hangs or crashes)

Thanks,
Amit

Re: org.apache.hadoop.mapreduce.Mapper setup method equivalent in streaming or pipes

[Chuck Lam <ch...@gmail.com>]

In Streaming, since your program is reading in records thru STDIN in a loop,
you can just put your setup code before the loop, like

# your setup code here
for record in sys.stdin:
    # your normal code here for processing record

As to the number of times the setup code will be called. First of all, you
don't completely specify the number of map tasks Hadoop will use. The actual
number depends on your spec, file size, block size, etc. (You do completely
specify the number of reduce tasks though.) Second, as you know, failed
tasks will get restarted, so the setup code can be called multiple times
because of that.

Furthermore, even if your tasks don't fail, Hadoop can do what's called
speculative execution and execute the same task multiple times (in different
nodes). This occurs when Hadoop thinks a task is taking too long to finish
and thinks that starting this task on a different node will finish faster.
In that situation one task can be running twice simultaneously. Hadoop uses
only the result of the first finisher, but the setup code will have been
executed multiple times.

Speculative execution is enabled by default but can be turned off using
mapred.map.tasks.speculative.execution and
mapred.reduce.tasks.speculative.execution.



On Mon, Jun 1, 2009 at 9:07 PM, amit handa <am...@gmail.com> wrote:

> Just to clarify, When i say equivalent in streaming, i am looking for some
> way of achieving this in streaming.
>
> On Tue, Jun 2, 2009 at 9:30 AM, amit handa <am...@gmail.com> wrote:
>
> > Hi,
> >
> > I want to do some initial one time set up before each of my map task
> > starts. Is there some method which is similar to
> > org.apache.hadoop.mapreduce.Mapper -setup method, in hadoop pipes or
> > streaming ?
> >
> > Another newbie question - Does hadoop keep the map tasks alive for a
> given
> > job e.g if i set max map tasks to 4 and  if i call setup method , can i
> > assume that it won't be called more than 4 times (once for each map
> task)?
> > I am assuming the task runs normally (no task hangs or crashes)
> >
> > Thanks,
> > Amit
> >
> >
>

Re: org.apache.hadoop.mapreduce.Mapper setup method equivalent in streaming or pipes

[amit handa <am...@gmail.com>]

Just to clarify, When i say equivalent in streaming, i am looking for some
way of achieving this in streaming.

On Tue, Jun 2, 2009 at 9:30 AM, amit handa <am...@gmail.com> wrote:

> Hi,
>
> I want to do some initial one time set up before each of my map task
> starts. Is there some method which is similar to
> org.apache.hadoop.mapreduce.Mapper -setup method, in hadoop pipes or
> streaming ?
>
> Another newbie question - Does hadoop keep the map tasks alive for a given
> job e.g if i set max map tasks to 4 and  if i call setup method , can i
> assume that it won't be called more than 4 times (once for each map task)?
> I am assuming the task runs normally (no task hangs or crashes)
>
> Thanks,
> Amit
>
>