Pipes task being killed

[Rahul Sood <rs...@yahoo-inc.com>]

Hi,

We have a Pipes C++ application where the Reduce task does a lot of
computation. After some time the task gets killed by the Hadoop
framework. The job output shows the following error:

Task task_200803051654_0001_r_000000_0 failed to report status for 604
seconds. Killing!

Is there any way to send a heartbeat to the TaskTracker from a Pipes
application. I believe this is possible in Java using
org.apache.hadoop.util.Progress and we're looking for something
equivalent in the C++ Pipes API.

-Rahul Sood
rsood@yahoo-inc.com

Re: Pipes task being killed

[Miles Osborne <mi...@inf.ed.ac.uk>]

Is this also true for streaming?

Miles

On 05/03/2008, Richard Kasperski <rk...@yahoo-inc.com> wrote:
>
> I think you just need to write to stderr. My understanding is that
> hadoop is happy as long as input is being consumed, output is being
> generated or status is being generated.
>
>
> Rahul Sood wrote:
> > Hi,
> >
> > We have a Pipes C++ application where the Reduce task does a lot of
> > computation. After some time the task gets killed by the Hadoop
> > framework. The job output shows the following error:
> >
> > Task task_200803051654_0001_r_000000_0 failed to report status for 604
> > seconds. Killing!
> >
> > Is there any way to send a heartbeat to the TaskTracker from a Pipes
> > application. I believe this is possible in Java using
> > org.apache.hadoop.util.Progress and we're looking for something
> > equivalent in the C++ Pipes API.
> >
> > -Rahul Sood
> > rsood@yahoo-inc.com
> >
> >
> >
>
>


-- 
The University of Edinburgh is a charitable body, registered in Scotland,
with registration number SC005336.

Re: Pipes task being killed

[Vlad Kudelin <vk...@yahoo-inc.com>]

There is or was an option you could try to play with, smth. like
-jobconf mapred.task.timeout=600000
The number is in milliseconds.

// I actually needed to use it to *decrease* the default timeout which I 
believe is quite big; at times things hang, and this is a possible 
workaround to get a hung one killed by the f/w.

PS: I totally am with Richard's know-how: hadoop doesn't care whether 
your app has finished (and how it's finished); all what matters is that 
stdin is consumed...

Vlad.


Richard Kasperski wrote:
> I think you just need to write to stderr. My understanding is that 
> hadoop is happy as long as input is being consumed, output is being 
> generated or status is being generated.
>
> Rahul Sood wrote:
>> Hi,
>>
>> We have a Pipes C++ application where the Reduce task does a lot of
>> computation. After some time the task gets killed by the Hadoop
>> framework. The job output shows the following error:
>>
>> Task task_200803051654_0001_r_000000_0 failed to report status for 604
>> seconds. Killing!
>>
>> Is there any way to send a heartbeat to the TaskTracker from a Pipes
>> application. I believe this is possible in Java using
>> org.apache.hadoop.util.Progress and we're looking for something
>> equivalent in the C++ Pipes API.
>>
>> -Rahul Sood
>> rsood@yahoo-inc.com
>>
>>
>>   
>

Re: Pipes task being killed

[Richard Kasperski <rk...@yahoo-inc.com>]

I think you just need to write to stderr. My understanding is that 
hadoop is happy as long as input is being consumed, output is being 
generated or status is being generated.

Rahul Sood wrote:
> Hi,
>
> We have a Pipes C++ application where the Reduce task does a lot of
> computation. After some time the task gets killed by the Hadoop
> framework. The job output shows the following error:
>
> Task task_200803051654_0001_r_000000_0 failed to report status for 604
> seconds. Killing!
>
> Is there any way to send a heartbeat to the TaskTracker from a Pipes
> application. I believe this is possible in Java using
> org.apache.hadoop.util.Progress and we're looking for something
> equivalent in the C++ Pipes API.
>
> -Rahul Sood
> rsood@yahoo-inc.com
>
>
>   

Re: Pipes task being killed

[Owen O'Malley <oo...@yahoo-inc.com>]

On Mar 5, 2008, at 9:31 AM, Rahul Sood wrote:

> Hi,
>
> We have a Pipes C++ application where the Reduce task does a lot of
> computation. After some time the task gets killed by the Hadoop
> framework. The job output shows the following error:
>
> Task task_200803051654_0001_r_000000_0 failed to report status for 604
> seconds. Killing!
>
> Is there any way to send a heartbeat to the TaskTracker from a Pipes
> application. I believe this is possible in Java using
> org.apache.hadoop.util.Progress and we're looking for something
> equivalent in the C++ Pipes API.

The context object has a progress method that should be called during  
long computations...
http://tinyurl.com/yt7hyx
search for progress...


-- Owen