Overriding mapred.tasktracker.expiry.interval on a per-job basis

[John Bond <jo...@gmail.com>]

Hello,

Im running a map/reduce job which dose not send progress updates
during the reduces phase so if this takes longer then 10 minutes the
task is seen to fail and restarted.  Under normall operations this
works as the reduce phase only takes a few minutes; however i am
trying to run this job with some historical data and the reduce phase
is taking longer then 10 minutes and constantly being restarted.

Obviously the correct fix is to implement a reporter[1] which has been
corrected in the dev branch and will be rolled out once it has gone
through release management 8-|.  In the mean time is there a way to
override mapred.tasktracker.expiry.interval on for a specific job
without changing mapred-site.xml and restarting the cluster.

I attempted to do the following:

`hadoop jar /path/to/jar/job.jar  class.to.run
-Dmapred.tasktracker.expiry.interval=600000000 arg1 arg2`

And in the job.conf i can see the following is set however the jobs
are still seen as failed after 10 minutes

mapred.tasktracker.expiry.interval	600000000

Thanks
John



[1]http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html

Re: Overriding mapred.tasktracker.expiry.interval on a per-job basis

[John Bond <jo...@gmail.com>]

On 14 December 2011 15:34, John Bond <jo...@gmail.com> wrote:
> I attempted to do the following:
>
> `hadoop jar /path/to/jar/job.jar  class.to.run
> -Dmapred.tasktracker.expiry.interval=600000000 arg1 arg2`
>
> And in the job.conf i can see the following is set however the jobs
> are still seen as failed after 10 minutes
>
> mapred.tasktracker.expiry.interval      600000000
Ok i think i have worked it out.  I needed to use
mapred.task.timeout
and not mapred.tasktracker.expiry.interval

Should have spend another 5 mins reading the docs before hitting send