You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Juan P." <go...@gmail.com> on 2012/08/27 15:13:28 UTC
Locked Job
Hi guys!
I need some clarification on the expected behavior for a hadoop MapReduce
job.
Say I was to create a Mapper task which never ends. It reads the first line
of input and then reads data from an external service eternally. If the
service is empty it will lock until data is available.
- Will the jobtracker continue to receive the Heartbeat?
- Will the jobtracker kill the task at some point?
I know that it's not the way Hadoop was intended to be used, I just need to
clarify this specific scenario.
Thank you!
Re: Locked Job
Posted by Michael Segel <mi...@hotmail.com>.
It depends...
If you are in Mapper.map() method and you 'lock', then you will most certainly time out after 10min by default and the task dies. Enough tasks die, then your job dies.
If in Mapper.setup() method you create a heartbeat thread where every minute you wake up and update the thread's status with a changing value like time in ms, you can keep the thread alive.
Having said that... you really don't want to do what you're thinking.
Like you really don't want to do it because you tie up a slot and that limits the ability for another job to take its place and run.
HTH
-Mike
On Aug 27, 2012, at 8:13 AM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce job.
>
> Say I was to create a Mapper task which never ends. It reads the first line of input and then reads data from an external service eternally. If the service is empty it will lock until data is available.
>
> Will the jobtracker continue to receive the Heartbeat?
> Will the jobtracker kill the task at some point?
>
> I know that it's not the way Hadoop was intended to be used, I just need to clarify this specific scenario.
>
> Thank you!
Re: Locked Job
Posted by Michael Segel <mi...@hotmail.com>.
It depends...
If you are in Mapper.map() method and you 'lock', then you will most certainly time out after 10min by default and the task dies. Enough tasks die, then your job dies.
If in Mapper.setup() method you create a heartbeat thread where every minute you wake up and update the thread's status with a changing value like time in ms, you can keep the thread alive.
Having said that... you really don't want to do what you're thinking.
Like you really don't want to do it because you tie up a slot and that limits the ability for another job to take its place and run.
HTH
-Mike
On Aug 27, 2012, at 8:13 AM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce job.
>
> Say I was to create a Mapper task which never ends. It reads the first line of input and then reads data from an external service eternally. If the service is empty it will lock until data is available.
>
> Will the jobtracker continue to receive the Heartbeat?
> Will the jobtracker kill the task at some point?
>
> I know that it's not the way Hadoop was intended to be used, I just need to clarify this specific scenario.
>
> Thank you!
Re: Locked Job
Posted by Harsh J <ha...@cloudera.com>.
Hi,
On Mon, Aug 27, 2012 at 6:43 PM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce
> job.
>
> Say I was to create a Mapper task which never ends. It reads the first line
> of input and then reads data from an external service eternally. If the
> service is empty it will lock until data is available.
This is possible to do. However, I urge you to instead drive your job
based on an input, and launch jobs based on an available input event.
Apache Oozie (incubating) helps you do this, as one example framework.
> Will the jobtracker continue to receive the Heartbeat?
If you manually (and periodically) send a status or a progress update
from the task (via the Reporter/Context APIs), it will be received by
the JobTracker as a healthy sign (i.e. a heartbeat from a task).
> Will the jobtracker kill the task at some point?
If you do not do the above, then the task is killed at end of 10
minutes of inactivity, controlled via config param
mapred.task.timeout.
> I know that it's not the way Hadoop was intended to be used, I just need to
> clarify this specific scenario.
>
> Thank you!
+1 to what Michael's said.
--
Harsh J
Re: Locked Job
Posted by Harsh J <ha...@cloudera.com>.
Hi,
On Mon, Aug 27, 2012 at 6:43 PM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce
> job.
>
> Say I was to create a Mapper task which never ends. It reads the first line
> of input and then reads data from an external service eternally. If the
> service is empty it will lock until data is available.
This is possible to do. However, I urge you to instead drive your job
based on an input, and launch jobs based on an available input event.
Apache Oozie (incubating) helps you do this, as one example framework.
> Will the jobtracker continue to receive the Heartbeat?
If you manually (and periodically) send a status or a progress update
from the task (via the Reporter/Context APIs), it will be received by
the JobTracker as a healthy sign (i.e. a heartbeat from a task).
> Will the jobtracker kill the task at some point?
If you do not do the above, then the task is killed at end of 10
minutes of inactivity, controlled via config param
mapred.task.timeout.
> I know that it's not the way Hadoop was intended to be used, I just need to
> clarify this specific scenario.
>
> Thank you!
+1 to what Michael's said.
--
Harsh J
Re: Locked Job
Posted by Harsh J <ha...@cloudera.com>.
Hi,
On Mon, Aug 27, 2012 at 6:43 PM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce
> job.
>
> Say I was to create a Mapper task which never ends. It reads the first line
> of input and then reads data from an external service eternally. If the
> service is empty it will lock until data is available.
This is possible to do. However, I urge you to instead drive your job
based on an input, and launch jobs based on an available input event.
Apache Oozie (incubating) helps you do this, as one example framework.
> Will the jobtracker continue to receive the Heartbeat?
If you manually (and periodically) send a status or a progress update
from the task (via the Reporter/Context APIs), it will be received by
the JobTracker as a healthy sign (i.e. a heartbeat from a task).
> Will the jobtracker kill the task at some point?
If you do not do the above, then the task is killed at end of 10
minutes of inactivity, controlled via config param
mapred.task.timeout.
> I know that it's not the way Hadoop was intended to be used, I just need to
> clarify this specific scenario.
>
> Thank you!
+1 to what Michael's said.
--
Harsh J
Re: Locked Job
Posted by Michael Segel <mi...@hotmail.com>.
It depends...
If you are in Mapper.map() method and you 'lock', then you will most certainly time out after 10min by default and the task dies. Enough tasks die, then your job dies.
If in Mapper.setup() method you create a heartbeat thread where every minute you wake up and update the thread's status with a changing value like time in ms, you can keep the thread alive.
Having said that... you really don't want to do what you're thinking.
Like you really don't want to do it because you tie up a slot and that limits the ability for another job to take its place and run.
HTH
-Mike
On Aug 27, 2012, at 8:13 AM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce job.
>
> Say I was to create a Mapper task which never ends. It reads the first line of input and then reads data from an external service eternally. If the service is empty it will lock until data is available.
>
> Will the jobtracker continue to receive the Heartbeat?
> Will the jobtracker kill the task at some point?
>
> I know that it's not the way Hadoop was intended to be used, I just need to clarify this specific scenario.
>
> Thank you!
Re: Locked Job
Posted by Michael Segel <mi...@hotmail.com>.
It depends...
If you are in Mapper.map() method and you 'lock', then you will most certainly time out after 10min by default and the task dies. Enough tasks die, then your job dies.
If in Mapper.setup() method you create a heartbeat thread where every minute you wake up and update the thread's status with a changing value like time in ms, you can keep the thread alive.
Having said that... you really don't want to do what you're thinking.
Like you really don't want to do it because you tie up a slot and that limits the ability for another job to take its place and run.
HTH
-Mike
On Aug 27, 2012, at 8:13 AM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce job.
>
> Say I was to create a Mapper task which never ends. It reads the first line of input and then reads data from an external service eternally. If the service is empty it will lock until data is available.
>
> Will the jobtracker continue to receive the Heartbeat?
> Will the jobtracker kill the task at some point?
>
> I know that it's not the way Hadoop was intended to be used, I just need to clarify this specific scenario.
>
> Thank you!
Re: Locked Job
Posted by Harsh J <ha...@cloudera.com>.
Hi,
On Mon, Aug 27, 2012 at 6:43 PM, Juan P. <go...@gmail.com> wrote:
> Hi guys!
>
> I need some clarification on the expected behavior for a hadoop MapReduce
> job.
>
> Say I was to create a Mapper task which never ends. It reads the first line
> of input and then reads data from an external service eternally. If the
> service is empty it will lock until data is available.
This is possible to do. However, I urge you to instead drive your job
based on an input, and launch jobs based on an available input event.
Apache Oozie (incubating) helps you do this, as one example framework.
> Will the jobtracker continue to receive the Heartbeat?
If you manually (and periodically) send a status or a progress update
from the task (via the Reporter/Context APIs), it will be received by
the JobTracker as a healthy sign (i.e. a heartbeat from a task).
> Will the jobtracker kill the task at some point?
If you do not do the above, then the task is killed at end of 10
minutes of inactivity, controlled via config param
mapred.task.timeout.
> I know that it's not the way Hadoop was intended to be used, I just need to
> clarify this specific scenario.
>
> Thank you!
+1 to what Michael's said.
--
Harsh J