You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Ajay Srivastava <Aj...@guavus.com> on 2012/12/06 15:57:44 UTC
Query about Speculative Execution
Hi,
What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
Regards,
Ajay Srivastava
Re: Query about Speculative Execution
Posted by Harsh J <ha...@cloudera.com>.
Given that Speculative Execution *is* the answer to such scenarios,
I'd say the answer to your question without it, is *nothing*.
If a task does not report status for over 10 minutes (default), it is
killed and retried. If it does report status changes (such as
counters, task status, etc.) but is slow due to environmental or other
reasons, then the JobTracker, without speculative execution logic
turned on, will assume it is normal.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
--
Harsh J
Re: Query about Speculative Execution
Posted by Harsh J <ha...@cloudera.com>.
Given that Speculative Execution *is* the answer to such scenarios,
I'd say the answer to your question without it, is *nothing*.
If a task does not report status for over 10 minutes (default), it is
killed and retried. If it does report status changes (such as
counters, task status, etc.) but is slow due to environmental or other
reasons, then the JobTracker, without speculative execution logic
turned on, will assume it is normal.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
--
Harsh J
Re: Query about Speculative Execution
Posted by Srinivas Chamarthi <sr...@gmail.com>.
Hi,
may I know where should I find the sources related to speculative
scheduling happens ? and also how do we discard the output coming from
already completed mapper ?
I am actually trying to do something similar, like spawn map tasks
redundantly but not for speculative reasons but for each mapper and reducer
so that I can do a integrity check between the nodes where the tasks are
running.
any help is greatly appreciated.
thx
srinivas
On Thu, Dec 6, 2012 at 8:40 PM, Ajay Srivastava <Aj...@guavus.com>
wrote:
> Thanks Mahesh & Harsh.
>
>
>
> On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
>
> To simply, if you turn-off the speculative execution then the system will
> never bother about slow running jobs unless they won't report beyond
> specified time (10 minutes).
> If you have set speculative execution to true then the system may spawn
> another instance of mapper and consider the output of the fast running once
> or early completing task.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <
> Ajay.Srivastava@guavus.com> wrote:
>
>> Hi,
>>
>> What is the behavior of jobTracker if speculative execution is off and a
>> task on data node is running extremely slow?
>> Will the jobTracker simply wait till the slow running task finishes or it
>> will try to heal the situation? Assuming that heartbeat from the node
>> running slow task are regular.
>>
>>
>>
>> Regards,
>> Ajay Srivastava
>
>
>
>
Re: Query about Speculative Execution
Posted by Srinivas Chamarthi <sr...@gmail.com>.
Hi,
may I know where should I find the sources related to speculative
scheduling happens ? and also how do we discard the output coming from
already completed mapper ?
I am actually trying to do something similar, like spawn map tasks
redundantly but not for speculative reasons but for each mapper and reducer
so that I can do a integrity check between the nodes where the tasks are
running.
any help is greatly appreciated.
thx
srinivas
On Thu, Dec 6, 2012 at 8:40 PM, Ajay Srivastava <Aj...@guavus.com>
wrote:
> Thanks Mahesh & Harsh.
>
>
>
> On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
>
> To simply, if you turn-off the speculative execution then the system will
> never bother about slow running jobs unless they won't report beyond
> specified time (10 minutes).
> If you have set speculative execution to true then the system may spawn
> another instance of mapper and consider the output of the fast running once
> or early completing task.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <
> Ajay.Srivastava@guavus.com> wrote:
>
>> Hi,
>>
>> What is the behavior of jobTracker if speculative execution is off and a
>> task on data node is running extremely slow?
>> Will the jobTracker simply wait till the slow running task finishes or it
>> will try to heal the situation? Assuming that heartbeat from the node
>> running slow task are regular.
>>
>>
>>
>> Regards,
>> Ajay Srivastava
>
>
>
>
Re: Query about Speculative Execution
Posted by Srinivas Chamarthi <sr...@gmail.com>.
Hi,
may I know where should I find the sources related to speculative
scheduling happens ? and also how do we discard the output coming from
already completed mapper ?
I am actually trying to do something similar, like spawn map tasks
redundantly but not for speculative reasons but for each mapper and reducer
so that I can do a integrity check between the nodes where the tasks are
running.
any help is greatly appreciated.
thx
srinivas
On Thu, Dec 6, 2012 at 8:40 PM, Ajay Srivastava <Aj...@guavus.com>
wrote:
> Thanks Mahesh & Harsh.
>
>
>
> On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
>
> To simply, if you turn-off the speculative execution then the system will
> never bother about slow running jobs unless they won't report beyond
> specified time (10 minutes).
> If you have set speculative execution to true then the system may spawn
> another instance of mapper and consider the output of the fast running once
> or early completing task.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <
> Ajay.Srivastava@guavus.com> wrote:
>
>> Hi,
>>
>> What is the behavior of jobTracker if speculative execution is off and a
>> task on data node is running extremely slow?
>> Will the jobTracker simply wait till the slow running task finishes or it
>> will try to heal the situation? Assuming that heartbeat from the node
>> running slow task are regular.
>>
>>
>>
>> Regards,
>> Ajay Srivastava
>
>
>
>
Re: Query about Speculative Execution
Posted by Srinivas Chamarthi <sr...@gmail.com>.
Hi,
may I know where should I find the sources related to speculative
scheduling happens ? and also how do we discard the output coming from
already completed mapper ?
I am actually trying to do something similar, like spawn map tasks
redundantly but not for speculative reasons but for each mapper and reducer
so that I can do a integrity check between the nodes where the tasks are
running.
any help is greatly appreciated.
thx
srinivas
On Thu, Dec 6, 2012 at 8:40 PM, Ajay Srivastava <Aj...@guavus.com>
wrote:
> Thanks Mahesh & Harsh.
>
>
>
> On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
>
> To simply, if you turn-off the speculative execution then the system will
> never bother about slow running jobs unless they won't report beyond
> specified time (10 minutes).
> If you have set speculative execution to true then the system may spawn
> another instance of mapper and consider the output of the fast running once
> or early completing task.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <
> Ajay.Srivastava@guavus.com> wrote:
>
>> Hi,
>>
>> What is the behavior of jobTracker if speculative execution is off and a
>> task on data node is running extremely slow?
>> Will the jobTracker simply wait till the slow running task finishes or it
>> will try to heal the situation? Assuming that heartbeat from the node
>> running slow task are regular.
>>
>>
>>
>> Regards,
>> Ajay Srivastava
>
>
>
>
Re: Query about Speculative Execution
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Mahesh & Harsh.
On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
To simply, if you turn-off the speculative execution then the system will never bother about slow running jobs unless they won't report beyond specified time (10 minutes).
If you have set speculative execution to true then the system may spawn another instance of mapper and consider the output of the fast running once or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <Aj...@guavus.com>> wrote:
Hi,
What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
Regards,
Ajay Srivastava
Re: Query about Speculative Execution
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Mahesh & Harsh.
On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
To simply, if you turn-off the speculative execution then the system will never bother about slow running jobs unless they won't report beyond specified time (10 minutes).
If you have set speculative execution to true then the system may spawn another instance of mapper and consider the output of the fast running once or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <Aj...@guavus.com>> wrote:
Hi,
What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
Regards,
Ajay Srivastava
Re: Query about Speculative Execution
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Mahesh & Harsh.
On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
To simply, if you turn-off the speculative execution then the system will never bother about slow running jobs unless they won't report beyond specified time (10 minutes).
If you have set speculative execution to true then the system may spawn another instance of mapper and consider the output of the fast running once or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <Aj...@guavus.com>> wrote:
Hi,
What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
Regards,
Ajay Srivastava
Re: Query about Speculative Execution
Posted by Ajay Srivastava <Aj...@guavus.com>.
Thanks Mahesh & Harsh.
On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
To simply, if you turn-off the speculative execution then the system will never bother about slow running jobs unless they won't report beyond specified time (10 minutes).
If you have set speculative execution to true then the system may spawn another instance of mapper and consider the output of the fast running once or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <Aj...@guavus.com>> wrote:
Hi,
What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
Regards,
Ajay Srivastava
Re: Query about Speculative Execution
Posted by Mahesh Balija <ba...@gmail.com>.
To simply, if you turn-off the speculative execution then the system will
never bother about slow running jobs unless they won't report beyond
specified time (10 minutes).
If you have set speculative execution to true then the system may spawn
another instance of mapper and consider the output of the fast running once
or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com>wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a
> task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it
> will try to heal the situation? Assuming that heartbeat from the node
> running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
Re: Query about Speculative Execution
Posted by Mahesh Balija <ba...@gmail.com>.
To simply, if you turn-off the speculative execution then the system will
never bother about slow running jobs unless they won't report beyond
specified time (10 minutes).
If you have set speculative execution to true then the system may spawn
another instance of mapper and consider the output of the fast running once
or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com>wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a
> task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it
> will try to heal the situation? Assuming that heartbeat from the node
> running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
Re: Query about Speculative Execution
Posted by Mahesh Balija <ba...@gmail.com>.
To simply, if you turn-off the speculative execution then the system will
never bother about slow running jobs unless they won't report beyond
specified time (10 minutes).
If you have set speculative execution to true then the system may spawn
another instance of mapper and consider the output of the fast running once
or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com>wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a
> task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it
> will try to heal the situation? Assuming that heartbeat from the node
> running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
Re: Query about Speculative Execution
Posted by Harsh J <ha...@cloudera.com>.
Given that Speculative Execution *is* the answer to such scenarios,
I'd say the answer to your question without it, is *nothing*.
If a task does not report status for over 10 minutes (default), it is
killed and retried. If it does report status changes (such as
counters, task status, etc.) but is slow due to environmental or other
reasons, then the JobTracker, without speculative execution logic
turned on, will assume it is normal.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
--
Harsh J
Re: Query about Speculative Execution
Posted by Mahesh Balija <ba...@gmail.com>.
To simply, if you turn-off the speculative execution then the system will
never bother about slow running jobs unless they won't report beyond
specified time (10 minutes).
If you have set speculative execution to true then the system may spawn
another instance of mapper and consider the output of the fast running once
or early completing task.
Best,
Mahesh Balija,
Calsoft Labs.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com>wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a
> task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it
> will try to heal the situation? Assuming that heartbeat from the node
> running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
Re: Query about Speculative Execution
Posted by Harsh J <ha...@cloudera.com>.
Given that Speculative Execution *is* the answer to such scenarios,
I'd say the answer to your question without it, is *nothing*.
If a task does not report status for over 10 minutes (default), it is
killed and retried. If it does report status changes (such as
counters, task status, etc.) but is slow due to environmental or other
reasons, then the JobTracker, without speculative execution logic
turned on, will assume it is normal.
On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava
<Aj...@guavus.com> wrote:
> Hi,
>
> What is the behavior of jobTracker if speculative execution is off and a task on data node is running extremely slow?
> Will the jobTracker simply wait till the slow running task finishes or it will try to heal the situation? Assuming that heartbeat from the node running slow task are regular.
>
>
>
> Regards,
> Ajay Srivastava
--
Harsh J