Re: Query about Speculative Execution

[Srinivas Chamarthi <sr...@gmail.com>]

Hi,

may I know where should I find the sources related to speculative
scheduling happens ? and also how do we discard the output coming from
already completed mapper ?

I am actually trying to do something similar, like spawn map tasks
redundantly but not for speculative reasons but for each mapper and reducer
so that I can do a integrity check between the nodes where the tasks are
running.

any help is greatly appreciated.

thx
srinivas

On Thu, Dec 6, 2012 at 8:40 PM, Ajay Srivastava <Aj...@guavus.com>
wrote:

>  Thanks Mahesh & Harsh.
>
>
>
>  On 07-Dec-2012, at 7:42 AM, Mahesh Balija wrote:
>
> To simply, if you turn-off the speculative execution then the system will
> never bother about slow running jobs unless they won't report beyond
> specified time (10 minutes).
> If you have set speculative execution to true then the system may spawn
> another instance of mapper and consider the output of the fast running once
> or early completing task.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Thu, Dec 6, 2012 at 8:27 PM, Ajay Srivastava <
> Ajay.Srivastava@guavus.com> wrote:
>
>> Hi,
>>
>> What is the behavior of jobTracker if speculative execution is off and a
>> task on data node is running extremely slow?
>> Will the jobTracker simply wait till the slow running task finishes or it
>> will try to heal the situation? Assuming that heartbeat from the node
>> running slow task are regular.
>>
>>
>>
>> Regards,
>> Ajay Srivastava
>
>
>
>