You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Pramod Biligiri <pr...@gmail.com> on 2014/09/16 00:30:02 UTC

Does Spark always wait for stragglers to finish running?

Hi,
I'm running Spark tasks with speculation enabled. I'm noticing that Spark
seems to wait in a given stage for all stragglers to finish, even though
the speculated alternative might have finished sooner. Is that correct?

Is there a way to indicate to Spark not to wait for stragglers to finish?

Thanks,
Pramod
-- 
http://twitter.com/pramodbiligiri

Re: Does Spark always wait for stragglers to finish running?

Posted by Matei Zaharia <ma...@gmail.com>.
It's true that it does not send a kill command right now -- we should probably add that. This code was written before tasks were killable AFAIK. However, the *job* should still finish while a speculative task is running as far as I know, and it will just leave that task behind.

Matei

On September 15, 2014 at 4:51:59 PM, Pramod Biligiri (pramodbiligiri@gmail.com) wrote:

I'm already running with speculation set to true and the speculated tasks are launching, but the issue I'm observing is that Spark does not kill the long running task even if the shorter alternative has finished successfully. Therefore the overall turnaround time is still the same as without speculation.

Pramod

On Mon, Sep 15, 2014 at 4:22 PM, Du Li <li...@yahoo-inc.com> wrote:
There is a parameter spark.speculation that is turned off by default. Look at the configuration doc: http://spark.apache.org/docs/latest/configuration.html



From: Pramod Biligiri <pr...@gmail.com>
Date: Monday, September 15, 2014 at 3:30 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Does Spark always wait for stragglers to finish running?

Hi,
I'm running Spark tasks with speculation enabled. I'm noticing that Spark seems to wait in a given stage for all stragglers to finish, even though the speculated alternative might have finished sooner. Is that correct?

Is there a way to indicate to Spark not to wait for stragglers to finish?

Thanks,
Pramod
-- 
http://twitter.com/pramodbiligiri

Re: Does Spark always wait for stragglers to finish running?

Posted by Pramod Biligiri <pr...@gmail.com>.
I'm already running with speculation set to true and the speculated tasks
are launching, but the issue I'm observing is that Spark does not kill the
long running task even if the shorter alternative has finished
successfully. Therefore the overall turnaround time is still the same as
without speculation.

Pramod

On Mon, Sep 15, 2014 at 4:22 PM, Du Li <li...@yahoo-inc.com> wrote:

>   There is a parameter spark.speculation that is turned off by default.
> Look at the configuration doc:
> http://spark.apache.org/docs/latest/configuration.html
>
>
>
>   From: Pramod Biligiri <pr...@gmail.com>
> Date: Monday, September 15, 2014 at 3:30 PM
> To: "user@spark.apache.org" <us...@spark.apache.org>
> Subject: Does Spark always wait for stragglers to finish running?
>
>   Hi,
> I'm running Spark tasks with speculation enabled. I'm noticing that Spark
> seems to wait in a given stage for all stragglers to finish, even though
> the speculated alternative might have finished sooner. Is that correct?
>
>  Is there a way to indicate to Spark not to wait for stragglers to finish?
>
>  Thanks,
> Pramod
> --
> http://twitter.com/pramodbiligiri
>

Re: Does Spark always wait for stragglers to finish running?

Posted by Du Li <li...@yahoo-inc.com.INVALID>.
There is a parameter spark.speculation that is turned off by default. Look at the configuration doc: http://spark.apache.org/docs/latest/configuration.html



From: Pramod Biligiri <pr...@gmail.com>>
Date: Monday, September 15, 2014 at 3:30 PM
To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Does Spark always wait for stragglers to finish running?

Hi,
I'm running Spark tasks with speculation enabled. I'm noticing that Spark seems to wait in a given stage for all stragglers to finish, even though the speculated alternative might have finished sooner. Is that correct?

Is there a way to indicate to Spark not to wait for stragglers to finish?

Thanks,
Pramod
--
http://twitter.com/pramodbiligiri