Questions about Spark speculation

[Nicolas Mai <ni...@gmail.com>]

Hi, guys

My current project is using Spark 0.9.1, and after increasing the level of
parallelism and partitions in our RDDs, stages and tasks seem to complete
much faster. However it also seems that our cluster becomes more "unstable"
after some time:
- stalled stages still showing under "active stages" in the Spark app web
dashboard
- incomplete stages showing under "completed stages"
- stages with failures

I was thinking about reducing/tuning the number of parallelism, but I was
also considering using "spark.speculation" which is currently turned off but
seems promising.

Questions about speculation:
- Just wondering why it is turned off by default?
- Are there any risks using speculation?
- Is it possible that a speculative task straggles, and would trigger
another new speculative task to finish the job... and so on... (some kind of
loop until there's no more executors available).
- What configuration do you guys usually use for spark.speculation?
(interval, quantile, multiplier) I guess it depends on the project, it may
give some ideas about how to use it properly.

Thank you! :)
Nicolas



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Questions-about-Spark-speculation-tp14398.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Questions about Spark speculation

[Andrew Ash <an...@andrewash.com>]

Hi Nicolas,

I've had suspicions about speculation causing problems on my cluster but
don't have any hard evidence of it yet.

I'm also interested in why it's turned off by default.

On Tue, Sep 16, 2014 at 3:01 PM, Nicolas Mai <ni...@gmail.com> wrote:

> Hi, guys
>
> My current project is using Spark 0.9.1, and after increasing the level of
> parallelism and partitions in our RDDs, stages and tasks seem to complete
> much faster. However it also seems that our cluster becomes more "unstable"
> after some time:
> - stalled stages still showing under "active stages" in the Spark app web
> dashboard
> - incomplete stages showing under "completed stages"
> - stages with failures
>
> I was thinking about reducing/tuning the number of parallelism, but I was
> also considering using "spark.speculation" which is currently turned off
> but
> seems promising.
>
> Questions about speculation:
> - Just wondering why it is turned off by default?
> - Are there any risks using speculation?
> - Is it possible that a speculative task straggles, and would trigger
> another new speculative task to finish the job... and so on... (some kind
> of
> loop until there's no more executors available).
> - What configuration do you guys usually use for spark.speculation?
> (interval, quantile, multiplier) I guess it depends on the project, it may
> give some ideas about how to use it properly.
>
> Thank you! :)
> Nicolas
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Questions-about-Spark-speculation-tp14398.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>