[jira] [Created] (SPARK-24049) Add a feature to not start speculative tasks before an absolute minimum amount of time has passed

["Lars Francke (JIRA)" <ji...@apache.org>]

Lars Francke created SPARK-24049:
------------------------------------

             Summary: Add a feature to not start speculative tasks before an absolute minimum amount of time has passed
                 Key: SPARK-24049
                 URL: https://issues.apache.org/jira/browse/SPARK-24049
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.3.0
            Reporter: Lars Francke


Speculation currently has four different configuration options according to the docs:
* spark.speculation to turn it on or off
* .interval - how often it'll check
* .multiplier - how many times slower than average a task must be
* .quantile - fraction of tasks that have to be completed before speculation starts

What I'd love to see is a feature that allows me to set a minimum average duration. We are using a Jobserver to submit varying queries to Spark. Some of those have stages with tasks of an average length of ~20ms (yes, not idea but it happens). So with a multiplier of 2 a speculative task starts after 40ms. This happens quite a lot and in our use-case it doesn't make sense. We don't consider 40ms tasks "stragglers".

So it would make sense to add a parameter to only start speculation when the average task time is greater than X seconds/minutes/millis.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org