You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2019/10/08 05:42:21 UTC
[jira] [Resolved] (SPARK-24049) Add a feature to not start
speculative tasks when average task duration is less than a configurable
absolute number
[ https://issues.apache.org/jira/browse/SPARK-24049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-24049.
----------------------------------
Resolution: Incomplete
> Add a feature to not start speculative tasks when average task duration is less than a configurable absolute number
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-24049
> URL: https://issues.apache.org/jira/browse/SPARK-24049
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Lars Francke
> Priority: Minor
> Labels: bulk-closed
>
> Speculation currently has four different configuration options according to the docs:
> * spark.speculation to turn it on or off
> * .interval - how often it'll check
> * .multiplier - how many times slower than average a task must be
> * .quantile - fraction of tasks that have to be completed before speculation starts
> What I'd love to see is a feature that allows me to set a minimum average duration. We are using a Jobserver to submit varying queries to Spark. Some of those have stages with tasks of an average length of ~20ms (yes, not idea but it happens). So with a multiplier of 2 a speculative task starts after 40ms. This happens quite a lot and in our use-case it doesn't make sense. We don't consider 40ms tasks "stragglers".
> So it would make sense to add a parameter to only start speculation when the average task time is greater than X seconds/minutes/millis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org