You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wei Xue (Jira)" <ji...@apache.org> on 2019/11/22 03:50:00 UTC

[jira] [Created] (SPARK-29994) Add WILDCARD task location

Wei Xue created SPARK-29994:
-------------------------------

             Summary: Add WILDCARD task location
                 Key: SPARK-29994
                 URL: https://issues.apache.org/jira/browse/SPARK-29994
             Project: Spark
          Issue Type: Task
          Components: Spark Core
    Affects Versions: 3.0.0
            Reporter: Wei Xue


We have a "locality wait time" configuration that allows a task set to downgrade locality requirement after a certain time has passed. Yet, tasks all differ in penalty of locality miss. We propose a WILDCARD task location for tasks that do have a preferred location but would rather be assigned sooner than wait if the desired location is not available.

This requirement comes from LocalShuffledRowRDD. When the number of initial mappers of LocalShuffledRDD is smaller than the number of worker nodes, it can cause serious regressions if short-running tasks all wait on their preferred locations while they could have otherwise finished quickly on non-preferred locations too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org