[jira] [Created] (SPARK-4167) Schedule task on Executor will be Imbalance while task run less than local-wait time

["SuYan (JIRA)" <ji...@apache.org>]

SuYan created SPARK-4167:
----------------------------

             Summary: Schedule task on Executor will be Imbalance while task run less than local-wait time
                 Key: SPARK-4167
                 URL: https://issues.apache.org/jira/browse/SPARK-4167
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.1.0
            Reporter: SuYan


Recently, when run a spark on yarn job. it occurs executor schedules imbalance. 
the procedure is that: 
1. because user's mistake, the spark on yarn job's input split contains 0 byte empty splits. 
1.1: task0-99 , no-preference task(0 byte) task100-800, node-local task 1.2: user will run task 500 loops
1.3: 60 executor

 2.
 executor A only have 2 node-local task in the first loop, executor A first finished node-local-task, the it will run no-preference task, and the no-preference task in our situation have smaller input split than node-local task. So executor A finished all no-reference task, while others still run node-local job. 

in the second loop, all task have process-local level, and all task finished in 3 seconds, so while executor A is still run process-local task while others are all finished process-local task. but all process-task run by executor A will finished in 3 seconds, so the local level will always be process-local. 

it results other executors are all wait for executor A the same situation in the left loops. 

To solve this situation, we let user to delete the empty input split.
 but is still have implied imbalance, while in some loops, a executor got more process-local task than others in one loop, and this task all less-3 seconds task. and then in the left loops, the others executor will wait that executor to finished all process-local tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org