[GitHub] spark issue #11205: [SPARK-11334][Core] Handle maximum task failure situatio...

[rustagi <gi...@git.apache.org>]

Github user rustagi commented on the issue:

    https://github.com/apache/spark/pull/11205
  
    I am seeing this issue quite frequently. Not sure what is causing it but frequently we will get a onTaskEnd event after a stage has ended. This will cause the numRunningTasks to become negative. If executor number is updated then number of required executors(maxNumExecutorsNeeded) becomes negative & have issues in new executor allocation and deallocation. Best case you get executors that are unable to deallocate & over time spark does not allocate new executors even if there are tasks pending.
    There is a simple hacky patch here: https://github.com/apache/spark/pull/9288 & this one is an attempt to correct it with more accountability. 
    I am seeing this issue so frequently that I am not sure its possible to run Spark with dynamic allocation successfully for long duration without fixing it. I'll try the hacky patch & confirm. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org