[GitHub] [spark] tgravescs commented on pull request #35858: [SPARK-38448] [YARN] [CORE] Sending Available Resources in Yarn Cluster Information to Spark Driver

[GitBox <gi...@apache.org>]

tgravescs commented on pull request #35858:
URL: https://github.com/apache/spark/pull/35858#issuecomment-1070981659


   Yeah, the resources available on a large cluster can change very rapidly and it should not be relied upon.  I guess your proposal here is to specifically request hosts?  In some ways this is like the locality requests, but there is no way to guarantee what YARN told you was available in one heartbeat will still be available in the next one.  Spark can figure out what it wants for requirements - locality for data, networks, etc.. but seems very perilous to try to assume we can know what YARN is doing.   Even with likely data locality, generally you have 3 replicas and request 3 hosts and it only tries to get those for a limited amount of time.  I have seen way to many times we request specific hosts and jobs take longer because of it vs just running on what is available (which YARN decides).
   
   In the end what is your end goal by making these changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org