[GitHub] [spark] tgravescs commented on a diff in pull request #39127: [SPARK-41585][YARN] The Spark exclude node functionality for YARN should work independently of dynamic allocation

["tgravescs (via GitHub)" <gi...@apache.org>]

tgravescs commented on code in PR #39127:
URL: https://github.com/apache/spark/pull/39127#discussion_r1109108543


##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -563,6 +565,7 @@ private[yarn] class YarnAllocator(
           }
         }
 
+        allocatorNodeHealthTracker.setSchedulerExcludedNodes(excludeNodes)

Review Comment:
   this feels like a weird way to do this based on the YarnAllocatorNodeHealthTracker already reads the excludeNode and has it as a private variable.  This api was meant for excluding nodes that had dynamic failures... ie the scheduler excluded them.  These are not excluded by the scheduler but by user manually.
   
   what if we just have it call refreshExcludedNodes when the YarnAllocatorNodeHealthTracker  is created?  Unless you are trying to dynamically change this config which it wasn't really meant for.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org