[GitHub] [spark] sjrand commented on issue #26343: [SPARK-29683][YARN] Job will fail due to executor failures all available nodes are blacklisted

[GitBox <gi...@apache.org>]

sjrand commented on issue #26343: [SPARK-29683][YARN] Job will fail due to executor failures all available nodes are blacklisted
URL: https://github.com/apache/spark/pull/26343#issuecomment-601145754
 
 
   There's still a similar issue in the proposal to do `def isAllNodeBlacklisted: Boolean = numClusterNodes != 0 && currentBlacklistedYarnNodes.size >= numClusterNodes`. Since the NodeManagers register with the newly active RM one at a time after the failover, you can wind up with a situation where `numClusterNodes` is temporarily 1 during the failover (or some other small number). If we happen to have 1 blacklisted node at this time, and we happen to also trigger the `isAllNodeBlacklisted` check, then we still fail the application.
   
   That said, I think the situation I'm describing should happen quite rarely in practice, so it's still a small change that makes things much better. I'm supportive of that idea. @uncleGen, does that seem reasonable to you?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org