[GitHub] [spark] LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time

[GitBox <gi...@apache.org>]

LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r270745197
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
 ##########
 @@ -811,8 +818,38 @@ private[spark] class TaskSchedulerImpl(
     blacklistTrackerOpt.map(_.nodeBlacklist()).getOrElse(scala.collection.immutable.Set())
   }
 
+  // Add a on-off switch to save time for rack resolving
+  private def skipRackResolving: Boolean = sc.conf.get(LOCALITY_WAIT_RACK) == 0L
 
 Review comment:
   I knew that. Isolation deployment of computing and storage is more and more popular in industry. If someone set `LOCALITY_WAIT` to 0 on purpose, almost time it is no need data locality (especially rack level) at all. Rack resolving and locality algorithm still spends time in no need data locality case. Could we open a new ticket to address this isolation case?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org