[GitHub] [spark] cxzl25 commented on a change in pull request #30036: [SPARK-33147][CORE] Avoid distribute user jar from driver in yarn client mode

[GitBox <gi...@apache.org>]

cxzl25 commented on a change in pull request #30036:
URL: https://github.com/apache/spark/pull/30036#discussion_r583521775



##########
File path: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala
##########
@@ -169,12 +169,13 @@ private[deploy] object DependencyUtils extends Logging {
   }
 
   /**
-   * Merge a sequence of comma-separated file lists, some of which may be null to indicate
-   * no files, into a single comma-separated string.
+   * Merge and de-duplicate a sequence of comma-separated file lists,
+   * some of which may be null to indicate no files,
+   * into a single comma-separated string.
    */
   def mergeFileLists(lists: String*): String = {
     val merged = lists.filterNot(StringUtils.isBlank)
-      .flatMap(Utils.stringToSeq)
+      .flatMap(Utils.stringToSeq).distinct

Review comment:
       
   Avoid uploading the same file
   
   https://github.com/apache/spark/blob/5c7d019b609c87a9427fa9309f3aa03d02f61878/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L455-L460
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org