[GitHub] spark pull request #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-upl...

[loneknightpy <gi...@git.apache.org>]

Github user loneknightpy commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19074#discussion_r139247692
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -366,7 +376,7 @@ object SparkSubmit extends CommandLineUtils {
             // If a python file is provided, add it to the child arguments and list of files to deploy.
             // Usage: PythonAppRunner <main python file> <extra python files> [app arguments]
             args.mainClass = "org.apache.spark.deploy.PythonRunner"
    -        args.childArgs = ArrayBuffer(args.primaryResource, args.pyFiles) ++ args.childArgs
    +        args.childArgs = ArrayBuffer(localPrimaryResource, localPyFiles) ++ args.childArgs
             if (clusterManager != YARN) {
               // The YARN backend distributes the primary file differently, so don't merge it.
               args.files = mergeFileLists(args.files, args.primaryResource)
    --- End diff --
    
    It is a behavior change here. I think we should use `localFiles` and `localPrimaryResource` instead of `args.files` and `args.primaryResource`. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org