[GitHub] [spark] HyukjinKwon commented on a change in pull request #23972: [SPARK-26602] Insert into table fails after querying UDF loaded with wrong hdfs path

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #23972: [SPARK-26602] Insert into table fails after querying UDF loaded with wrong hdfs path
URL: https://github.com/apache/spark/pull/23972#discussion_r262914744
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/SparkContext.scala
 ##########
 @@ -1814,12 +1814,16 @@ class SparkContext(config: SparkConf) extends Logging {
       }
       if (key != null) {
         val timestamp = System.currentTimeMillis
-        if (addedJars.putIfAbsent(key, timestamp).isEmpty) {
-          logInfo(s"Added JAR $path at $key with timestamp $timestamp")
-          postEnvironmentUpdate()
-        } else {
-          logWarning(s"The jar $path has been added already. Overwriting of added jars " +
-            "is not supported in the current version.")
+        //  SPARK-26602: Make sure the path exists,before adding jar to addedJars
+        val fs = FileSystem.get(hadoopConfiguration)
+        if (fs.exists(new Path(path))) {
 
 Review comment:
   BTW, this looks going to fail if we use `local` scheme.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org